Sep 15 21:10:30.884450 ip-10-0-139-184 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 21:10:31.235004 ip-10-0-139-184 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:10:31.235004 ip-10-0-139-184 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 21:10:31.235004 ip-10-0-139-184 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:10:31.235004 ip-10-0-139-184 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 21:10:31.235004 ip-10-0-139-184 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:10:31.237111 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.237031    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 21:10:31.240858 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240843    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:10:31.240858 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240858    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240862    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240869    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240873    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240876    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240880    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240883    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240886    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240889    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240892    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240895    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240898    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240902    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240905    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240907    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240911    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240913    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240916    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240919    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240922    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:10:31.240920 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240925    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240927    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240931    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240933    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240936    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240939    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240941    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240944    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240947    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240949    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240951    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240954    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240956    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240959    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240962    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240964    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240967    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240969    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240971    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240974    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:10:31.241424 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240976    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240979    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240981    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240984    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240986    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240989    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240992    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240994    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240997    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.240999    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241002    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241004    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241007    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241009    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241013    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241016    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241019    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241023    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241025    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:10:31.242126 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241029    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241031    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241034    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241036    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241039    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241041    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241044    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241046    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241049    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241052    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241054    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241057    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241059    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241062    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241065    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241068    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241071    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241074    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241077    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241079    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:10:31.242701 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241082    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241085    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241087    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241092    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241096    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241098    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241102    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241107    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241110    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241114    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241117    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241119    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241123    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241126    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241129    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241131    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241134    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241136    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.241139    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:10:31.243214 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242622    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242633    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242636    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242639    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242644    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242649    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242653    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242656    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242659    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242661    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242665    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242667    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242670    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242672    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242675    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242677    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242680    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242682    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242685    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:10:31.243672 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242688    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242692    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242694    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242697    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242700    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242702    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242706    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242717    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242721    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242723    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242726    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242729    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242731    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242734    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242736    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242739    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242743    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242747    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242750    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:10:31.244137 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242752    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242755    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242758    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242760    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242763    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242766    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242769    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242771    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242774    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242776    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242780    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242782    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242785    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242788    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242791    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242793    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242796    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242799    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242802    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242804    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242807    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:10:31.244655 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242809    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242812    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242815    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242817    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242820    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242822    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242825    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242828    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242830    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242833    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242837    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242839    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242842    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242844    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242847    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242849    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242852    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242855    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242857    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242860    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:10:31.245216 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242862    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242865    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242868    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242870    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242875    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242878    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242880    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242883    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242886    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242888    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242891    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242893    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242896    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242899    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242902    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242904    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242907    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242909    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242912    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.242915    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:10:31.245696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243300    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243305    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243308    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243311    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243314    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243317    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243321    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243325    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243329    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243332    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243335    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243338    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243341    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243343    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243346    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243350    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243354    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243356    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243359    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:10:31.246199 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243362    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243364    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243367    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243369    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243372    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243374    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243377    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243379    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243382    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243384    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243387    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243389    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243392    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243395    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243403    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243405    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243408    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243410    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243413    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243415    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:10:31.246667 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243418    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243420    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243423    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243426    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243428    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243430    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243433    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243435    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243438    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243441    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243444    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243447    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243449    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243452    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243455    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243458    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243460    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243462    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243465    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243467    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:10:31.247156 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243470    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243473    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243475    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243477    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243480    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243483    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243485    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243487    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243490    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243492    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243495    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243497    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243500    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243502    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243507    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243510    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243513    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243517    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:10:31.247641 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243521    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243524    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243527    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243530    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243533    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243536    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243539    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243541    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243544    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243546    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243549    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243552    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243554    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243557    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243559    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243562    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243564    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243567    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243569    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243571    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:10:31.248066 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243574    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.243576    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243644    2584 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243682    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243689    2584 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243694    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243698    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243701    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243705    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243708    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243712    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243715    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243718    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243721    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243724    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243727    2584 flags.go:64] FLAG: --cgroup-root=""
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243730    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243733    2584 flags.go:64] FLAG: --client-ca-file=""
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243736    2584 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243739    2584 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243742    2584 flags.go:64] FLAG: --cluster-domain=""
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243745    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243748    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243751    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 21:10:31.248551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243754    2584 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243758    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243761    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243764    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243767    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243770    2584 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243773    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243776    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243779    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243782    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243786    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243789    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243791    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243794    2584 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243797    2584 flags.go:64] FLAG: --enable-server="true"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243800    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243805    2584 flags.go:64] FLAG: --event-burst="100"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243810    2584 flags.go:64] FLAG: --event-qps="50"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243813    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243816    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243819    2584 flags.go:64] FLAG: --eviction-hard=""
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243823    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243827    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243831    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243833    2584 flags.go:64] FLAG: --eviction-soft=""
Sep 15 21:10:31.249121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243836    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243839    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243842    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243846    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243849    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243852    2584 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243855    2584 flags.go:64] FLAG: --feature-gates=""
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243858    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243861    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243864    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243867    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243870    2584 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243873    2584 flags.go:64] FLAG: --help="false"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243876    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243879    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243882    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243885    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243888    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243892    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243894    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243897    2584 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243900    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243903    2584 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 21:10:31.249725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243905    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243910    2584 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243912    2584 flags.go:64] FLAG: --kube-reserved=""
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243915    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243918    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243921    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243925    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243928    2584 flags.go:64] FLAG: --lock-file=""
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243931    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243934    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243937    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243942    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243945    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243948    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243951    2584 flags.go:64] FLAG: --logging-format="text"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243954    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243957    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243960    2584 flags.go:64] FLAG: --manifest-url=""
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243962    2584 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243966    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243969    2584 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243973    2584 flags.go:64] FLAG: --max-pods="110"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243976    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243979    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243982    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 21:10:31.250356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243984    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243987    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243990    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.243993    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244000    2584 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244003    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244006    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244009    2584 flags.go:64] FLAG: --pod-cidr=""
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244013    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244018    2584 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244021    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244024    2584 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244027    2584 flags.go:64] FLAG: --port="10250"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244032    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244034    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-002afa0b8b2b177a1"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244037    2584 flags.go:64] FLAG: --qos-reserved=""
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244040    2584 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244043    2584 flags.go:64] FLAG: --register-node="true"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244046    2584 flags.go:64] FLAG: --register-with-taints=""
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244050    2584 flags.go:64] FLAG: --registry-burst="10"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244053    2584 flags.go:64] FLAG: --registry-qps="5"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244056    2584 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244059    2584 flags.go:64] FLAG: --reserved-memory=""
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244063    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244066    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 21:10:31.250958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244069    2584 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244071    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244074    2584 flags.go:64] FLAG: --runonce="false"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244077    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244080    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244083    2584 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244086    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244089    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244092    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244095    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244098    2584 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244101    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244104    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244107    2584 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244109    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244112    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244117    2584 flags.go:64] FLAG: --system-cgroups=""
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244120    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244125    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244128    2584 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244132    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244136    2584 flags.go:64] FLAG: --tls-min-version=""
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244138    2584 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244141    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244144    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 21:10:31.251568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244161    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244167    2584 flags.go:64] FLAG: --v="2"
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244172    2584 flags.go:64] FLAG: --version="false"
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244176    2584 flags.go:64] FLAG: --vmodule=""
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244180    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244183    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244304    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244308    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244311    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244314    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244317    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244320    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244323    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244325    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244328    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244331    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244333    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244336    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244338    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244341    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244344    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:10:31.252187 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244346    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244349    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244353    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244356    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244359    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244361    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244365    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244368    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244370    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244373    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244375    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244378    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244380    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244383    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244386    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244389    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244391    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244395    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244398    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:10:31.252783 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244402    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244404    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244407    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244409    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244412    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244414    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244417    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244419    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244422    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244425    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244427    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244430    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244432    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244435    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244437    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244441    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244443    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244447    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244451    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:10:31.253276 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244455    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244458    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244461    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244464    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244467    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244469    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244472    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244475    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244478    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244480    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244483    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244486    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244489    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244491    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244494    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244497    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244500    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244502    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244505    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244508    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:10:31.253749 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244512    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244515    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244517    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244520    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244522    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244525    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244528    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244530    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244534    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244537    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244540    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244542    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244547    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244549    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244552    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244555    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244557    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244560    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244562    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244565    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:10:31.254278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244567    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:10:31.254784 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244570    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:10:31.254784 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244572    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:10:31.254784 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244575    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:10:31.254784 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244578    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:10:31.254784 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.244580    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:10:31.254784 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.244971    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:10:31.254784 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.252267    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 21:10:31.254784 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.252283    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 21:10:31.254784 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.252301    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 21:10:31.254784 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.252307    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 21:10:31.254784 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252362    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:10:31.254784 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252366    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:10:31.254784 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252370    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:10:31.254784 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252374    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:10:31.254784 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252377    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252380    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252384    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252386    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252389    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252392    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252395    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252398    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252401    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252404    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252406    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252409    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252411    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252414    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252417    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252419    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252422    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252425    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252427    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252430    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:10:31.255173 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252433    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252438    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252442    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252445    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252448    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252451    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252455    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252458    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252461    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252463    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252466    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252469    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252472    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252475    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252477    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252480    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252482    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252485    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252488    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:10:31.255677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252490    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252493    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252495    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252498    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252500    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252504    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252508    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252511    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252513    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252516    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252519    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252522    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252524    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252527    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252530    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252533    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252536    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252539    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252542    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:10:31.256143 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252544    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252548    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252552    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252555    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252558    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252560    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252563    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252566    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252569    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252572    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252574    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252577    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252580    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252582    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252584    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252587    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252590    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252592    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252595    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:10:31.256618 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252598    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252600    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252603    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252605    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252608    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252610    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252613    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252615    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252618    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252621    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252624    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252627    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252629    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252632    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252634    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252637    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252639    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:10:31.257075 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252642    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:10:31.257562 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.252647    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:10:31.257562 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252757    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:10:31.257562 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252761    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:10:31.257562 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252764    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:10:31.257562 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252767    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:10:31.257562 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252770    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:10:31.257562 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252773    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:10:31.257562 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252775    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:10:31.257562 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252778    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:10:31.257562 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252780    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:10:31.257562 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252784    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:10:31.257562 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252786    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:10:31.257562 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252789    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:10:31.257562 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252792    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:10:31.257562 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252795    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252798    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252800    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252803    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252806    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252808    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252811    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252814    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252817    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252820    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252823    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252825    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252828    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252831    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252833    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252836    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252838    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252841    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252843    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:10:31.257925 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252846    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252849    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252852    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252855    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252858    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252860    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252863    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252866    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252868    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252871    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252873    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252876    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252878    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252881    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252883    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252896    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252899    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252901    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252904    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252907    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:10:31.258404 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252910    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252912    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252915    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252918    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252921    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252924    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252926    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252929    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252931    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252934    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252937    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252939    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252942    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252945    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252947    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252950    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252953    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252955    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252958    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252960    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:10:31.258898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252963    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252965    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252968    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252970    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252973    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252975    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252978    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252980    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252983    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252985    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252988    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252990    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252993    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252996    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.252998    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.253001    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.253004    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.253006    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.253009    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.253012    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:10:31.259401 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.253014    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:10:31.260112 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.253017    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:10:31.260112 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.253020    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:10:31.260112 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.253022    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:10:31.260112 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.253025    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:10:31.260112 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.253029    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:10:31.260112 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:31.253032    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:10:31.260112 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.253038    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:10:31.260112 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.253129    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 21:10:31.260112 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.255165    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 21:10:31.260112 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.255935    2584 server.go:1019] "Starting client certificate rotation"
Sep 15 21:10:31.260112 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.256029    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 21:10:31.260112 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.256058    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 21:10:31.260443 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.258413    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 21:10:31.260443 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.259514    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 21:10:31.273993 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.273973    2584 log.go:25] "Validated CRI v1 runtime API"
Sep 15 21:10:31.277481 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.277464    2584 log.go:25] "Validated CRI v1 image API"
Sep 15 21:10:31.278349 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.278321    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 21:10:31.281599 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.281579    2584 fs.go:135] Filesystem UUIDs: map[6e596a03-d641-4bad-9c2e-3215917c1d87:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9f0a3612-3560-4556-9551-c1cf2834a032:/dev/nvme0n1p3]
Sep 15 21:10:31.281708 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.281597    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 21:10:31.282662 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.282641    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 21:10:31.287367 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.287259    2584 manager.go:216] Machine: {Timestamp:2026-09-15 21:10:31.285250097 +0000 UTC m=+0.297412047 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102627 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22646be610c82a757007123baf456b SystemUUID:ec22646b-e610-c82a-7570-07123baf456b BootID:6e06371b-63f8-4a7d-af21-842337e6b34c Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:b2:6a:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:b2:6a:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:9e:77:ea:db:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 21:10:31.287367 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.287357    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 21:10:31.287536 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.287489    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 21:10:31.290329 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.290304    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 21:10:31.290463 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.290331    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-184.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 21:10:31.291055 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.291046    2584 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 21:10:31.291088 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.291059    2584 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 21:10:31.291088 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.291069    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 21:10:31.291617 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.291607    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 21:10:31.292433 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.292424    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 21:10:31.296356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.296345    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 21:10:31.296467 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.296458    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 21:10:31.296504 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.296495    2584 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 21:10:31.296543 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.296511    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 21:10:31.296543 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.296525    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 21:10:31.296543 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.296533    2584 kubelet.go:398] "Adding apiserver pod source"
Sep 15 21:10:31.296543 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.296544    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 21:10:31.296905 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.296894    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 21:10:31.296938 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.296912    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 21:10:31.299468 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.299454    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 21:10:31.301952 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.301939    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 21:10:31.302018 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.301967    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 21:10:31.303947 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.303936    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 21:10:31.303986 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.303953    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 21:10:31.303986 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.303959    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 21:10:31.303986 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.303965    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 21:10:31.303986 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.303974    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 21:10:31.303986 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.303979    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 21:10:31.304117 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.303992    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 21:10:31.304117 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.303997    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 21:10:31.304117 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.304003    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 21:10:31.304117 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.304015    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 21:10:31.304117 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.304027    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 21:10:31.304117 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.304032    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 21:10:31.305224 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.305213    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 21:10:31.305224 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.305223    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 21:10:31.305396 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.305377    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 21:10:31.305455 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.305377    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-184.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 21:10:31.308618 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.308606    2584 server.go:1268] "Started kubelet"
Sep 15 21:10:31.308756 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.308699    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 21:10:31.308852 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.308774    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 21:10:31.308852 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.308777    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 21:10:31.309384 ip-10-0-139-184 systemd[1]: Started Kubernetes Kubelet.
Sep 15 21:10:31.310094 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.310020    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rm2pd"
Sep 15 21:10:31.310094 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.310106    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 21:10:31.314472 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.314454    2584 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 21:10:31.316100 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.316075    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 21:10:31.317869 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.317853    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rm2pd"
Sep 15 21:10:31.318520 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.317443    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-184.ec2.internal.18d59accca5a9ca5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-184.ec2.internal,UID:ip-10-0-139-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-184.ec2.internal,},FirstTimestamp:2026-09-15 21:10:31.308582053 +0000 UTC m=+0.320744002,LastTimestamp:2026-09-15 21:10:31.308582053 +0000 UTC m=+0.320744002,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-184.ec2.internal,}"
Sep 15 21:10:31.319421 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.319402    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 21:10:31.319828 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.319816    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 21:10:31.319996 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.319979    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-184.ec2.internal\" not found"
Sep 15 21:10:31.320480 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.320392    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 21:10:31.320480 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.320402    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 21:10:31.320480 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.320430    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 21:10:31.320662 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.320552    2584 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 21:10:31.320662 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.320569    2584 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 21:10:31.320754 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.320670    2584 factory.go:55] Registering systemd factory
Sep 15 21:10:31.320754 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.320683    2584 factory.go:223] Registration of the systemd container factory successfully
Sep 15 21:10:31.320853 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.320831    2584 factory.go:153] Registering CRI-O factory
Sep 15 21:10:31.320853 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.320849    2584 factory.go:223] Registration of the crio container factory successfully
Sep 15 21:10:31.320936 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.320893    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 21:10:31.320936 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.320911    2584 factory.go:103] Registering Raw factory
Sep 15 21:10:31.320936 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.320925    2584 manager.go:1196] Started watching for new ooms in manager
Sep 15 21:10:31.321317 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.321299    2584 manager.go:349] Starting recovery of all containers
Sep 15 21:10:31.325104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.325076    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:10:31.328667 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.328645    2584 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-184.ec2.internal\" not found" node="ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.330245 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.330224    2584 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-184.ec2.internal" not found
Sep 15 21:10:31.331773 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.331754    2584 manager.go:354] Recovery completed
Sep 15 21:10:31.336131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.336117    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:10:31.338515 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.338500    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:10:31.338612 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.338527    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:10:31.338612 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.338542    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:10:31.339100 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.339087    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 21:10:31.339173 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.339101    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 21:10:31.339173 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.339120    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 21:10:31.339173 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.339093    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:10:31.339909 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.339893    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:10:31.339989 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.339937    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:10:31.339989 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.339950    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:10:31.347480 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.347466    2584 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-184.ec2.internal" not found
Sep 15 21:10:31.404903 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.404886    2584 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-184.ec2.internal" not found
Sep 15 21:10:31.481329 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.420078    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-184.ec2.internal\" not found"
Sep 15 21:10:31.481329 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.480338    2584 policy_none.go:49] "None policy: Start"
Sep 15 21:10:31.481329 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.480353    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 21:10:31.481329 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.480363    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 21:10:31.482343 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.482332    2584 policy_none.go:47] "Start"
Sep 15 21:10:31.513790 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.513668    2584 manager.go:337] "Starting Device Plugin manager"
Sep 15 21:10:31.513910 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.513821    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 21:10:31.513910 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.513835    2584 server.go:87] "Starting device plugin registration server"
Sep 15 21:10:31.514005 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.513958    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 21:10:31.514093 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.514079    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 21:10:31.514144 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.514098    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 21:10:31.514319 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.514301    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 21:10:31.514405 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.514383    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 21:10:31.514405 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.514393    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 21:10:31.514877 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.514857    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 21:10:31.515017 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.514911    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-184.ec2.internal\" not found"
Sep 15 21:10:31.515422 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.515406    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 21:10:31.515483 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.515456    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 21:10:31.515483 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.515479    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 21:10:31.515568 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.515557    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 21:10:31.517896 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.517877    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:10:31.614418 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.614367    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:10:31.615341 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.615324    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:10:31.615421 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.615353    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:10:31.615421 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.615365    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:10:31.615421 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.615392    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.615602 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.615589    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-184.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal"]
Sep 15 21:10:31.615653 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.615644    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:10:31.616366 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.616351    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:10:31.616459 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.616379    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:10:31.616459 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.616394    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:10:31.618674 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.618662    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:10:31.618791 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.618777    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.618844 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.618811    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:10:31.619338 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.619316    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:10:31.619394 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.619336    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:10:31.619394 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.619343    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:10:31.619394 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.619356    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:10:31.619394 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.619358    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:10:31.619394 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.619366    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:10:31.621479 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.621466    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.621550 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.621491    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:10:31.622198 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.622182    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:10:31.622282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.622207    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:10:31.622282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.622218    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:10:31.622282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.622236    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaddcc5db67cf457ad141690276d44eb-config\") pod \"kube-apiserver-proxy-ip-10-0-139-184.ec2.internal\" (UID: \"eaddcc5db67cf457ad141690276d44eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.622282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.622261    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/419cc084d1ac102a64f9f91ed2a6aa6e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal\" (UID: \"419cc084d1ac102a64f9f91ed2a6aa6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.622282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.622279    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/419cc084d1ac102a64f9f91ed2a6aa6e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal\" (UID: \"419cc084d1ac102a64f9f91ed2a6aa6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.624214 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.624195    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.624295 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.624218    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-184.ec2.internal\": node \"ip-10-0-139-184.ec2.internal\" not found"
Sep 15 21:10:31.640869 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.640848    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-184.ec2.internal\" not found"
Sep 15 21:10:31.649762 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.649744    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-184.ec2.internal\" not found" node="ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.654162 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.654134    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-184.ec2.internal\" not found" node="ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.722629 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.722607    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/419cc084d1ac102a64f9f91ed2a6aa6e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal\" (UID: \"419cc084d1ac102a64f9f91ed2a6aa6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.722709 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.722632    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/419cc084d1ac102a64f9f91ed2a6aa6e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal\" (UID: \"419cc084d1ac102a64f9f91ed2a6aa6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.722709 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.722648    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaddcc5db67cf457ad141690276d44eb-config\") pod \"kube-apiserver-proxy-ip-10-0-139-184.ec2.internal\" (UID: \"eaddcc5db67cf457ad141690276d44eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.722709 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.722683    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaddcc5db67cf457ad141690276d44eb-config\") pod \"kube-apiserver-proxy-ip-10-0-139-184.ec2.internal\" (UID: \"eaddcc5db67cf457ad141690276d44eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.722829 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.722706    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/419cc084d1ac102a64f9f91ed2a6aa6e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal\" (UID: \"419cc084d1ac102a64f9f91ed2a6aa6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.722829 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.722730    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/419cc084d1ac102a64f9f91ed2a6aa6e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal\" (UID: \"419cc084d1ac102a64f9f91ed2a6aa6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.741105 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.741040    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-184.ec2.internal\" not found"
Sep 15 21:10:31.842136 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.842109    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-184.ec2.internal\" not found"
Sep 15 21:10:31.942648 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:31.942630    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-184.ec2.internal\" not found"
Sep 15 21:10:31.952810 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.952794    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-184.ec2.internal"
Sep 15 21:10:31.957868 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:31.957696    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal"
Sep 15 21:10:32.043016 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:32.042963    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-184.ec2.internal\" not found"
Sep 15 21:10:33.293690 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:33.293662    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-184.ec2.internal\" not found"
Sep 15 21:10:33.394128 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:33.394101    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-184.ec2.internal\" not found"
Sep 15 21:10:33.406264 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:33.406248    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 21:10:33.406391 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:33.406362    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 21:10:33.406433 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:33.406412    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 21:10:33.471825 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:33.471778    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 21:05:32 +0000 UTC" deadline="2028-03-29 04:41:25.58056835 +0000 UTC"
Sep 15 21:10:33.471825 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:33.471799    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13447h30m52.108771237s"
Sep 15 21:10:33.495089 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:33.495065    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-184.ec2.internal\" not found"
Sep 15 21:10:33.595536 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:33.595512    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-184.ec2.internal\" not found"
Sep 15 21:10:33.696578 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:33.696550    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-184.ec2.internal\" not found"
Sep 15 21:10:33.778905 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:33.778866    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419cc084d1ac102a64f9f91ed2a6aa6e.slice/crio-d7dac344cf883e9a5ef6001e12e7b4cb51fcd4218fad0620ba03b73dcdf05678 WatchSource:0}: Error finding container d7dac344cf883e9a5ef6001e12e7b4cb51fcd4218fad0620ba03b73dcdf05678: Status 404 returned error can't find the container with id d7dac344cf883e9a5ef6001e12e7b4cb51fcd4218fad0620ba03b73dcdf05678
Sep 15 21:10:33.779252 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:33.779231    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaddcc5db67cf457ad141690276d44eb.slice/crio-5c5a4054515f2fc26713125c891f620aa7a3e8e85689071eb813fac7e6eaf6df WatchSource:0}: Error finding container 5c5a4054515f2fc26713125c891f620aa7a3e8e85689071eb813fac7e6eaf6df: Status 404 returned error can't find the container with id 5c5a4054515f2fc26713125c891f620aa7a3e8e85689071eb813fac7e6eaf6df
Sep 15 21:10:33.783445 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:33.783431    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:10:33.797171 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:33.797152    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-184.ec2.internal\" not found"
Sep 15 21:10:33.897572 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:33.897540    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-184.ec2.internal\" not found"
Sep 15 21:10:33.950408 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:33.950390    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:10:33.985849 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:33.985830    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:10:34.070520 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.070473    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal"
Sep 15 21:10:34.083019 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.083004    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 21:10:34.084028 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.084016    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-184.ec2.internal"
Sep 15 21:10:34.093287 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.093268    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 21:10:34.364975 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.364903    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:10:34.448122 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.448096    2584 apiserver.go:52] "Watching apiserver"
Sep 15 21:10:34.455783 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.455612    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 21:10:34.456775 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.456748    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-rhc4j","openshift-multus/network-metrics-daemon-pt55j","openshift-ovn-kubernetes/ovnkube-node-rrz98","openshift-cluster-node-tuning-operator/tuned-jddxk","openshift-dns/node-resolver-q2rvv","openshift-image-registry/node-ca-rh5pt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal","openshift-multus/multus-9lfnw","openshift-network-diagnostics/network-check-target-9dgmv","openshift-network-operator/iptables-alerter-zk9g4","kube-system/global-pull-secret-syncer-wghbm","kube-system/konnectivity-agent-jrzds","kube-system/kube-apiserver-proxy-ip-10-0-139-184.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"]
Sep 15 21:10:34.459623 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.459511    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zk9g4"
Sep 15 21:10:34.462136 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.461558    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:34.462136 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:34.461642    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:10:34.462136 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.461975    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qnnv5\""
Sep 15 21:10:34.463331 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.463054    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 21:10:34.463331 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.463261    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 21:10:34.466725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.464291    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:10:34.466725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.464901    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.468207 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.467238    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 21:10:34.468207 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.467359    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 21:10:34.468207 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.467668    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.468207 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.467754    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 21:10:34.468207 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.467791    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v5kc5\""
Sep 15 21:10:34.468207 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.468009    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 21:10:34.468655 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.468590    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 21:10:34.470932 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.469425    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 21:10:34.470932 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.470467    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:10:34.470932 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.470633    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q2rvv"
Sep 15 21:10:34.470932 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.470651    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 21:10:34.470932 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.470714    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lv5gd\""
Sep 15 21:10:34.470932 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.470906    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 21:10:34.472913 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.472666    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 21:10:34.473004 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.472974    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 21:10:34.473200 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.473182    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-klvt7\""
Sep 15 21:10:34.473488 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.473429    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rh5pt"
Sep 15 21:10:34.475278 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.475201    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:34.475634 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:34.475575    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:10:34.475776 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.475690    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 21:10:34.475912 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.475896    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sw57n\""
Sep 15 21:10:34.475961 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.475933    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 21:10:34.476028 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.475971    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 21:10:34.478957 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.478628    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.480533 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.480507    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 21:10:34.480833 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.480815    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2v89c\""
Sep 15 21:10:34.481036 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.481017    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 21:10:34.481789 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.481772    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 21:10:34.482889 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.482579    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:34.482889 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:34.482633    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:10:34.482889 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.482721    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvrzb\" (UniqueName: \"kubernetes.io/projected/72452630-f37e-48fb-8243-1b52655dd088-kube-api-access-mvrzb\") pod \"iptables-alerter-zk9g4\" (UID: \"72452630-f37e-48fb-8243-1b52655dd088\") " pod="openshift-network-operator/iptables-alerter-zk9g4"
Sep 15 21:10:34.482889 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.482748    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-node-log\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.482889 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.482774    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e2e853c-6b97-4b1e-9588-7afdc0d95932-ovnkube-config\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.482889 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.482798    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e2e853c-6b97-4b1e-9588-7afdc0d95932-env-overrides\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.482889 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.482822    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e2e853c-6b97-4b1e-9588-7afdc0d95932-ovn-node-metrics-cert\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.482889 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.482848    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-sysctl-conf\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.482889 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.482887    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-var-lib-kubelet\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.483355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.482910    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-run-openvswitch\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.483355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.482931    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-cni-netd\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.483355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.482953    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-modprobe-d\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.483355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.482986    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-kubelet\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.483355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483024    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-run-netns\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.483355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483048    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1ff81ab-8f51-438e-a910-4d6624e8cbbe-serviceca\") pod \"node-ca-rh5pt\" (UID: \"c1ff81ab-8f51-438e-a910-4d6624e8cbbe\") " pod="openshift-image-registry/node-ca-rh5pt"
Sep 15 21:10:34.483355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483070    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-slash\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.483355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483092    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-run-systemd\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.483355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483128    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-var-lib-openvswitch\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.483355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483151    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-run-ovn\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.483355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483190    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-cni-bin\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.483355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483213    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2lz\" (UniqueName: \"kubernetes.io/projected/6e2e853c-6b97-4b1e-9588-7afdc0d95932-kube-api-access-zr2lz\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.483355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483248    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-sys\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.483355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483271    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-host\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.483355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483320    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c27eb1d3-c379-4e71-9694-7cfce96c4a47-hosts-file\") pod \"node-resolver-q2rvv\" (UID: \"c27eb1d3-c379-4e71-9694-7cfce96c4a47\") " pod="openshift-dns/node-resolver-q2rvv"
Sep 15 21:10:34.483355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483344    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2260413-c64c-4707-92f9-ed64418eeda4-kubelet-config\") pod \"global-pull-secret-syncer-wghbm\" (UID: \"a2260413-c64c-4707-92f9-ed64418eeda4\") " pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483368    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-kubernetes\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483408    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92ftx\" (UniqueName: \"kubernetes.io/projected/e0c44d74-0807-47f8-a864-f939b64dfea7-kube-api-access-92ftx\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483432    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvmrr\" (UniqueName: \"kubernetes.io/projected/c27eb1d3-c379-4e71-9694-7cfce96c4a47-kube-api-access-bvmrr\") pod \"node-resolver-q2rvv\" (UID: \"c27eb1d3-c379-4e71-9694-7cfce96c4a47\") " pod="openshift-dns/node-resolver-q2rvv"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483475    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1ff81ab-8f51-438e-a910-4d6624e8cbbe-host\") pod \"node-ca-rh5pt\" (UID: \"c1ff81ab-8f51-438e-a910-4d6624e8cbbe\") " pod="openshift-image-registry/node-ca-rh5pt"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483523    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72452630-f37e-48fb-8243-1b52655dd088-host-slash\") pod \"iptables-alerter-zk9g4\" (UID: \"72452630-f37e-48fb-8243-1b52655dd088\") " pod="openshift-network-operator/iptables-alerter-zk9g4"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483565    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-systemd-units\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483598    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-sysconfig\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483620    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-systemd\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483645    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-lib-modules\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483679    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27eb1d3-c379-4e71-9694-7cfce96c4a47-tmp-dir\") pod \"node-resolver-q2rvv\" (UID: \"c27eb1d3-c379-4e71-9694-7cfce96c4a47\") " pod="openshift-dns/node-resolver-q2rvv"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483693    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483702    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w5w7\" (UniqueName: \"kubernetes.io/projected/c1ff81ab-8f51-438e-a910-4d6624e8cbbe-kube-api-access-6w5w7\") pod \"node-ca-rh5pt\" (UID: \"c1ff81ab-8f51-438e-a910-4d6624e8cbbe\") " pod="openshift-image-registry/node-ca-rh5pt"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483723    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2260413-c64c-4707-92f9-ed64418eeda4-dbus\") pod \"global-pull-secret-syncer-wghbm\" (UID: \"a2260413-c64c-4707-92f9-ed64418eeda4\") " pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483744    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-etc-openvswitch\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483764    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-log-socket\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483784    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-sysctl-d\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.484107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483803    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-run\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.484950 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483825    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72452630-f37e-48fb-8243-1b52655dd088-iptables-alerter-script\") pod \"iptables-alerter-zk9g4\" (UID: \"72452630-f37e-48fb-8243-1b52655dd088\") " pod="openshift-network-operator/iptables-alerter-zk9g4"
Sep 15 21:10:34.484950 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483848    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret\") pod \"global-pull-secret-syncer-wghbm\" (UID: \"a2260413-c64c-4707-92f9-ed64418eeda4\") " pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:34.484950 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483873    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-run-ovn-kubernetes\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.484950 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483896    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.484950 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483920    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e2e853c-6b97-4b1e-9588-7afdc0d95932-ovnkube-script-lib\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.484950 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483945    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-tuned\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.484950 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.483968    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0c44d74-0807-47f8-a864-f939b64dfea7-tmp\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.485275 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.485224    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 21:10:34.486341 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.485766    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r5t9p\""
Sep 15 21:10:34.486341 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.485828    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 21:10:34.486341 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.485909    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 21:10:34.486341 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.485952    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 21:10:34.486341 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.486110    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 21:10:34.486341 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.486151    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.486341 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.486194    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 21:10:34.488075 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.488056    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 21:10:34.488187 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.488165    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jrzds"
Sep 15 21:10:34.488346 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.488328    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-plmfd\""
Sep 15 21:10:34.490091 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.490027    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 21:10:34.490435 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.490368    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 21:10:34.490542 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.490501    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8wf4r\""
Sep 15 21:10:34.503961 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.503929    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ns5q8"
Sep 15 21:10:34.514548 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.514528    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ns5q8"
Sep 15 21:10:34.572537 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.572509    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 21:10:34.584686 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.584655    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2lz\" (UniqueName: \"kubernetes.io/projected/6e2e853c-6b97-4b1e-9588-7afdc0d95932-kube-api-access-zr2lz\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.584799 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.584700    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-registration-dir\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.584799 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.584729    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmdt8\" (UniqueName: \"kubernetes.io/projected/23853078-70a1-4509-a2a9-d1080307d222-kube-api-access-mmdt8\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.584799 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.584780    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2260413-c64c-4707-92f9-ed64418eeda4-kubelet-config\") pod \"global-pull-secret-syncer-wghbm\" (UID: \"a2260413-c64c-4707-92f9-ed64418eeda4\") " pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:34.584976 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.584830    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-kubernetes\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.584976 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.584895    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2260413-c64c-4707-92f9-ed64418eeda4-kubelet-config\") pod \"global-pull-secret-syncer-wghbm\" (UID: \"a2260413-c64c-4707-92f9-ed64418eeda4\") " pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:34.584976 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.584929    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1ff81ab-8f51-438e-a910-4d6624e8cbbe-host\") pod \"node-ca-rh5pt\" (UID: \"c1ff81ab-8f51-438e-a910-4d6624e8cbbe\") " pod="openshift-image-registry/node-ca-rh5pt"
Sep 15 21:10:34.584976 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.584953    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-etc-openvswitch\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.585170 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.584989    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-sysconfig\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.585170 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585012    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-lib-modules\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.585170 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585013    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1ff81ab-8f51-438e-a910-4d6624e8cbbe-host\") pod \"node-ca-rh5pt\" (UID: \"c1ff81ab-8f51-438e-a910-4d6624e8cbbe\") " pod="openshift-image-registry/node-ca-rh5pt"
Sep 15 21:10:34.585170 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585053    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27eb1d3-c379-4e71-9694-7cfce96c4a47-tmp-dir\") pod \"node-resolver-q2rvv\" (UID: \"c27eb1d3-c379-4e71-9694-7cfce96c4a47\") " pod="openshift-dns/node-resolver-q2rvv"
Sep 15 21:10:34.585170 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585059    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-etc-openvswitch\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.585170 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585076    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w5w7\" (UniqueName: \"kubernetes.io/projected/c1ff81ab-8f51-438e-a910-4d6624e8cbbe-kube-api-access-6w5w7\") pod \"node-ca-rh5pt\" (UID: \"c1ff81ab-8f51-438e-a910-4d6624e8cbbe\") " pod="openshift-image-registry/node-ca-rh5pt"
Sep 15 21:10:34.585170 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585100    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-run-ovn-kubernetes\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.585170 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585156    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e2e853c-6b97-4b1e-9588-7afdc0d95932-ovnkube-script-lib\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.585170 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585165    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-kubernetes\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585183    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-multus-socket-dir-parent\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585208    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-sysconfig\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585209    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-run-k8s-cni-cncf-io\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585246    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-sysctl-d\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585269    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-run\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585294    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72452630-f37e-48fb-8243-1b52655dd088-iptables-alerter-script\") pod \"iptables-alerter-zk9g4\" (UID: \"72452630-f37e-48fb-8243-1b52655dd088\") " pod="openshift-network-operator/iptables-alerter-zk9g4"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585320    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585348    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-socket-dir\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585371    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-multus-cni-dir\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585461    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-cnibin\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585478    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-run-multus-certs\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585494    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-node-log\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585508    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e2e853c-6b97-4b1e-9588-7afdc0d95932-ovnkube-config\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585521    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27eb1d3-c379-4e71-9694-7cfce96c4a47-tmp-dir\") pod \"node-resolver-q2rvv\" (UID: \"c27eb1d3-c379-4e71-9694-7cfce96c4a47\") " pod="openshift-dns/node-resolver-q2rvv"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585528    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585556    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-var-lib-cni-multus\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.585630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585579    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-multus-conf-dir\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.586429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585603    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/821589e9-d2dd-49ed-8b13-c9dae33d8747-cni-binary-copy\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.586429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585628    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/821589e9-d2dd-49ed-8b13-c9dae33d8747-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.586429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585654    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-run-openvswitch\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.586429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585749    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-device-dir\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.586429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585782    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/821589e9-d2dd-49ed-8b13-c9dae33d8747-system-cni-dir\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.586429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585824    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf87w\" (UniqueName: \"kubernetes.io/projected/821589e9-d2dd-49ed-8b13-c9dae33d8747-kube-api-access-lf87w\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.586429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585862    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-modprobe-d\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.586429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585886    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-kubelet\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.586429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585923    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/821589e9-d2dd-49ed-8b13-c9dae33d8747-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.586429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585956    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/821589e9-d2dd-49ed-8b13-c9dae33d8747-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.586429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.585982    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1ff81ab-8f51-438e-a910-4d6624e8cbbe-serviceca\") pod \"node-ca-rh5pt\" (UID: \"c1ff81ab-8f51-438e-a910-4d6624e8cbbe\") " pod="openshift-image-registry/node-ca-rh5pt"
Sep 15 21:10:34.586429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586007    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-run-systemd\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.586429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586031    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-var-lib-openvswitch\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.586429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586063    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-run-ovn\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.586429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586088    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c27eb1d3-c379-4e71-9694-7cfce96c4a47-hosts-file\") pod \"node-resolver-q2rvv\" (UID: \"c27eb1d3-c379-4e71-9694-7cfce96c4a47\") " pod="openshift-dns/node-resolver-q2rvv"
Sep 15 21:10:34.586429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586113    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ba752fdf-05e3-4c36-adf6-4f443cf4f2e4-konnectivity-ca\") pod \"konnectivity-agent-jrzds\" (UID: \"ba752fdf-05e3-4c36-adf6-4f443cf4f2e4\") " pod="kube-system/konnectivity-agent-jrzds"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586153    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92ftx\" (UniqueName: \"kubernetes.io/projected/e0c44d74-0807-47f8-a864-f939b64dfea7-kube-api-access-92ftx\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586178    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvmrr\" (UniqueName: \"kubernetes.io/projected/c27eb1d3-c379-4e71-9694-7cfce96c4a47-kube-api-access-bvmrr\") pod \"node-resolver-q2rvv\" (UID: \"c27eb1d3-c379-4e71-9694-7cfce96c4a47\") " pod="openshift-dns/node-resolver-q2rvv"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586202    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72452630-f37e-48fb-8243-1b52655dd088-host-slash\") pod \"iptables-alerter-zk9g4\" (UID: \"72452630-f37e-48fb-8243-1b52655dd088\") " pod="openshift-network-operator/iptables-alerter-zk9g4"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586225    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-systemd-units\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586248    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-log-socket\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586272    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e2e853c-6b97-4b1e-9588-7afdc0d95932-env-overrides\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586292    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e2e853c-6b97-4b1e-9588-7afdc0d95932-ovnkube-script-lib\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586425    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e2e853c-6b97-4b1e-9588-7afdc0d95932-ovn-node-metrics-cert\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586433    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c27eb1d3-c379-4e71-9694-7cfce96c4a47-hosts-file\") pod \"node-resolver-q2rvv\" (UID: \"c27eb1d3-c379-4e71-9694-7cfce96c4a47\") " pod="openshift-dns/node-resolver-q2rvv"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586456    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-var-lib-kubelet\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586485    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-systemd\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586519    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2260413-c64c-4707-92f9-ed64418eeda4-dbus\") pod \"global-pull-secret-syncer-wghbm\" (UID: \"a2260413-c64c-4707-92f9-ed64418eeda4\") " pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586544    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs\") pod \"network-metrics-daemon-pt55j\" (UID: \"d8ed8d4e-9c80-4109-a61a-440ac53e69e6\") " pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586568    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/821589e9-d2dd-49ed-8b13-c9dae33d8747-cnibin\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586595    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret\") pod \"global-pull-secret-syncer-wghbm\" (UID: \"a2260413-c64c-4707-92f9-ed64418eeda4\") " pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586620    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-var-lib-cni-bin\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.587178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586644    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-tuned\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586670    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0c44d74-0807-47f8-a864-f939b64dfea7-tmp\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586678    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-modprobe-d\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586696    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvrzb\" (UniqueName: \"kubernetes.io/projected/72452630-f37e-48fb-8243-1b52655dd088-kube-api-access-mvrzb\") pod \"iptables-alerter-zk9g4\" (UID: \"72452630-f37e-48fb-8243-1b52655dd088\") " pod="openshift-network-operator/iptables-alerter-zk9g4"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586716    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-kubelet\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586723    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1065273-8280-40da-8d5f-9fa0223a961d-cni-binary-copy\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586754    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b1065273-8280-40da-8d5f-9fa0223a961d-multus-daemon-config\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586781    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xng6l\" (UniqueName: \"kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l\") pod \"network-check-target-9dgmv\" (UID: \"d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6\") " pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586784    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-run-ovn-kubernetes\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586809    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-sysctl-conf\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586834    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-var-lib-kubelet\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586859    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-cni-netd\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586883    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-system-cni-dir\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586896    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-lib-modules\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586910    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24gzd\" (UniqueName: \"kubernetes.io/projected/b1065273-8280-40da-8d5f-9fa0223a961d-kube-api-access-24gzd\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586946    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-run-netns\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586971    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-etc-selinux\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.588051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.586997    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-sys-fs\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587022    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chbvk\" (UniqueName: \"kubernetes.io/projected/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-kube-api-access-chbvk\") pod \"network-metrics-daemon-pt55j\" (UID: \"d8ed8d4e-9c80-4109-a61a-440ac53e69e6\") " pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587049    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ba752fdf-05e3-4c36-adf6-4f443cf4f2e4-agent-certs\") pod \"konnectivity-agent-jrzds\" (UID: \"ba752fdf-05e3-4c36-adf6-4f443cf4f2e4\") " pod="kube-system/konnectivity-agent-jrzds"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587075    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-slash\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587106    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-cni-bin\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587336    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-run-netns\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587364    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-hostroot\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587409    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-sys\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587436    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-host\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587461    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-etc-kubernetes\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587489    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-cni-bin\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587504    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-node-log\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587857    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-run-systemd\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587897    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72452630-f37e-48fb-8243-1b52655dd088-host-slash\") pod \"iptables-alerter-zk9g4\" (UID: \"72452630-f37e-48fb-8243-1b52655dd088\") " pod="openshift-network-operator/iptables-alerter-zk9g4"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587911    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e2e853c-6b97-4b1e-9588-7afdc0d95932-env-overrides\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587929    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e2e853c-6b97-4b1e-9588-7afdc0d95932-ovnkube-config\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587938    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-systemd-units\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587928    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 21:10:34.588803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587965    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-run-openvswitch\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587989    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.588002    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-run-netns\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.588016    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-var-lib-kubelet\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.588032    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-log-socket\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.588044    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-systemd\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.588048    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-cni-netd\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.588082    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2260413-c64c-4707-92f9-ed64418eeda4-dbus\") pod \"global-pull-secret-syncer-wghbm\" (UID: \"a2260413-c64c-4707-92f9-ed64418eeda4\") " pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.588087    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-run-ovn\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.588125    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-host-slash\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.588209    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-sysctl-d\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.588225    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-sysctl-conf\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.587366    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1ff81ab-8f51-438e-a910-4d6624e8cbbe-serviceca\") pod \"node-ca-rh5pt\" (UID: \"c1ff81ab-8f51-438e-a910-4d6624e8cbbe\") " pod="openshift-image-registry/node-ca-rh5pt"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.588264    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-run\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.588295    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e2e853c-6b97-4b1e-9588-7afdc0d95932-var-lib-openvswitch\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.588316    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-sys\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.588357    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0c44d74-0807-47f8-a864-f939b64dfea7-host\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:34.588629    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:34.589616 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:34.588696    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret podName:a2260413-c64c-4707-92f9-ed64418eeda4 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:35.088676924 +0000 UTC m=+2.950610727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret") pod "global-pull-secret-syncer-wghbm" (UID: "a2260413-c64c-4707-92f9-ed64418eeda4") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:34.590459 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.588918    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72452630-f37e-48fb-8243-1b52655dd088-iptables-alerter-script\") pod \"iptables-alerter-zk9g4\" (UID: \"72452630-f37e-48fb-8243-1b52655dd088\") " pod="openshift-network-operator/iptables-alerter-zk9g4"
Sep 15 21:10:34.591920 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.591878    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e0c44d74-0807-47f8-a864-f939b64dfea7-etc-tuned\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.592010 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.591962    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0c44d74-0807-47f8-a864-f939b64dfea7-tmp\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.594081 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.593602    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e2e853c-6b97-4b1e-9588-7afdc0d95932-ovn-node-metrics-cert\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.595198 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.595049    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w5w7\" (UniqueName: \"kubernetes.io/projected/c1ff81ab-8f51-438e-a910-4d6624e8cbbe-kube-api-access-6w5w7\") pod \"node-ca-rh5pt\" (UID: \"c1ff81ab-8f51-438e-a910-4d6624e8cbbe\") " pod="openshift-image-registry/node-ca-rh5pt"
Sep 15 21:10:34.595504 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.595482    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2lz\" (UniqueName: \"kubernetes.io/projected/6e2e853c-6b97-4b1e-9588-7afdc0d95932-kube-api-access-zr2lz\") pod \"ovnkube-node-rrz98\" (UID: \"6e2e853c-6b97-4b1e-9588-7afdc0d95932\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.595604 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.595510    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92ftx\" (UniqueName: \"kubernetes.io/projected/e0c44d74-0807-47f8-a864-f939b64dfea7-kube-api-access-92ftx\") pod \"tuned-jddxk\" (UID: \"e0c44d74-0807-47f8-a864-f939b64dfea7\") " pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.595775 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.595728    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvmrr\" (UniqueName: \"kubernetes.io/projected/c27eb1d3-c379-4e71-9694-7cfce96c4a47-kube-api-access-bvmrr\") pod \"node-resolver-q2rvv\" (UID: \"c27eb1d3-c379-4e71-9694-7cfce96c4a47\") " pod="openshift-dns/node-resolver-q2rvv"
Sep 15 21:10:34.600455 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.600350    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvrzb\" (UniqueName: \"kubernetes.io/projected/72452630-f37e-48fb-8243-1b52655dd088-kube-api-access-mvrzb\") pod \"iptables-alerter-zk9g4\" (UID: \"72452630-f37e-48fb-8243-1b52655dd088\") " pod="openshift-network-operator/iptables-alerter-zk9g4"
Sep 15 21:10:34.671235 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.671143    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal" event={"ID":"419cc084d1ac102a64f9f91ed2a6aa6e","Type":"ContainerStarted","Data":"d7dac344cf883e9a5ef6001e12e7b4cb51fcd4218fad0620ba03b73dcdf05678"}
Sep 15 21:10:34.672558 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.672531    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-184.ec2.internal" event={"ID":"eaddcc5db67cf457ad141690276d44eb","Type":"ContainerStarted","Data":"5c5a4054515f2fc26713125c891f620aa7a3e8e85689071eb813fac7e6eaf6df"}
Sep 15 21:10:34.687836 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.687776    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-etc-kubernetes\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.687836 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.687827    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-registration-dir\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.688001 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.687854    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmdt8\" (UniqueName: \"kubernetes.io/projected/23853078-70a1-4509-a2a9-d1080307d222-kube-api-access-mmdt8\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.688001 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.687888    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-multus-socket-dir-parent\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.688001 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.687909    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-run-k8s-cni-cncf-io\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.688001 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.687926    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-socket-dir\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.688001 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.687940    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-multus-cni-dir\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.688001 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.687886    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-etc-kubernetes\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.688001 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.687955    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-cnibin\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.688001 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.687976    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-run-multus-certs\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.688001 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688001    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688007    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-multus-socket-dir-parent\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688019    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-var-lib-cni-multus\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.687937    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-registration-dir\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688055    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-run-k8s-cni-cncf-io\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688084    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-multus-conf-dir\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688097    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-cnibin\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688112    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/821589e9-d2dd-49ed-8b13-c9dae33d8747-cni-binary-copy\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688139    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/821589e9-d2dd-49ed-8b13-c9dae33d8747-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688164    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-device-dir\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688187    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/821589e9-d2dd-49ed-8b13-c9dae33d8747-system-cni-dir\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688211    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf87w\" (UniqueName: \"kubernetes.io/projected/821589e9-d2dd-49ed-8b13-c9dae33d8747-kube-api-access-lf87w\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688236    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/821589e9-d2dd-49ed-8b13-c9dae33d8747-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688258    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/821589e9-d2dd-49ed-8b13-c9dae33d8747-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688288    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ba752fdf-05e3-4c36-adf6-4f443cf4f2e4-konnectivity-ca\") pod \"konnectivity-agent-jrzds\" (UID: \"ba752fdf-05e3-4c36-adf6-4f443cf4f2e4\") " pod="kube-system/konnectivity-agent-jrzds"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688318    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-var-lib-kubelet\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688343    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs\") pod \"network-metrics-daemon-pt55j\" (UID: \"d8ed8d4e-9c80-4109-a61a-440ac53e69e6\") " pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:34.688597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688358    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688367    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/821589e9-d2dd-49ed-8b13-c9dae33d8747-cnibin\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688407    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-var-lib-cni-multus\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688424    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-var-lib-cni-bin\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688451    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1065273-8280-40da-8d5f-9fa0223a961d-cni-binary-copy\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688474    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b1065273-8280-40da-8d5f-9fa0223a961d-multus-daemon-config\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688498    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xng6l\" (UniqueName: \"kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l\") pod \"network-check-target-9dgmv\" (UID: \"d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6\") " pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688533    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-system-cni-dir\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688557    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24gzd\" (UniqueName: \"kubernetes.io/projected/b1065273-8280-40da-8d5f-9fa0223a961d-kube-api-access-24gzd\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688583    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-etc-selinux\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688606    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-sys-fs\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688629    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chbvk\" (UniqueName: \"kubernetes.io/projected/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-kube-api-access-chbvk\") pod \"network-metrics-daemon-pt55j\" (UID: \"d8ed8d4e-9c80-4109-a61a-440ac53e69e6\") " pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688654    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ba752fdf-05e3-4c36-adf6-4f443cf4f2e4-agent-certs\") pod \"konnectivity-agent-jrzds\" (UID: \"ba752fdf-05e3-4c36-adf6-4f443cf4f2e4\") " pod="kube-system/konnectivity-agent-jrzds"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688679    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-run-netns\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688702    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-hostroot\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688724    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-socket-dir\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688775    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-hostroot\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688799    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-multus-cni-dir\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.689566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688839    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-multus-conf-dir\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688884    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-device-dir\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688925    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/821589e9-d2dd-49ed-8b13-c9dae33d8747-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.689106    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/821589e9-d2dd-49ed-8b13-c9dae33d8747-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.689159    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/821589e9-d2dd-49ed-8b13-c9dae33d8747-system-cni-dir\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.688321    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-run-multus-certs\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.689364    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/821589e9-d2dd-49ed-8b13-c9dae33d8747-cnibin\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.689390    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-var-lib-cni-bin\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.689397    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b1065273-8280-40da-8d5f-9fa0223a961d-multus-daemon-config\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.689448    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-var-lib-kubelet\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.689485    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-sys-fs\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.689535    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-system-cni-dir\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.689626    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/821589e9-d2dd-49ed-8b13-c9dae33d8747-cni-binary-copy\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.689640    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/821589e9-d2dd-49ed-8b13-c9dae33d8747-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.689687    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1065273-8280-40da-8d5f-9fa0223a961d-host-run-netns\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.689712    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23853078-70a1-4509-a2a9-d1080307d222-etc-selinux\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:34.689788    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:34.690586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.689801    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1065273-8280-40da-8d5f-9fa0223a961d-cni-binary-copy\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.691449 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:34.689843    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs podName:d8ed8d4e-9c80-4109-a61a-440ac53e69e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:35.189825937 +0000 UTC m=+3.051759721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs") pod "network-metrics-daemon-pt55j" (UID: "d8ed8d4e-9c80-4109-a61a-440ac53e69e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:34.691449 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.689899    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ba752fdf-05e3-4c36-adf6-4f443cf4f2e4-konnectivity-ca\") pod \"konnectivity-agent-jrzds\" (UID: \"ba752fdf-05e3-4c36-adf6-4f443cf4f2e4\") " pod="kube-system/konnectivity-agent-jrzds"
Sep 15 21:10:34.692929 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.692906    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ba752fdf-05e3-4c36-adf6-4f443cf4f2e4-agent-certs\") pod \"konnectivity-agent-jrzds\" (UID: \"ba752fdf-05e3-4c36-adf6-4f443cf4f2e4\") " pod="kube-system/konnectivity-agent-jrzds"
Sep 15 21:10:34.694361 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:34.694340    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:10:34.694449 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:34.694367    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:10:34.694449 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:34.694398    2584 projected.go:196] Error preparing data for projected volume kube-api-access-xng6l for pod openshift-network-diagnostics/network-check-target-9dgmv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:34.694550 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:34.694461    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l podName:d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:35.194446815 +0000 UTC m=+3.056380617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xng6l" (UniqueName: "kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l") pod "network-check-target-9dgmv" (UID: "d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:34.696958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.696935    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf87w\" (UniqueName: \"kubernetes.io/projected/821589e9-d2dd-49ed-8b13-c9dae33d8747-kube-api-access-lf87w\") pod \"multus-additional-cni-plugins-rhc4j\" (UID: \"821589e9-d2dd-49ed-8b13-c9dae33d8747\") " pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.697424 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.697373    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmdt8\" (UniqueName: \"kubernetes.io/projected/23853078-70a1-4509-a2a9-d1080307d222-kube-api-access-mmdt8\") pod \"aws-ebs-csi-driver-node-jwrxd\" (UID: \"23853078-70a1-4509-a2a9-d1080307d222\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.700248 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.700222    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24gzd\" (UniqueName: \"kubernetes.io/projected/b1065273-8280-40da-8d5f-9fa0223a961d-kube-api-access-24gzd\") pod \"multus-9lfnw\" (UID: \"b1065273-8280-40da-8d5f-9fa0223a961d\") " pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.700921 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.700899    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chbvk\" (UniqueName: \"kubernetes.io/projected/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-kube-api-access-chbvk\") pod \"network-metrics-daemon-pt55j\" (UID: \"d8ed8d4e-9c80-4109-a61a-440ac53e69e6\") " pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:34.776909 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.776877    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zk9g4"
Sep 15 21:10:34.787213 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.786878    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:34.788087 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:34.788063    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72452630_f37e_48fb_8243_1b52655dd088.slice/crio-45b7e1b8f6b2ba01dd6ef0b6a5891b93d88605767d81b0ca37da3529a818416f WatchSource:0}: Error finding container 45b7e1b8f6b2ba01dd6ef0b6a5891b93d88605767d81b0ca37da3529a818416f: Status 404 returned error can't find the container with id 45b7e1b8f6b2ba01dd6ef0b6a5891b93d88605767d81b0ca37da3529a818416f
Sep 15 21:10:34.789974 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.789954    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:10:34.796463 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.796444    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jddxk"
Sep 15 21:10:34.802115 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.802098    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q2rvv"
Sep 15 21:10:34.810523 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.810496    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rh5pt"
Sep 15 21:10:34.820105 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.820085    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd"
Sep 15 21:10:34.830719 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.830702    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rhc4j"
Sep 15 21:10:34.840221 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.840201    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9lfnw"
Sep 15 21:10:34.845671 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:34.845654    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jrzds"
Sep 15 21:10:35.090420 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:35.090337    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret\") pod \"global-pull-secret-syncer-wghbm\" (UID: \"a2260413-c64c-4707-92f9-ed64418eeda4\") " pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:35.090547 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:35.090472    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:35.090547 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:35.090530    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret podName:a2260413-c64c-4707-92f9-ed64418eeda4 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:36.090514989 +0000 UTC m=+3.952448772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret") pod "global-pull-secret-syncer-wghbm" (UID: "a2260413-c64c-4707-92f9-ed64418eeda4") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:35.190949 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:35.190910    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs\") pod \"network-metrics-daemon-pt55j\" (UID: \"d8ed8d4e-9c80-4109-a61a-440ac53e69e6\") " pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:35.191119 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:35.191085    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:35.191186 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:35.191163    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs podName:d8ed8d4e-9c80-4109-a61a-440ac53e69e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:36.191142832 +0000 UTC m=+4.053076624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs") pod "network-metrics-daemon-pt55j" (UID: "d8ed8d4e-9c80-4109-a61a-440ac53e69e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:35.292066 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:35.292035    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xng6l\" (UniqueName: \"kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l\") pod \"network-check-target-9dgmv\" (UID: \"d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6\") " pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:35.292251 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:35.292215    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:10:35.292251 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:35.292237    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:10:35.292251 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:35.292247    2584 projected.go:196] Error preparing data for projected volume kube-api-access-xng6l for pod openshift-network-diagnostics/network-check-target-9dgmv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:35.292399 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:35.292304    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l podName:d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:36.292290137 +0000 UTC m=+4.154223920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xng6l" (UniqueName: "kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l") pod "network-check-target-9dgmv" (UID: "d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:35.518294 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:35.518185    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:05:34 +0000 UTC" deadline="2028-05-18 06:43:51.311569149 +0000 UTC"
Sep 15 21:10:35.518294 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:35.518248    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14649h33m15.793324487s"
Sep 15 21:10:35.666279 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:35.666251    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:35.666450 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:35.666361    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:10:35.675948 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:35.675915    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zk9g4" event={"ID":"72452630-f37e-48fb-8243-1b52655dd088","Type":"ContainerStarted","Data":"45b7e1b8f6b2ba01dd6ef0b6a5891b93d88605767d81b0ca37da3529a818416f"}
Sep 15 21:10:35.814855 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:35.814808    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ff81ab_8f51_438e_a910_4d6624e8cbbe.slice/crio-10dde5233afa987d30f1585092ee8f4f11e949ba802d75dc01cb69d14645b036 WatchSource:0}: Error finding container 10dde5233afa987d30f1585092ee8f4f11e949ba802d75dc01cb69d14645b036: Status 404 returned error can't find the container with id 10dde5233afa987d30f1585092ee8f4f11e949ba802d75dc01cb69d14645b036
Sep 15 21:10:35.839335 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:35.839293    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e2e853c_6b97_4b1e_9588_7afdc0d95932.slice/crio-1148273c2271423ad4c649a888b6c2c7a55f1a03c158e9e3f5b782fdd21099a8 WatchSource:0}: Error finding container 1148273c2271423ad4c649a888b6c2c7a55f1a03c158e9e3f5b782fdd21099a8: Status 404 returned error can't find the container with id 1148273c2271423ad4c649a888b6c2c7a55f1a03c158e9e3f5b782fdd21099a8
Sep 15 21:10:35.840644 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:35.840620    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1065273_8280_40da_8d5f_9fa0223a961d.slice/crio-50c4aab66e3d3e8c1ce0302960d090d46c513cdea53165424c971924f964f14a WatchSource:0}: Error finding container 50c4aab66e3d3e8c1ce0302960d090d46c513cdea53165424c971924f964f14a: Status 404 returned error can't find the container with id 50c4aab66e3d3e8c1ce0302960d090d46c513cdea53165424c971924f964f14a
Sep 15 21:10:35.848562 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:10:35.848503    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c44d74_0807_47f8_a864_f939b64dfea7.slice/crio-bc1bf3da67432fa42fe1b1976e85fc9d9d44b4362b2cfecdc9fe6eeca823f22c WatchSource:0}: Error finding container bc1bf3da67432fa42fe1b1976e85fc9d9d44b4362b2cfecdc9fe6eeca823f22c: Status 404 returned error can't find the container with id bc1bf3da67432fa42fe1b1976e85fc9d9d44b4362b2cfecdc9fe6eeca823f22c
Sep 15 21:10:36.097521 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.097328    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret\") pod \"global-pull-secret-syncer-wghbm\" (UID: \"a2260413-c64c-4707-92f9-ed64418eeda4\") " pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:36.097617 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:36.097490    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:36.097673 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:36.097668    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret podName:a2260413-c64c-4707-92f9-ed64418eeda4 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:38.097647153 +0000 UTC m=+5.959580949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret") pod "global-pull-secret-syncer-wghbm" (UID: "a2260413-c64c-4707-92f9-ed64418eeda4") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:36.198877 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.198840    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs\") pod \"network-metrics-daemon-pt55j\" (UID: \"d8ed8d4e-9c80-4109-a61a-440ac53e69e6\") " pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:36.199038 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:36.198967    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:36.199111 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:36.199048    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs podName:d8ed8d4e-9c80-4109-a61a-440ac53e69e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:38.199027637 +0000 UTC m=+6.060961426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs") pod "network-metrics-daemon-pt55j" (UID: "d8ed8d4e-9c80-4109-a61a-440ac53e69e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:36.299541 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.299480    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xng6l\" (UniqueName: \"kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l\") pod \"network-check-target-9dgmv\" (UID: \"d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6\") " pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:36.299736 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:36.299713    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:10:36.299736 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:36.299738    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:10:36.299951 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:36.299751    2584 projected.go:196] Error preparing data for projected volume kube-api-access-xng6l for pod openshift-network-diagnostics/network-check-target-9dgmv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:36.299951 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:36.299808    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l podName:d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:38.299790066 +0000 UTC m=+6.161723856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xng6l" (UniqueName: "kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l") pod "network-check-target-9dgmv" (UID: "d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:36.518485 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.518359    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:05:34 +0000 UTC" deadline="2028-05-04 11:32:58.604558002 +0000 UTC"
Sep 15 21:10:36.518485 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.518433    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14318h22m22.086130118s"
Sep 15 21:10:36.667091 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.667064    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:36.667252 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:36.667183    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:10:36.667709 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.667687    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:36.667833 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:36.667799    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:10:36.680538 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.680499    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-184.ec2.internal" event={"ID":"eaddcc5db67cf457ad141690276d44eb","Type":"ContainerStarted","Data":"3888167350c4f8f40d9b693f81f475cecf9e073bd9378cb4826ee0ee4a3cbe5e"}
Sep 15 21:10:36.681971 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.681947    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd" event={"ID":"23853078-70a1-4509-a2a9-d1080307d222","Type":"ContainerStarted","Data":"70b86b15b121bf020e5ac922c96b0ed1a133aa75bca33deb855bc3188525e662"}
Sep 15 21:10:36.683924 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.683898    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jrzds" event={"ID":"ba752fdf-05e3-4c36-adf6-4f443cf4f2e4","Type":"ContainerStarted","Data":"30b713094357adbb6bf9d0d76643251a0e58c51bdf13ebb3ef1d674dbaabd951"}
Sep 15 21:10:36.685907 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.685885    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jddxk" event={"ID":"e0c44d74-0807-47f8-a864-f939b64dfea7","Type":"ContainerStarted","Data":"bc1bf3da67432fa42fe1b1976e85fc9d9d44b4362b2cfecdc9fe6eeca823f22c"}
Sep 15 21:10:36.687181 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.687159    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q2rvv" event={"ID":"c27eb1d3-c379-4e71-9694-7cfce96c4a47","Type":"ContainerStarted","Data":"a7396d231ff0e4d6df5faf3e50ebf6661402dd438693353fd36ee585aaebb7c5"}
Sep 15 21:10:36.690774 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.690752    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rhc4j" event={"ID":"821589e9-d2dd-49ed-8b13-c9dae33d8747","Type":"ContainerStarted","Data":"59a07923705d84fc6a01fe1213593ade64db93e4ee0d94cdfe3b106818c2a4a1"}
Sep 15 21:10:36.693262 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.693237    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rh5pt" event={"ID":"c1ff81ab-8f51-438e-a910-4d6624e8cbbe","Type":"ContainerStarted","Data":"10dde5233afa987d30f1585092ee8f4f11e949ba802d75dc01cb69d14645b036"}
Sep 15 21:10:36.697184 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.696552    2584 generic.go:356] "Generic (PLEG): container finished" podID="419cc084d1ac102a64f9f91ed2a6aa6e" containerID="4ae30d98e48d922433b07b66369a341e1fef268aa8d1ea2c8372400924085d71" exitCode=0
Sep 15 21:10:36.697184 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.696632    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal" event={"ID":"419cc084d1ac102a64f9f91ed2a6aa6e","Type":"ContainerDied","Data":"4ae30d98e48d922433b07b66369a341e1fef268aa8d1ea2c8372400924085d71"}
Sep 15 21:10:36.699268 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.699217    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9lfnw" event={"ID":"b1065273-8280-40da-8d5f-9fa0223a961d","Type":"ContainerStarted","Data":"50c4aab66e3d3e8c1ce0302960d090d46c513cdea53165424c971924f964f14a"}
Sep 15 21:10:36.701283 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.701261    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98" event={"ID":"6e2e853c-6b97-4b1e-9588-7afdc0d95932","Type":"ContainerStarted","Data":"1148273c2271423ad4c649a888b6c2c7a55f1a03c158e9e3f5b782fdd21099a8"}
Sep 15 21:10:36.711968 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:36.711922    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-184.ec2.internal" podStartSLOduration=2.7119048919999997 podStartE2EDuration="2.711904892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:36.694403672 +0000 UTC m=+4.556337495" watchObservedRunningTime="2026-09-15 21:10:36.711904892 +0000 UTC m=+4.573838698"
Sep 15 21:10:37.666125 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:37.666100    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:37.666774 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:37.666221    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:10:37.707476 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:37.707430    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal" event={"ID":"419cc084d1ac102a64f9f91ed2a6aa6e","Type":"ContainerStarted","Data":"4ecdac07ea972e9bd8bae73123427b73ea71b7d507dac39f21cd04c80b96cd93"}
Sep 15 21:10:38.115472 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:38.115432    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret\") pod \"global-pull-secret-syncer-wghbm\" (UID: \"a2260413-c64c-4707-92f9-ed64418eeda4\") " pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:38.115655 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:38.115583    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:38.115722 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:38.115656    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret podName:a2260413-c64c-4707-92f9-ed64418eeda4 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:42.115635565 +0000 UTC m=+9.977569362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret") pod "global-pull-secret-syncer-wghbm" (UID: "a2260413-c64c-4707-92f9-ed64418eeda4") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:38.215844 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:38.215808    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs\") pod \"network-metrics-daemon-pt55j\" (UID: \"d8ed8d4e-9c80-4109-a61a-440ac53e69e6\") " pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:38.216037 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:38.216010    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:38.216102 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:38.216080    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs podName:d8ed8d4e-9c80-4109-a61a-440ac53e69e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:42.216061104 +0000 UTC m=+10.077994908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs") pod "network-metrics-daemon-pt55j" (UID: "d8ed8d4e-9c80-4109-a61a-440ac53e69e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:38.316291 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:38.316205    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xng6l\" (UniqueName: \"kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l\") pod \"network-check-target-9dgmv\" (UID: \"d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6\") " pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:38.316483 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:38.316341    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:10:38.316483 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:38.316360    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:10:38.316483 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:38.316372    2584 projected.go:196] Error preparing data for projected volume kube-api-access-xng6l for pod openshift-network-diagnostics/network-check-target-9dgmv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:38.316483 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:38.316442    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l podName:d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:42.316425564 +0000 UTC m=+10.178359353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xng6l" (UniqueName: "kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l") pod "network-check-target-9dgmv" (UID: "d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:38.667143 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:38.666812    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:38.667143 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:38.666940    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:10:38.667654 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:38.666812    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:38.667654 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:38.667351    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:10:39.666716 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:39.666321    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:39.666716 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:39.666444    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:10:40.666984 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:40.666588    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:40.666984 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:40.666709    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:10:40.666984 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:40.666808    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:40.666984 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:40.666925    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:10:41.667201 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:41.666720    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:41.667201 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:41.666858    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:10:42.149229 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:42.149131    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret\") pod \"global-pull-secret-syncer-wghbm\" (UID: \"a2260413-c64c-4707-92f9-ed64418eeda4\") " pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:42.149394 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:42.149233    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:42.149394 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:42.149302    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret podName:a2260413-c64c-4707-92f9-ed64418eeda4 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:50.149282887 +0000 UTC m=+18.011216681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret") pod "global-pull-secret-syncer-wghbm" (UID: "a2260413-c64c-4707-92f9-ed64418eeda4") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:42.250292 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:42.250246    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs\") pod \"network-metrics-daemon-pt55j\" (UID: \"d8ed8d4e-9c80-4109-a61a-440ac53e69e6\") " pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:42.250496 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:42.250440    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:42.250566 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:42.250520    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs podName:d8ed8d4e-9c80-4109-a61a-440ac53e69e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:50.250500359 +0000 UTC m=+18.112434156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs") pod "network-metrics-daemon-pt55j" (UID: "d8ed8d4e-9c80-4109-a61a-440ac53e69e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:42.351068 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:42.351032    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xng6l\" (UniqueName: \"kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l\") pod \"network-check-target-9dgmv\" (UID: \"d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6\") " pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:42.351242 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:42.351206    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:10:42.351242 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:42.351226    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:10:42.351242 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:42.351239    2584 projected.go:196] Error preparing data for projected volume kube-api-access-xng6l for pod openshift-network-diagnostics/network-check-target-9dgmv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:42.351417 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:42.351300    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l podName:d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:50.351280073 +0000 UTC m=+18.213213879 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xng6l" (UniqueName: "kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l") pod "network-check-target-9dgmv" (UID: "d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:42.666972 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:42.666942    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:42.667161 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:42.667050    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:10:42.667227 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:42.667166    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:42.667591 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:42.667277    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:10:43.666485 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:43.666445    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:43.666648 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:43.666596    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:10:44.666498 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:44.666463    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:44.666498 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:44.666496    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:44.667070 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:44.666601    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:10:44.667070 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:44.666709    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:10:45.666295 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:45.666264    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:45.666493 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:45.666412    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:10:46.666752 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:46.666717    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:46.667129 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:46.666717    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:46.667129 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:46.666853    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:10:46.667129 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:46.666910    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:10:47.666860 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:47.666835    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:47.667305 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:47.666957    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:10:48.666508 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:48.666472    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:48.666679 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:48.666610    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:10:48.666679 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:48.666664    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:48.666795 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:48.666771    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:10:49.666409 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:49.666363    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:49.666824 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:49.666489    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:10:50.209718 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:50.209682    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret\") pod \"global-pull-secret-syncer-wghbm\" (UID: \"a2260413-c64c-4707-92f9-ed64418eeda4\") " pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:50.209885 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:50.209829    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:50.209933 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:50.209886    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret podName:a2260413-c64c-4707-92f9-ed64418eeda4 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:06.209870401 +0000 UTC m=+34.071804186 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret") pod "global-pull-secret-syncer-wghbm" (UID: "a2260413-c64c-4707-92f9-ed64418eeda4") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:10:50.311000 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:50.310965    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs\") pod \"network-metrics-daemon-pt55j\" (UID: \"d8ed8d4e-9c80-4109-a61a-440ac53e69e6\") " pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:50.311151 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:50.311083    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:50.311151 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:50.311137    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs podName:d8ed8d4e-9c80-4109-a61a-440ac53e69e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:06.311120133 +0000 UTC m=+34.173053922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs") pod "network-metrics-daemon-pt55j" (UID: "d8ed8d4e-9c80-4109-a61a-440ac53e69e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:10:50.412182 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:50.412144    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xng6l\" (UniqueName: \"kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l\") pod \"network-check-target-9dgmv\" (UID: \"d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6\") " pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:50.412365 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:50.412288    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:10:50.412365 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:50.412304    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:10:50.412365 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:50.412313    2584 projected.go:196] Error preparing data for projected volume kube-api-access-xng6l for pod openshift-network-diagnostics/network-check-target-9dgmv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:50.412365 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:50.412365    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l podName:d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:06.412346043 +0000 UTC m=+34.274279828 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xng6l" (UniqueName: "kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l") pod "network-check-target-9dgmv" (UID: "d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:10:50.666416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:50.666317    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:50.666416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:50.666346    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:50.666901 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:50.666465    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:10:50.666901 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:50.666597    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:10:51.666796 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:51.666750    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:51.667232 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:51.666883    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:10:52.667323 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.666933    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:52.667900 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.667019    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:52.667900 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:52.667440    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:10:52.667900 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:52.667547    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:10:52.737329 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.737298    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9lfnw" event={"ID":"b1065273-8280-40da-8d5f-9fa0223a961d","Type":"ContainerStarted","Data":"151f0086e39b5c53a1e6fa1f7918830ca668355bd69b823734a4dffe710a47d2"}
Sep 15 21:10:52.740423 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.740395    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98" event={"ID":"6e2e853c-6b97-4b1e-9588-7afdc0d95932","Type":"ContainerStarted","Data":"27b5f6e4aa92eb4017dcf6c512bf22c4e21682385d4085fe95880df209deafb7"}
Sep 15 21:10:52.740537 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.740455    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98" event={"ID":"6e2e853c-6b97-4b1e-9588-7afdc0d95932","Type":"ContainerStarted","Data":"c6f9e86be224ba01f5e9f36ec7cf6537065047105f0f41b168ea63696399cd2a"}
Sep 15 21:10:52.740537 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.740497    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98" event={"ID":"6e2e853c-6b97-4b1e-9588-7afdc0d95932","Type":"ContainerStarted","Data":"9bf3feb90093360201e3b675ad01a3a57679df272071c1d31447c1ffddbd0261"}
Sep 15 21:10:52.740537 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.740529    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98" event={"ID":"6e2e853c-6b97-4b1e-9588-7afdc0d95932","Type":"ContainerStarted","Data":"cafb70fc1c6d7dd69593ee0dd22f1b164cc33d45e66502c9b4c450dd41467d56"}
Sep 15 21:10:52.740698 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.740558    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98" event={"ID":"6e2e853c-6b97-4b1e-9588-7afdc0d95932","Type":"ContainerStarted","Data":"0a1a40d33cf0658fa56fdc3a71a8e781e2110f341b28dd06b5a2458d8848a6e9"}
Sep 15 21:10:52.740698 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.740592    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98" event={"ID":"6e2e853c-6b97-4b1e-9588-7afdc0d95932","Type":"ContainerStarted","Data":"e2202c8dd33c77bae2234f52faa774c145eeea2e749a7b757277a1c5ad5cf6b0"}
Sep 15 21:10:52.741866 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.741843    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd" event={"ID":"23853078-70a1-4509-a2a9-d1080307d222","Type":"ContainerStarted","Data":"4893329c91fc18eeba69dfa691b8e7c0e25e2e35800bdcd047b5af2351dbe368"}
Sep 15 21:10:52.743137 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.743111    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jrzds" event={"ID":"ba752fdf-05e3-4c36-adf6-4f443cf4f2e4","Type":"ContainerStarted","Data":"5f97f1b72553c507ef984ec24cb2e42d424d84bdfceb5cc9c36f8778ac1f4af1"}
Sep 15 21:10:52.744455 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.744432    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jddxk" event={"ID":"e0c44d74-0807-47f8-a864-f939b64dfea7","Type":"ContainerStarted","Data":"615547de0acb45d112dd6534178234aa082efdac7ee8a8bfb51e504d6fc69fac"}
Sep 15 21:10:52.745750 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.745731    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q2rvv" event={"ID":"c27eb1d3-c379-4e71-9694-7cfce96c4a47","Type":"ContainerStarted","Data":"9f6acb6674f001444ded6b082275a2dd763114a5bd7c2409a2072979d6c681bb"}
Sep 15 21:10:52.747082 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.747062    2584 generic.go:356] "Generic (PLEG): container finished" podID="821589e9-d2dd-49ed-8b13-c9dae33d8747" containerID="8a16e13f8a84f59e5f572643e50d1af0a9ac8aa9ee44a7ac483d356710b34567" exitCode=0
Sep 15 21:10:52.747152 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.747120    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rhc4j" event={"ID":"821589e9-d2dd-49ed-8b13-c9dae33d8747","Type":"ContainerDied","Data":"8a16e13f8a84f59e5f572643e50d1af0a9ac8aa9ee44a7ac483d356710b34567"}
Sep 15 21:10:52.748597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.748560    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rh5pt" event={"ID":"c1ff81ab-8f51-438e-a910-4d6624e8cbbe","Type":"ContainerStarted","Data":"3b9e0b86cd0a9881c59ada38638cfe7fc9eccdc73218899a07247c1da8d717b5"}
Sep 15 21:10:52.753852 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.753816    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-184.ec2.internal" podStartSLOduration=18.753805462 podStartE2EDuration="18.753805462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:37.730982979 +0000 UTC m=+5.592916785" watchObservedRunningTime="2026-09-15 21:10:52.753805462 +0000 UTC m=+20.615739266"
Sep 15 21:10:52.753936 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.753879    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9lfnw" podStartSLOduration=4.341292588 podStartE2EDuration="20.75387541s" totalImagesPullingTime="16.412582822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:52.75369801 +0000 UTC m=+20.615631815" watchObservedRunningTime="2026-09-15 21:10:52.75387541 +0000 UTC m=+20.615809227"
Sep 15 21:10:52.785054 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.785019    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jrzds" podStartSLOduration=4.495648274 podStartE2EDuration="20.785007127s" totalImagesPullingTime="16.289358853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:52.784889751 +0000 UTC m=+20.646823556" watchObservedRunningTime="2026-09-15 21:10:52.785007127 +0000 UTC m=+20.646940932"
Sep 15 21:10:52.800324 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.800288    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jddxk" podStartSLOduration=4.537716185 podStartE2EDuration="20.800278588s" totalImagesPullingTime="16.262562403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:52.800163076 +0000 UTC m=+20.662096891" watchObservedRunningTime="2026-09-15 21:10:52.800278588 +0000 UTC m=+20.662212391"
Sep 15 21:10:52.815127 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.815090    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q2rvv" podStartSLOduration=11.968334158 podStartE2EDuration="20.815079661s" totalImagesPullingTime="8.846745503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:52.814770163 +0000 UTC m=+20.676703968" watchObservedRunningTime="2026-09-15 21:10:52.815079661 +0000 UTC m=+20.677013466"
Sep 15 21:10:52.828905 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:52.828873    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rh5pt" podStartSLOduration=4.890376528 podStartE2EDuration="20.828863129s" totalImagesPullingTime="15.938486601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:52.828532061 +0000 UTC m=+20.690465866" watchObservedRunningTime="2026-09-15 21:10:52.828863129 +0000 UTC m=+20.690796933"
Sep 15 21:10:53.339401 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:53.339358    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jrzds"
Sep 15 21:10:53.340057 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:53.340035    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jrzds"
Sep 15 21:10:53.628704 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:53.628675    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 21:10:53.666460 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:53.666439    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:53.666567 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:53.666540    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:10:53.676220 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:53.676149    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T21:10:53.628700015Z","UUID":"9faa8ad8-c86a-4848-bb29-e968544410e7","Handler":null,"Name":"","Endpoint":""}
Sep 15 21:10:53.677853 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:53.677834    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 21:10:53.677941 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:53.677866    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 21:10:53.752974 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:53.752887    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zk9g4" event={"ID":"72452630-f37e-48fb-8243-1b52655dd088","Type":"ContainerStarted","Data":"58596cb277d369add79d74ef9d08d40c90086def5fc093e47797cd828d793669"}
Sep 15 21:10:53.754988 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:53.754958    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd" event={"ID":"23853078-70a1-4509-a2a9-d1080307d222","Type":"ContainerStarted","Data":"678c9aca9cf672de317e71889fee4e0f8f64fbd9a3fcc4065bf0f5c3b0de7ed9"}
Sep 15 21:10:53.755690 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:53.755670    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jrzds"
Sep 15 21:10:53.756123 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:53.756103    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jrzds"
Sep 15 21:10:53.773709 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:53.773667    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zk9g4" podStartSLOduration=12.884848439 podStartE2EDuration="21.773656478s" totalImagesPullingTime="8.888808039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:53.773171743 +0000 UTC m=+21.635105560" watchObservedRunningTime="2026-09-15 21:10:53.773656478 +0000 UTC m=+21.635590290"
Sep 15 21:10:54.666638 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:54.666470    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:54.666794 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:54.666508    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:54.666794 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:54.666736    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:10:54.666911 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:54.666819    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:10:54.762720 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:54.762686    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98" event={"ID":"6e2e853c-6b97-4b1e-9588-7afdc0d95932","Type":"ContainerStarted","Data":"4070a2e2204725e36f1415e932a121ab6ddedad1dc9bc25add5e502ec1ef8b75"}
Sep 15 21:10:54.766165 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:54.765600    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd" event={"ID":"23853078-70a1-4509-a2a9-d1080307d222","Type":"ContainerStarted","Data":"7450fcaa02e7b1227c4b52118aa4922af824fbc32adfd44d056c26b7dd62ff9a"}
Sep 15 21:10:54.786340 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:54.786285    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jwrxd" podStartSLOduration=4.897796862 podStartE2EDuration="22.786264225s" totalImagesPullingTime="17.888467363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:54.785842885 +0000 UTC m=+22.647776691" watchObservedRunningTime="2026-09-15 21:10:54.786264225 +0000 UTC m=+22.648198031"
Sep 15 21:10:55.666542 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:55.666514    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:55.666743 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:55.666619    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:10:56.666392 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:56.666346    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:56.666911 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:56.666474    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:10:56.666911 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:56.666524    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:56.666911 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:56.666620    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:10:57.666912 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:57.666708    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:57.667239 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:57.666945    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:10:57.774191 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:57.774158    2584 generic.go:356] "Generic (PLEG): container finished" podID="821589e9-d2dd-49ed-8b13-c9dae33d8747" containerID="64e3c3e7e49bf4e11ed1a8bf581cf3f9bb6964df33c988b3c3b1c262870bf809" exitCode=0
Sep 15 21:10:57.774330 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:57.774244    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rhc4j" event={"ID":"821589e9-d2dd-49ed-8b13-c9dae33d8747","Type":"ContainerDied","Data":"64e3c3e7e49bf4e11ed1a8bf581cf3f9bb6964df33c988b3c3b1c262870bf809"}
Sep 15 21:10:57.778246 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:57.778224    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98" event={"ID":"6e2e853c-6b97-4b1e-9588-7afdc0d95932","Type":"ContainerStarted","Data":"d32b77b30334439b8b587dd4b8d8abc747d0bcbf48a76468f94b58815b2ee168"}
Sep 15 21:10:57.778532 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:57.778511    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:57.778606 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:57.778582    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:57.793257 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:57.793240    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:57.819669 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:57.819622    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98" podStartSLOduration=9.486539442 podStartE2EDuration="25.819608194s" totalImagesPullingTime="16.333068752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:10:57.819347793 +0000 UTC m=+25.681281599" watchObservedRunningTime="2026-09-15 21:10:57.819608194 +0000 UTC m=+25.681541998"
Sep 15 21:10:58.666747 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:58.666724    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:58.666842 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:58.666823    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:10:58.666925 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:58.666908    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:58.667473 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:58.667256    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:10:58.781667 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:58.781594    2584 generic.go:356] "Generic (PLEG): container finished" podID="821589e9-d2dd-49ed-8b13-c9dae33d8747" containerID="237e359d1e84dd868ab6eed4f8e68b671cffa3f10df363efd3b89b06c15e2b82" exitCode=0
Sep 15 21:10:58.781761 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:58.781685    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rhc4j" event={"ID":"821589e9-d2dd-49ed-8b13-c9dae33d8747","Type":"ContainerDied","Data":"237e359d1e84dd868ab6eed4f8e68b671cffa3f10df363efd3b89b06c15e2b82"}
Sep 15 21:10:58.782254 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:58.782237    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:58.798188 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:58.798168    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:10:58.887466 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:58.887431    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wghbm"]
Sep 15 21:10:58.887636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:58.887538    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:10:58.887700 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:58.887659    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:10:58.890750 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:58.890715    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9dgmv"]
Sep 15 21:10:58.890917 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:58.890826    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:10:58.890980 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:58.890931    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:10:58.891574 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:58.891532    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pt55j"]
Sep 15 21:10:58.891671 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:58.891637    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:10:58.891745 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:10:58.891720    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:10:59.786135 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:59.785882    2584 generic.go:356] "Generic (PLEG): container finished" podID="821589e9-d2dd-49ed-8b13-c9dae33d8747" containerID="e1ece1642aea92a6116ed4eead18bb74ac21a9a0cc8b243d464c828b32608e94" exitCode=0
Sep 15 21:10:59.786476 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:10:59.785961    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rhc4j" event={"ID":"821589e9-d2dd-49ed-8b13-c9dae33d8747","Type":"ContainerDied","Data":"e1ece1642aea92a6116ed4eead18bb74ac21a9a0cc8b243d464c828b32608e94"}
Sep 15 21:11:00.666899 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:00.666869    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:11:00.666899 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:00.666903    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:11:00.667104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:00.666870    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:11:00.667104 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:00.666993    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:11:00.667104 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:00.667060    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:11:00.667242 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:00.667134    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:11:02.667726 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:02.667689    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:11:02.668227 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:02.667794    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:11:02.668227 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:02.667840    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:11:02.668227 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:02.667841    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:11:02.668227 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:02.667920    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:11:02.668227 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:02.667994    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:11:04.666024 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:04.665990    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:11:04.666488 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:04.665997    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:11:04.666488 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:04.666105    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9dgmv" podUID="d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6"
Sep 15 21:11:04.666488 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:04.666003    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:11:04.666488 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:04.666171    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt55j" podUID="d8ed8d4e-9c80-4109-a61a-440ac53e69e6"
Sep 15 21:11:04.666488 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:04.666262    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wghbm" podUID="a2260413-c64c-4707-92f9-ed64418eeda4"
Sep 15 21:11:04.908786 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:04.908625    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-184.ec2.internal" event="NodeReady"
Sep 15 21:11:04.908905 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:04.908893    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 21:11:04.976449 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:04.976424    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vg2k9"]
Sep 15 21:11:04.990759 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:04.990727    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-grkbv"]
Sep 15 21:11:04.990879 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:04.990862    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:04.993096 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:04.992951    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nfdpp\""
Sep 15 21:11:04.993096 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:04.992977    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 21:11:04.993096 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:04.992960    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 21:11:05.009203 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.009184    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-grkbv"]
Sep 15 21:11:05.009295 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.009214    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vg2k9"]
Sep 15 21:11:05.009343 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.009295    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-grkbv"
Sep 15 21:11:05.011487 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.011473    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8lhn5\""
Sep 15 21:11:05.011557 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.011494    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 21:11:05.011601 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.011494    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 21:11:05.011601 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.011475    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 21:11:05.119112 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.119092    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62fa88e4-9983-484f-87b7-65a070871dfa-config-volume\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:05.119226 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.119127    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5drsf\" (UniqueName: \"kubernetes.io/projected/62fa88e4-9983-484f-87b7-65a070871dfa-kube-api-access-5drsf\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:05.119284 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.119242    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:05.119284 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.119277    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e36f14f-f82d-49f1-aacf-aba017667671-cert\") pod \"ingress-canary-grkbv\" (UID: \"7e36f14f-f82d-49f1-aacf-aba017667671\") " pod="openshift-ingress-canary/ingress-canary-grkbv"
Sep 15 21:11:05.119397 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.119295    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdp2q\" (UniqueName: \"kubernetes.io/projected/7e36f14f-f82d-49f1-aacf-aba017667671-kube-api-access-qdp2q\") pod \"ingress-canary-grkbv\" (UID: \"7e36f14f-f82d-49f1-aacf-aba017667671\") " pod="openshift-ingress-canary/ingress-canary-grkbv"
Sep 15 21:11:05.119397 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.119362    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62fa88e4-9983-484f-87b7-65a070871dfa-tmp-dir\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:05.219826 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.219776    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:05.219826 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.219803    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e36f14f-f82d-49f1-aacf-aba017667671-cert\") pod \"ingress-canary-grkbv\" (UID: \"7e36f14f-f82d-49f1-aacf-aba017667671\") " pod="openshift-ingress-canary/ingress-canary-grkbv"
Sep 15 21:11:05.219826 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.219818    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdp2q\" (UniqueName: \"kubernetes.io/projected/7e36f14f-f82d-49f1-aacf-aba017667671-kube-api-access-qdp2q\") pod \"ingress-canary-grkbv\" (UID: \"7e36f14f-f82d-49f1-aacf-aba017667671\") " pod="openshift-ingress-canary/ingress-canary-grkbv"
Sep 15 21:11:05.220008 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.219850    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62fa88e4-9983-484f-87b7-65a070871dfa-tmp-dir\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:05.220008 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.219874    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62fa88e4-9983-484f-87b7-65a070871dfa-config-volume\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:05.220008 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.219901    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5drsf\" (UniqueName: \"kubernetes.io/projected/62fa88e4-9983-484f-87b7-65a070871dfa-kube-api-access-5drsf\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:05.220008 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:05.219933    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:11:05.220008 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:05.220000    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls podName:62fa88e4-9983-484f-87b7-65a070871dfa nodeName:}" failed. No retries permitted until 2026-09-15 21:11:05.719979625 +0000 UTC m=+33.581913415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls") pod "dns-default-vg2k9" (UID: "62fa88e4-9983-484f-87b7-65a070871dfa") : secret "dns-default-metrics-tls" not found
Sep 15 21:11:05.220210 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.220177    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62fa88e4-9983-484f-87b7-65a070871dfa-tmp-dir\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:05.220498 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.220482    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62fa88e4-9983-484f-87b7-65a070871dfa-config-volume\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:05.223830 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.223812    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e36f14f-f82d-49f1-aacf-aba017667671-cert\") pod \"ingress-canary-grkbv\" (UID: \"7e36f14f-f82d-49f1-aacf-aba017667671\") " pod="openshift-ingress-canary/ingress-canary-grkbv"
Sep 15 21:11:05.228747 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.228721    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5drsf\" (UniqueName: \"kubernetes.io/projected/62fa88e4-9983-484f-87b7-65a070871dfa-kube-api-access-5drsf\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:05.228872 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.228856    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdp2q\" (UniqueName: \"kubernetes.io/projected/7e36f14f-f82d-49f1-aacf-aba017667671-kube-api-access-qdp2q\") pod \"ingress-canary-grkbv\" (UID: \"7e36f14f-f82d-49f1-aacf-aba017667671\") " pod="openshift-ingress-canary/ingress-canary-grkbv"
Sep 15 21:11:05.343552 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.343529    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-grkbv"
Sep 15 21:11:05.654444 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.654358    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-grkbv"]
Sep 15 21:11:05.659025 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:11:05.658737    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e36f14f_f82d_49f1_aacf_aba017667671.slice/crio-3873637626dd1350920b76aa45a061a85859584df7b6ce95b7068ce517865793 WatchSource:0}: Error finding container 3873637626dd1350920b76aa45a061a85859584df7b6ce95b7068ce517865793: Status 404 returned error can't find the container with id 3873637626dd1350920b76aa45a061a85859584df7b6ce95b7068ce517865793
Sep 15 21:11:05.723085 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.723061    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:05.723722 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:05.723170    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:11:05.723722 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:05.723216    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls podName:62fa88e4-9983-484f-87b7-65a070871dfa nodeName:}" failed. No retries permitted until 2026-09-15 21:11:06.723203538 +0000 UTC m=+34.585137322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls") pod "dns-default-vg2k9" (UID: "62fa88e4-9983-484f-87b7-65a070871dfa") : secret "dns-default-metrics-tls" not found
Sep 15 21:11:05.801590 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.801565    2584 generic.go:356] "Generic (PLEG): container finished" podID="821589e9-d2dd-49ed-8b13-c9dae33d8747" containerID="44a44be80664d65f1a34fb67469b9d0cf7aebc6ebb6f7afb7addd8b9bdb803fb" exitCode=0
Sep 15 21:11:05.801717 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.801636    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rhc4j" event={"ID":"821589e9-d2dd-49ed-8b13-c9dae33d8747","Type":"ContainerDied","Data":"44a44be80664d65f1a34fb67469b9d0cf7aebc6ebb6f7afb7addd8b9bdb803fb"}
Sep 15 21:11:05.802838 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:05.802812    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-grkbv" event={"ID":"7e36f14f-f82d-49f1-aacf-aba017667671","Type":"ContainerStarted","Data":"3873637626dd1350920b76aa45a061a85859584df7b6ce95b7068ce517865793"}
Sep 15 21:11:06.226646 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:06.226574    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret\") pod \"global-pull-secret-syncer-wghbm\" (UID: \"a2260413-c64c-4707-92f9-ed64418eeda4\") " pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:11:06.226811 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:06.226729    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:11:06.226811 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:06.226795    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret podName:a2260413-c64c-4707-92f9-ed64418eeda4 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:38.226780633 +0000 UTC m=+66.088714420 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret") pod "global-pull-secret-syncer-wghbm" (UID: "a2260413-c64c-4707-92f9-ed64418eeda4") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:11:06.327488 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:06.327458    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs\") pod \"network-metrics-daemon-pt55j\" (UID: \"d8ed8d4e-9c80-4109-a61a-440ac53e69e6\") " pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:11:06.327634 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:06.327602    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:11:06.327683 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:06.327665    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs podName:d8ed8d4e-9c80-4109-a61a-440ac53e69e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:38.327649072 +0000 UTC m=+66.189582854 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs") pod "network-metrics-daemon-pt55j" (UID: "d8ed8d4e-9c80-4109-a61a-440ac53e69e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:11:06.428161 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:06.428124    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xng6l\" (UniqueName: \"kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l\") pod \"network-check-target-9dgmv\" (UID: \"d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6\") " pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:11:06.428312 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:06.428276    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:11:06.428312 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:06.428304    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:11:06.428402 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:06.428318    2584 projected.go:196] Error preparing data for projected volume kube-api-access-xng6l for pod openshift-network-diagnostics/network-check-target-9dgmv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:11:06.428402 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:06.428392    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l podName:d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:38.428362079 +0000 UTC m=+66.290295864 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xng6l" (UniqueName: "kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l") pod "network-check-target-9dgmv" (UID: "d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:11:06.666999 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:06.666923    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:11:06.667158 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:06.666923    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:11:06.667158 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:06.666933    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:11:06.670789 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:06.670597    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8fnh6\""
Sep 15 21:11:06.670789 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:06.670637    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 21:11:06.670789 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:06.670597    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6cwlg\""
Sep 15 21:11:06.670789 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:06.670639    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 21:11:06.670789 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:06.670598    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 21:11:06.670789 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:06.670678    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 21:11:06.730713 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:06.730684    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:06.731074 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:06.730811    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:11:06.731074 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:06.730871    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls podName:62fa88e4-9983-484f-87b7-65a070871dfa nodeName:}" failed. No retries permitted until 2026-09-15 21:11:08.730854967 +0000 UTC m=+36.592788755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls") pod "dns-default-vg2k9" (UID: "62fa88e4-9983-484f-87b7-65a070871dfa") : secret "dns-default-metrics-tls" not found
Sep 15 21:11:06.809622 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:06.809591    2584 generic.go:356] "Generic (PLEG): container finished" podID="821589e9-d2dd-49ed-8b13-c9dae33d8747" containerID="00c0eed75b6db5a82ee23c1344cf498770cd7ad7c8b7c3963297a553abab6493" exitCode=0
Sep 15 21:11:06.809754 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:06.809635    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rhc4j" event={"ID":"821589e9-d2dd-49ed-8b13-c9dae33d8747","Type":"ContainerDied","Data":"00c0eed75b6db5a82ee23c1344cf498770cd7ad7c8b7c3963297a553abab6493"}
Sep 15 21:11:07.817222 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:07.817046    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rhc4j" event={"ID":"821589e9-d2dd-49ed-8b13-c9dae33d8747","Type":"ContainerStarted","Data":"224817ccc9bb74513d7ecca01879b6d3b316941693906192304c10172ec00016"}
Sep 15 21:11:07.839414 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:07.839359    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rhc4j" podStartSLOduration=16.306122242 podStartE2EDuration="35.839342032s" totalImagesPullingTime="19.53321979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:11:07.83817077 +0000 UTC m=+35.700104575" watchObservedRunningTime="2026-09-15 21:11:07.839342032 +0000 UTC m=+35.701275836"
Sep 15 21:11:08.746507 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:08.746479    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:08.746653 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:08.746602    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:11:08.746693 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:08.746658    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls podName:62fa88e4-9983-484f-87b7-65a070871dfa nodeName:}" failed. No retries permitted until 2026-09-15 21:11:12.746642563 +0000 UTC m=+40.608576346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls") pod "dns-default-vg2k9" (UID: "62fa88e4-9983-484f-87b7-65a070871dfa") : secret "dns-default-metrics-tls" not found
Sep 15 21:11:08.820909 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:08.820876    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-grkbv" event={"ID":"7e36f14f-f82d-49f1-aacf-aba017667671","Type":"ContainerStarted","Data":"2e5953ca3d6d1fe4dd98676952746cf4ab42d6a99922c294431c092fb5202147"}
Sep 15 21:11:12.771934 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:12.771904    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:12.772299 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:12.771994    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:11:12.772299 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:12.772039    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls podName:62fa88e4-9983-484f-87b7-65a070871dfa nodeName:}" failed. No retries permitted until 2026-09-15 21:11:20.77202592 +0000 UTC m=+48.633959702 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls") pod "dns-default-vg2k9" (UID: "62fa88e4-9983-484f-87b7-65a070871dfa") : secret "dns-default-metrics-tls" not found
Sep 15 21:11:20.824959 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:20.824921    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:20.825327 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:20.825055    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:11:20.825327 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:20.825113    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls podName:62fa88e4-9983-484f-87b7-65a070871dfa nodeName:}" failed. No retries permitted until 2026-09-15 21:11:36.825097615 +0000 UTC m=+64.687031397 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls") pod "dns-default-vg2k9" (UID: "62fa88e4-9983-484f-87b7-65a070871dfa") : secret "dns-default-metrics-tls" not found
Sep 15 21:11:30.798814 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:30.798788    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrz98"
Sep 15 21:11:30.826957 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:30.826912    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-grkbv" podStartSLOduration=24.28548169 podStartE2EDuration="26.826896063s" totalImagesPullingTime="2.541414373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:11:08.836689802 +0000 UTC m=+36.698623608" watchObservedRunningTime="2026-09-15 21:11:30.826896063 +0000 UTC m=+58.688829881"
Sep 15 21:11:36.832890 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:36.832852    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:11:36.833265 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:36.832980    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:11:36.833265 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:36.833034    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls podName:62fa88e4-9983-484f-87b7-65a070871dfa nodeName:}" failed. No retries permitted until 2026-09-15 21:12:08.83301889 +0000 UTC m=+96.694952672 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls") pod "dns-default-vg2k9" (UID: "62fa88e4-9983-484f-87b7-65a070871dfa") : secret "dns-default-metrics-tls" not found
Sep 15 21:11:38.242610 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:38.242563    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret\") pod \"global-pull-secret-syncer-wghbm\" (UID: \"a2260413-c64c-4707-92f9-ed64418eeda4\") " pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:11:38.245080 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:38.245063    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 21:11:38.256889 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:38.256861    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2260413-c64c-4707-92f9-ed64418eeda4-original-pull-secret\") pod \"global-pull-secret-syncer-wghbm\" (UID: \"a2260413-c64c-4707-92f9-ed64418eeda4\") " pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:11:38.342844 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:38.342812    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs\") pod \"network-metrics-daemon-pt55j\" (UID: \"d8ed8d4e-9c80-4109-a61a-440ac53e69e6\") " pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:11:38.345307 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:38.345285    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 21:11:38.353347 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:38.353327    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 21:11:38.353448 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:38.353408    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs podName:d8ed8d4e-9c80-4109-a61a-440ac53e69e6 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:42.353372862 +0000 UTC m=+130.215306665 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs") pod "network-metrics-daemon-pt55j" (UID: "d8ed8d4e-9c80-4109-a61a-440ac53e69e6") : secret "metrics-daemon-secret" not found
Sep 15 21:11:38.443552 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:38.443519    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xng6l\" (UniqueName: \"kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l\") pod \"network-check-target-9dgmv\" (UID: \"d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6\") " pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:11:38.446294 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:38.446276    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 21:11:38.456494 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:38.456472    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 21:11:38.466648 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:38.466630    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xng6l\" (UniqueName: \"kubernetes.io/projected/d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6-kube-api-access-xng6l\") pod \"network-check-target-9dgmv\" (UID: \"d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6\") " pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:11:38.480692 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:38.480672    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6cwlg\""
Sep 15 21:11:38.489428 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:38.489397    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:11:38.491609 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:38.491589    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wghbm"
Sep 15 21:11:38.768914 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:38.767621    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wghbm"]
Sep 15 21:11:38.770462 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:38.770441    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9dgmv"]
Sep 15 21:11:38.772271 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:11:38.772245    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2260413_c64c_4707_92f9_ed64418eeda4.slice/crio-6fa89e0402a84361bd7c8e7b48b99bd8326d1bdfee4ca302b1379d7b5e11b6be WatchSource:0}: Error finding container 6fa89e0402a84361bd7c8e7b48b99bd8326d1bdfee4ca302b1379d7b5e11b6be: Status 404 returned error can't find the container with id 6fa89e0402a84361bd7c8e7b48b99bd8326d1bdfee4ca302b1379d7b5e11b6be
Sep 15 21:11:38.772966 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:11:38.772945    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd029f1c5_ba9a_48f7_a115_1b5ff8e1f9e6.slice/crio-276fbbe7c84db17fbb6123088780c218a12a19e9039beaa8403e621a0d1773e1 WatchSource:0}: Error finding container 276fbbe7c84db17fbb6123088780c218a12a19e9039beaa8403e621a0d1773e1: Status 404 returned error can't find the container with id 276fbbe7c84db17fbb6123088780c218a12a19e9039beaa8403e621a0d1773e1
Sep 15 21:11:38.889517 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:38.889480    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wghbm" event={"ID":"a2260413-c64c-4707-92f9-ed64418eeda4","Type":"ContainerStarted","Data":"6fa89e0402a84361bd7c8e7b48b99bd8326d1bdfee4ca302b1379d7b5e11b6be"}
Sep 15 21:11:38.890489 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:38.890461    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9dgmv" event={"ID":"d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6","Type":"ContainerStarted","Data":"276fbbe7c84db17fbb6123088780c218a12a19e9039beaa8403e621a0d1773e1"}
Sep 15 21:11:43.904475 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:43.904441    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wghbm" event={"ID":"a2260413-c64c-4707-92f9-ed64418eeda4","Type":"ContainerStarted","Data":"62b8526f8d2fe6c0ecf00b7ecabc2730fffb826445fabc13ccefc06b703ccb2a"}
Sep 15 21:11:43.909255 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:43.909230    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9dgmv" event={"ID":"d029f1c5-ba9a-48f7-a115-1b5ff8e1f9e6","Type":"ContainerStarted","Data":"1cd09dd5006e0009844d484b51806e517670e9f4046371fa8267a7fa31fc743d"}
Sep 15 21:11:43.909396 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:43.909295    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:11:43.927085 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:43.927043    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wghbm" podStartSLOduration=67.390840715 podStartE2EDuration="1m11.927028958s" totalImagesPullingTime="4.536188243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:11:43.923577562 +0000 UTC m=+71.785511367" watchObservedRunningTime="2026-09-15 21:11:43.927028958 +0000 UTC m=+71.788962763"
Sep 15 21:11:43.947019 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:43.946977    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9dgmv" podStartSLOduration=67.417125521 podStartE2EDuration="1m11.946964219s" totalImagesPullingTime="4.529838698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:11:43.946895559 +0000 UTC m=+71.808829363" watchObservedRunningTime="2026-09-15 21:11:43.946964219 +0000 UTC m=+71.808898024"
Sep 15 21:11:47.229991 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.229954    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"]
Sep 15 21:11:47.232816 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.232794    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:11:47.233352 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.233327    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-547bb554b6-j9fkz"]
Sep 15 21:11:47.235172 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.235152    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 15 21:11:47.235249 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.235206    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 21:11:47.236069 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.236050    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:47.236478 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.236460    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 21:11:47.236650 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.236633    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8w2jg\""
Sep 15 21:11:47.236768 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.236748    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 15 21:11:47.239545 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.239534    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 21:11:47.239889 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.239854    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 15 21:11:47.239941 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.239922    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bmvtp\""
Sep 15 21:11:47.240807 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.240787    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 15 21:11:47.241115 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.241100    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 15 21:11:47.242431 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.242414    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 21:11:47.242634 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.242622    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 15 21:11:47.244947 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.244925    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"]
Sep 15 21:11:47.251565 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.251547    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-547bb554b6-j9fkz"]
Sep 15 21:11:47.298726 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.298701    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/306fd13e-6e53-43f4-9903-e0a886c375e8-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-mfp8r\" (UID: \"306fd13e-6e53-43f4-9903-e0a886c375e8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:11:47.298863 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.298746    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd779\" (UniqueName: \"kubernetes.io/projected/63176ac2-9b27-4a17-88f4-0a934db86c97-kube-api-access-dd779\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:47.298863 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.298779    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:47.298863 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.298804    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-default-certificate\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:47.298863 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.298824    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:47.298863 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.298852    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-stats-auth\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:47.299053 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.298898    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r65r9\" (UniqueName: \"kubernetes.io/projected/306fd13e-6e53-43f4-9903-e0a886c375e8-kube-api-access-r65r9\") pod \"cluster-monitoring-operator-6cd954b897-mfp8r\" (UID: \"306fd13e-6e53-43f4-9903-e0a886c375e8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:11:47.299053 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.298943    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mfp8r\" (UID: \"306fd13e-6e53-43f4-9903-e0a886c375e8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:11:47.399741 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.399701    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-stats-auth\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:47.399741 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.399747    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r65r9\" (UniqueName: \"kubernetes.io/projected/306fd13e-6e53-43f4-9903-e0a886c375e8-kube-api-access-r65r9\") pod \"cluster-monitoring-operator-6cd954b897-mfp8r\" (UID: \"306fd13e-6e53-43f4-9903-e0a886c375e8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:11:47.400007 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.399777    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mfp8r\" (UID: \"306fd13e-6e53-43f4-9903-e0a886c375e8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:11:47.400007 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.399804    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/306fd13e-6e53-43f4-9903-e0a886c375e8-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-mfp8r\" (UID: \"306fd13e-6e53-43f4-9903-e0a886c375e8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:11:47.400007 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:47.399890    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 21:11:47.400007 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:47.399956    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls podName:306fd13e-6e53-43f4-9903-e0a886c375e8 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:47.899936384 +0000 UTC m=+75.761870180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mfp8r" (UID: "306fd13e-6e53-43f4-9903-e0a886c375e8") : secret "cluster-monitoring-operator-tls" not found
Sep 15 21:11:47.400220 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.400071    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd779\" (UniqueName: \"kubernetes.io/projected/63176ac2-9b27-4a17-88f4-0a934db86c97-kube-api-access-dd779\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:47.400220 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.400117    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:47.400220 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.400143    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-default-certificate\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:47.400220 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.400158    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:47.400461 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:47.400220    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 21:11:47.400461 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:47.400260    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs podName:63176ac2-9b27-4a17-88f4-0a934db86c97 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:47.900249118 +0000 UTC m=+75.762182901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs") pod "router-default-547bb554b6-j9fkz" (UID: "63176ac2-9b27-4a17-88f4-0a934db86c97") : secret "router-metrics-certs-default" not found
Sep 15 21:11:47.400461 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:47.400279    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle podName:63176ac2-9b27-4a17-88f4-0a934db86c97 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:47.900269288 +0000 UTC m=+75.762203076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle") pod "router-default-547bb554b6-j9fkz" (UID: "63176ac2-9b27-4a17-88f4-0a934db86c97") : configmap references non-existent config key: service-ca.crt
Sep 15 21:11:47.400699 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.400677    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/306fd13e-6e53-43f4-9903-e0a886c375e8-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-mfp8r\" (UID: \"306fd13e-6e53-43f4-9903-e0a886c375e8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:11:47.402708 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.402691    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-default-certificate\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:47.402803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.402787    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-stats-auth\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:47.413368 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.413340    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r65r9\" (UniqueName: \"kubernetes.io/projected/306fd13e-6e53-43f4-9903-e0a886c375e8-kube-api-access-r65r9\") pod \"cluster-monitoring-operator-6cd954b897-mfp8r\" (UID: \"306fd13e-6e53-43f4-9903-e0a886c375e8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:11:47.413513 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.413492    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd779\" (UniqueName: \"kubernetes.io/projected/63176ac2-9b27-4a17-88f4-0a934db86c97-kube-api-access-dd779\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:47.904044 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.904005    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mfp8r\" (UID: \"306fd13e-6e53-43f4-9903-e0a886c375e8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:11:47.904209 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.904086    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:47.904209 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:47.904113    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:47.904209 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:47.904158    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 21:11:47.904308 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:47.904201    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 21:11:47.904308 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:47.904220    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle podName:63176ac2-9b27-4a17-88f4-0a934db86c97 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:48.904202938 +0000 UTC m=+76.766136726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle") pod "router-default-547bb554b6-j9fkz" (UID: "63176ac2-9b27-4a17-88f4-0a934db86c97") : configmap references non-existent config key: service-ca.crt
Sep 15 21:11:47.904308 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:47.904238    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls podName:306fd13e-6e53-43f4-9903-e0a886c375e8 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:48.90423043 +0000 UTC m=+76.766164212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mfp8r" (UID: "306fd13e-6e53-43f4-9903-e0a886c375e8") : secret "cluster-monitoring-operator-tls" not found
Sep 15 21:11:47.904684 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:47.904660    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs podName:63176ac2-9b27-4a17-88f4-0a934db86c97 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:48.904636606 +0000 UTC m=+76.766570403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs") pod "router-default-547bb554b6-j9fkz" (UID: "63176ac2-9b27-4a17-88f4-0a934db86c97") : secret "router-metrics-certs-default" not found
Sep 15 21:11:48.911228 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:48.911192    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:48.911228 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:48.911231    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:48.911755 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:48.911260    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mfp8r\" (UID: \"306fd13e-6e53-43f4-9903-e0a886c375e8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:11:48.911755 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:48.911335    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 21:11:48.911755 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:48.911349    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 21:11:48.911755 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:48.911360    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle podName:63176ac2-9b27-4a17-88f4-0a934db86c97 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:50.911345516 +0000 UTC m=+78.773279298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle") pod "router-default-547bb554b6-j9fkz" (UID: "63176ac2-9b27-4a17-88f4-0a934db86c97") : configmap references non-existent config key: service-ca.crt
Sep 15 21:11:48.911755 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:48.911477    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls podName:306fd13e-6e53-43f4-9903-e0a886c375e8 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:50.911461235 +0000 UTC m=+78.773395018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mfp8r" (UID: "306fd13e-6e53-43f4-9903-e0a886c375e8") : secret "cluster-monitoring-operator-tls" not found
Sep 15 21:11:48.911755 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:48.911496    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs podName:63176ac2-9b27-4a17-88f4-0a934db86c97 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:50.911485821 +0000 UTC m=+78.773419609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs") pod "router-default-547bb554b6-j9fkz" (UID: "63176ac2-9b27-4a17-88f4-0a934db86c97") : secret "router-metrics-certs-default" not found
Sep 15 21:11:50.643409 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.643360    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4mgr8"]
Sep 15 21:11:50.647373 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.647354    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:50.649720 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.649704    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 21:11:50.650691 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.650673    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 21:11:50.650691 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.650684    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qznqd\""
Sep 15 21:11:50.650821 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.650717    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 21:11:50.651440 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.651425    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 21:11:50.657747 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.657726    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4mgr8"]
Sep 15 21:11:50.724538 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.724503    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:50.724662 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.724541    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ttf8\" (UniqueName: \"kubernetes.io/projected/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-api-access-7ttf8\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:50.724662 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.724593    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-data-volume\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:50.724777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.724666    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:50.724777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.724712    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-crio-socket\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:50.825917 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.825881    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-crio-socket\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:50.826008 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.825924    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:50.826008 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.825943    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ttf8\" (UniqueName: \"kubernetes.io/projected/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-api-access-7ttf8\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:50.826008 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.825976    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-data-volume\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:50.826008 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.825994    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-crio-socket\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:50.826194 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.826034    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:50.826194 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:50.826044    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:50.826194 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:50.826122    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls podName:2a0964c1-a9bf-466a-a96e-c51abcdb59cc nodeName:}" failed. No retries permitted until 2026-09-15 21:11:51.32610076 +0000 UTC m=+79.188034545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4mgr8" (UID: "2a0964c1-a9bf-466a-a96e-c51abcdb59cc") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:50.826194 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:50.826157    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:11:50.826349 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:50.826215    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm podName:2a0964c1-a9bf-466a-a96e-c51abcdb59cc nodeName:}" failed. No retries permitted until 2026-09-15 21:11:51.326199741 +0000 UTC m=+79.188133537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4mgr8" (UID: "2a0964c1-a9bf-466a-a96e-c51abcdb59cc") : configmap "kube-rbac-proxy" not found
Sep 15 21:11:50.826349 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.826245    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-data-volume\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:50.835636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.835619    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ttf8\" (UniqueName: \"kubernetes.io/projected/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-api-access-7ttf8\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:50.926347 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.926278    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:50.926347 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.926312    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:50.926545 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:50.926360    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mfp8r\" (UID: \"306fd13e-6e53-43f4-9903-e0a886c375e8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:11:50.926545 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:50.926430    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 21:11:50.926545 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:50.926495    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs podName:63176ac2-9b27-4a17-88f4-0a934db86c97 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:54.92647792 +0000 UTC m=+82.788411717 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs") pod "router-default-547bb554b6-j9fkz" (UID: "63176ac2-9b27-4a17-88f4-0a934db86c97") : secret "router-metrics-certs-default" not found
Sep 15 21:11:50.926545 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:50.926505    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 21:11:50.926545 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:50.926512    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle podName:63176ac2-9b27-4a17-88f4-0a934db86c97 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:54.926503662 +0000 UTC m=+82.788437447 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle") pod "router-default-547bb554b6-j9fkz" (UID: "63176ac2-9b27-4a17-88f4-0a934db86c97") : configmap references non-existent config key: service-ca.crt
Sep 15 21:11:50.926788 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:50.926552    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls podName:306fd13e-6e53-43f4-9903-e0a886c375e8 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:54.926537346 +0000 UTC m=+82.788471142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mfp8r" (UID: "306fd13e-6e53-43f4-9903-e0a886c375e8") : secret "cluster-monitoring-operator-tls" not found
Sep 15 21:11:51.329441 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:51.329402    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:51.329574 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:51.329468    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:51.329574 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:51.329527    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:11:51.329649 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:51.329591    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:51.329649 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:51.329599    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm podName:2a0964c1-a9bf-466a-a96e-c51abcdb59cc nodeName:}" failed. No retries permitted until 2026-09-15 21:11:52.329583698 +0000 UTC m=+80.191517498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4mgr8" (UID: "2a0964c1-a9bf-466a-a96e-c51abcdb59cc") : configmap "kube-rbac-proxy" not found
Sep 15 21:11:51.329649 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:51.329632    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls podName:2a0964c1-a9bf-466a-a96e-c51abcdb59cc nodeName:}" failed. No retries permitted until 2026-09-15 21:11:52.329619908 +0000 UTC m=+80.191553691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4mgr8" (UID: "2a0964c1-a9bf-466a-a96e-c51abcdb59cc") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:52.337573 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:52.337538    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:52.337952 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:52.337586    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:52.337952 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:52.337647    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:11:52.337952 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:52.337698    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:52.337952 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:52.337706    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm podName:2a0964c1-a9bf-466a-a96e-c51abcdb59cc nodeName:}" failed. No retries permitted until 2026-09-15 21:11:54.33769225 +0000 UTC m=+82.199626037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4mgr8" (UID: "2a0964c1-a9bf-466a-a96e-c51abcdb59cc") : configmap "kube-rbac-proxy" not found
Sep 15 21:11:52.337952 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:52.337736    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls podName:2a0964c1-a9bf-466a-a96e-c51abcdb59cc nodeName:}" failed. No retries permitted until 2026-09-15 21:11:54.337725422 +0000 UTC m=+82.199659232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4mgr8" (UID: "2a0964c1-a9bf-466a-a96e-c51abcdb59cc") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:52.589420 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:52.589345    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q2rvv_c27eb1d3-c379-4e71-9694-7cfce96c4a47/dns-node-resolver/0.log"
Sep 15 21:11:53.788329 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:53.788302    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rh5pt_c1ff81ab-8f51-438e-a910-4d6624e8cbbe/node-ca/0.log"
Sep 15 21:11:54.353548 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:54.353521    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:54.353716 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:54.353603    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:54.353716 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:54.353669    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:54.353793 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:54.353734    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls podName:2a0964c1-a9bf-466a-a96e-c51abcdb59cc nodeName:}" failed. No retries permitted until 2026-09-15 21:11:58.353718388 +0000 UTC m=+86.215652171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4mgr8" (UID: "2a0964c1-a9bf-466a-a96e-c51abcdb59cc") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:54.353793 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:54.353665    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:11:54.353793 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:54.353776    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm podName:2a0964c1-a9bf-466a-a96e-c51abcdb59cc nodeName:}" failed. No retries permitted until 2026-09-15 21:11:58.353766527 +0000 UTC m=+86.215700310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4mgr8" (UID: "2a0964c1-a9bf-466a-a96e-c51abcdb59cc") : configmap "kube-rbac-proxy" not found
Sep 15 21:11:54.388755 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:54.388730    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-grkbv_7e36f14f-f82d-49f1-aacf-aba017667671/serve-healthcheck-canary/0.log"
Sep 15 21:11:54.957954 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:54.957920    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mfp8r\" (UID: \"306fd13e-6e53-43f4-9903-e0a886c375e8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:11:54.958302 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:54.958058    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 21:11:54.958302 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:54.958085    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:54.958302 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:54.958122    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls podName:306fd13e-6e53-43f4-9903-e0a886c375e8 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:02.958104366 +0000 UTC m=+90.820038162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mfp8r" (UID: "306fd13e-6e53-43f4-9903-e0a886c375e8") : secret "cluster-monitoring-operator-tls" not found
Sep 15 21:11:54.958302 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:54.958153    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:11:54.958302 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:54.958157    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 21:11:54.958302 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:54.958234    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle podName:63176ac2-9b27-4a17-88f4-0a934db86c97 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:02.958223018 +0000 UTC m=+90.820156819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle") pod "router-default-547bb554b6-j9fkz" (UID: "63176ac2-9b27-4a17-88f4-0a934db86c97") : configmap references non-existent config key: service-ca.crt
Sep 15 21:11:54.958302 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:54.958249    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs podName:63176ac2-9b27-4a17-88f4-0a934db86c97 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:02.9582437 +0000 UTC m=+90.820177483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs") pod "router-default-547bb554b6-j9fkz" (UID: "63176ac2-9b27-4a17-88f4-0a934db86c97") : secret "router-metrics-certs-default" not found
Sep 15 21:11:57.192588 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.192555    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt"]
Sep 15 21:11:57.195401 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.195371    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt"
Sep 15 21:11:57.198298 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.198278    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:11:57.198298 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.198291    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pbvdb\""
Sep 15 21:11:57.199076 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.199055    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 15 21:11:57.199134 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.199125    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 15 21:11:57.204564 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.204538    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt"]
Sep 15 21:11:57.275919 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.275892    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db2l2\" (UniqueName: \"kubernetes.io/projected/36b7bbb9-15c6-47e8-af15-84b227e35766-kube-api-access-db2l2\") pod \"cluster-samples-operator-86499c89c4-2kvmt\" (UID: \"36b7bbb9-15c6-47e8-af15-84b227e35766\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt"
Sep 15 21:11:57.276659 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.276269    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2kvmt\" (UID: \"36b7bbb9-15c6-47e8-af15-84b227e35766\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt"
Sep 15 21:11:57.288043 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.288015    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-bhlld"]
Sep 15 21:11:57.290899 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.290879    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-bhlld"
Sep 15 21:11:57.291863 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.291840    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-4d7vv"]
Sep 15 21:11:57.293029 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.293015    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kbtgc\""
Sep 15 21:11:57.293126 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.293109    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 15 21:11:57.293179 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.293151    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:11:57.294440 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.294426    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:11:57.296922 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.296903    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 15 21:11:57.297480 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.297463    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:11:57.297656 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.297643    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5l6w9\""
Sep 15 21:11:57.298077 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.298060    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 15 21:11:57.298348 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.298329    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 15 21:11:57.300504 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.300485    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-bhlld"]
Sep 15 21:11:57.304626 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.304604    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 15 21:11:57.304958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.304938    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-4d7vv"]
Sep 15 21:11:57.379863 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.379829    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96e8185d-5c74-4462-88ef-a4c4e53f65f5-config\") pod \"console-operator-696884fb9c-4d7vv\" (UID: \"96e8185d-5c74-4462-88ef-a4c4e53f65f5\") " pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:11:57.380041 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.379879    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db2l2\" (UniqueName: \"kubernetes.io/projected/36b7bbb9-15c6-47e8-af15-84b227e35766-kube-api-access-db2l2\") pod \"cluster-samples-operator-86499c89c4-2kvmt\" (UID: \"36b7bbb9-15c6-47e8-af15-84b227e35766\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt"
Sep 15 21:11:57.380041 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.379897    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96e8185d-5c74-4462-88ef-a4c4e53f65f5-serving-cert\") pod \"console-operator-696884fb9c-4d7vv\" (UID: \"96e8185d-5c74-4462-88ef-a4c4e53f65f5\") " pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:11:57.380041 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.379934    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l587l\" (UniqueName: \"kubernetes.io/projected/96e8185d-5c74-4462-88ef-a4c4e53f65f5-kube-api-access-l587l\") pod \"console-operator-696884fb9c-4d7vv\" (UID: \"96e8185d-5c74-4462-88ef-a4c4e53f65f5\") " pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:11:57.380041 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.379959    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96e8185d-5c74-4462-88ef-a4c4e53f65f5-trusted-ca\") pod \"console-operator-696884fb9c-4d7vv\" (UID: \"96e8185d-5c74-4462-88ef-a4c4e53f65f5\") " pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:11:57.380041 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.379987    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dvj5\" (UniqueName: \"kubernetes.io/projected/8270c2c6-29f0-4ce5-b4a6-ec3b322eccaf-kube-api-access-9dvj5\") pod \"volume-data-source-validator-86695b7cc7-bhlld\" (UID: \"8270c2c6-29f0-4ce5-b4a6-ec3b322eccaf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-bhlld"
Sep 15 21:11:57.380259 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.380032    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2kvmt\" (UID: \"36b7bbb9-15c6-47e8-af15-84b227e35766\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt"
Sep 15 21:11:57.380259 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:57.380145    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 21:11:57.380259 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:57.380200    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls podName:36b7bbb9-15c6-47e8-af15-84b227e35766 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:57.880185384 +0000 UTC m=+85.742119167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2kvmt" (UID: "36b7bbb9-15c6-47e8-af15-84b227e35766") : secret "samples-operator-tls" not found
Sep 15 21:11:57.388181 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.388155    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db2l2\" (UniqueName: \"kubernetes.io/projected/36b7bbb9-15c6-47e8-af15-84b227e35766-kube-api-access-db2l2\") pod \"cluster-samples-operator-86499c89c4-2kvmt\" (UID: \"36b7bbb9-15c6-47e8-af15-84b227e35766\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt"
Sep 15 21:11:57.480836 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.480747    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96e8185d-5c74-4462-88ef-a4c4e53f65f5-trusted-ca\") pod \"console-operator-696884fb9c-4d7vv\" (UID: \"96e8185d-5c74-4462-88ef-a4c4e53f65f5\") " pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:11:57.480836 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.480798    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dvj5\" (UniqueName: \"kubernetes.io/projected/8270c2c6-29f0-4ce5-b4a6-ec3b322eccaf-kube-api-access-9dvj5\") pod \"volume-data-source-validator-86695b7cc7-bhlld\" (UID: \"8270c2c6-29f0-4ce5-b4a6-ec3b322eccaf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-bhlld"
Sep 15 21:11:57.481042 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.480858    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96e8185d-5c74-4462-88ef-a4c4e53f65f5-config\") pod \"console-operator-696884fb9c-4d7vv\" (UID: \"96e8185d-5c74-4462-88ef-a4c4e53f65f5\") " pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:11:57.481042 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.480881    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96e8185d-5c74-4462-88ef-a4c4e53f65f5-serving-cert\") pod \"console-operator-696884fb9c-4d7vv\" (UID: \"96e8185d-5c74-4462-88ef-a4c4e53f65f5\") " pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:11:57.481042 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.480942    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l587l\" (UniqueName: \"kubernetes.io/projected/96e8185d-5c74-4462-88ef-a4c4e53f65f5-kube-api-access-l587l\") pod \"console-operator-696884fb9c-4d7vv\" (UID: \"96e8185d-5c74-4462-88ef-a4c4e53f65f5\") " pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:11:57.481549 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.481532    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96e8185d-5c74-4462-88ef-a4c4e53f65f5-config\") pod \"console-operator-696884fb9c-4d7vv\" (UID: \"96e8185d-5c74-4462-88ef-a4c4e53f65f5\") " pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:11:57.481698 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.481680    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96e8185d-5c74-4462-88ef-a4c4e53f65f5-trusted-ca\") pod \"console-operator-696884fb9c-4d7vv\" (UID: \"96e8185d-5c74-4462-88ef-a4c4e53f65f5\") " pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:11:57.483106 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.483086    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96e8185d-5c74-4462-88ef-a4c4e53f65f5-serving-cert\") pod \"console-operator-696884fb9c-4d7vv\" (UID: \"96e8185d-5c74-4462-88ef-a4c4e53f65f5\") " pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:11:57.489446 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.489424    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dvj5\" (UniqueName: \"kubernetes.io/projected/8270c2c6-29f0-4ce5-b4a6-ec3b322eccaf-kube-api-access-9dvj5\") pod \"volume-data-source-validator-86695b7cc7-bhlld\" (UID: \"8270c2c6-29f0-4ce5-b4a6-ec3b322eccaf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-bhlld"
Sep 15 21:11:57.489522 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.489463    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l587l\" (UniqueName: \"kubernetes.io/projected/96e8185d-5c74-4462-88ef-a4c4e53f65f5-kube-api-access-l587l\") pod \"console-operator-696884fb9c-4d7vv\" (UID: \"96e8185d-5c74-4462-88ef-a4c4e53f65f5\") " pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:11:57.599483 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.599450    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-bhlld"
Sep 15 21:11:57.606468 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.606440    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:11:57.876118 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.876091    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-bhlld"]
Sep 15 21:11:57.878787 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:11:57.878761    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8270c2c6_29f0_4ce5_b4a6_ec3b322eccaf.slice/crio-3b459617d2c6f6bedad28e5ea0e0d1e256b195c8c3ab2d1fc8a39bd4d8b62689 WatchSource:0}: Error finding container 3b459617d2c6f6bedad28e5ea0e0d1e256b195c8c3ab2d1fc8a39bd4d8b62689: Status 404 returned error can't find the container with id 3b459617d2c6f6bedad28e5ea0e0d1e256b195c8c3ab2d1fc8a39bd4d8b62689
Sep 15 21:11:57.882094 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.882073    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-4d7vv"]
Sep 15 21:11:57.884490 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.884473    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2kvmt\" (UID: \"36b7bbb9-15c6-47e8-af15-84b227e35766\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt"
Sep 15 21:11:57.884637 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:57.884620    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 21:11:57.884693 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:11:57.884677    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e8185d_5c74_4462_88ef_a4c4e53f65f5.slice/crio-3e42ec9c6187d79ded013e2a4e02bb595b5fa875c909aba70a6076d8d30bc771 WatchSource:0}: Error finding container 3e42ec9c6187d79ded013e2a4e02bb595b5fa875c909aba70a6076d8d30bc771: Status 404 returned error can't find the container with id 3e42ec9c6187d79ded013e2a4e02bb595b5fa875c909aba70a6076d8d30bc771
Sep 15 21:11:57.884693 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:57.884684    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls podName:36b7bbb9-15c6-47e8-af15-84b227e35766 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:58.884669583 +0000 UTC m=+86.746603371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2kvmt" (UID: "36b7bbb9-15c6-47e8-af15-84b227e35766") : secret "samples-operator-tls" not found
Sep 15 21:11:57.944923 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.944892    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-bhlld" event={"ID":"8270c2c6-29f0-4ce5-b4a6-ec3b322eccaf","Type":"ContainerStarted","Data":"3b459617d2c6f6bedad28e5ea0e0d1e256b195c8c3ab2d1fc8a39bd4d8b62689"}
Sep 15 21:11:57.946000 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:57.945975    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv" event={"ID":"96e8185d-5c74-4462-88ef-a4c4e53f65f5","Type":"ContainerStarted","Data":"3e42ec9c6187d79ded013e2a4e02bb595b5fa875c909aba70a6076d8d30bc771"}
Sep 15 21:11:58.388173 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:58.388138    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:58.388546 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:58.388229    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:11:58.388546 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:58.388288    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:58.388546 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:58.388349    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls podName:2a0964c1-a9bf-466a-a96e-c51abcdb59cc nodeName:}" failed. No retries permitted until 2026-09-15 21:12:06.388333817 +0000 UTC m=+94.250267603 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4mgr8" (UID: "2a0964c1-a9bf-466a-a96e-c51abcdb59cc") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:11:58.388546 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:58.388297    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:11:58.388546 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:58.388394    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm podName:2a0964c1-a9bf-466a-a96e-c51abcdb59cc nodeName:}" failed. No retries permitted until 2026-09-15 21:12:06.388373777 +0000 UTC m=+94.250307562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4mgr8" (UID: "2a0964c1-a9bf-466a-a96e-c51abcdb59cc") : configmap "kube-rbac-proxy" not found
Sep 15 21:11:58.892222 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:11:58.892184    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2kvmt\" (UID: \"36b7bbb9-15c6-47e8-af15-84b227e35766\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt"
Sep 15 21:11:58.892426 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:58.892351    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 21:11:58.892486 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:11:58.892434    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls podName:36b7bbb9-15c6-47e8-af15-84b227e35766 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:00.892414149 +0000 UTC m=+88.754347951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2kvmt" (UID: "36b7bbb9-15c6-47e8-af15-84b227e35766") : secret "samples-operator-tls" not found
Sep 15 21:12:00.911020 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:00.910983    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2kvmt\" (UID: \"36b7bbb9-15c6-47e8-af15-84b227e35766\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt"
Sep 15 21:12:00.911406 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:00.911134    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 21:12:00.911406 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:00.911200    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls podName:36b7bbb9-15c6-47e8-af15-84b227e35766 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:04.911181352 +0000 UTC m=+92.773115154 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2kvmt" (UID: "36b7bbb9-15c6-47e8-af15-84b227e35766") : secret "samples-operator-tls" not found
Sep 15 21:12:00.954832 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:00.954798    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-bhlld" event={"ID":"8270c2c6-29f0-4ce5-b4a6-ec3b322eccaf","Type":"ContainerStarted","Data":"49c772da2a5fdc066f05e6dd02142f6d25a4ad928bbb2059e3bbad2ae6cdaa39"}
Sep 15 21:12:00.956364 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:00.956345    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/0.log"
Sep 15 21:12:00.956466 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:00.956399    2584 generic.go:356] "Generic (PLEG): container finished" podID="96e8185d-5c74-4462-88ef-a4c4e53f65f5" containerID="c6de37eff28037fd2f5f68c50e70543e32d916c20e0b815192546436328560ed" exitCode=255
Sep 15 21:12:00.956466 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:00.956449    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv" event={"ID":"96e8185d-5c74-4462-88ef-a4c4e53f65f5","Type":"ContainerDied","Data":"c6de37eff28037fd2f5f68c50e70543e32d916c20e0b815192546436328560ed"}
Sep 15 21:12:00.956635 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:00.956622    2584 scope.go:117] "RemoveContainer" containerID="c6de37eff28037fd2f5f68c50e70543e32d916c20e0b815192546436328560ed"
Sep 15 21:12:00.969687 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:00.969649    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-bhlld" podStartSLOduration=1.8693229040000001 podStartE2EDuration="3.969635977s" totalImagesPullingTime="2.100313073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:00.968813999 +0000 UTC m=+88.830747803" watchObservedRunningTime="2026-09-15 21:12:00.969635977 +0000 UTC m=+88.831569781"
Sep 15 21:12:01.960831 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:01.960804    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/1.log"
Sep 15 21:12:01.961228 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:01.961213    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/0.log"
Sep 15 21:12:01.961264 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:01.961247    2584 generic.go:356] "Generic (PLEG): container finished" podID="96e8185d-5c74-4462-88ef-a4c4e53f65f5" containerID="e9da45f2232abea6d808bef7a74405c64465d963db0c20b732dafa256286af57" exitCode=255
Sep 15 21:12:01.961352 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:01.961338    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv" event={"ID":"96e8185d-5c74-4462-88ef-a4c4e53f65f5","Type":"ContainerDied","Data":"e9da45f2232abea6d808bef7a74405c64465d963db0c20b732dafa256286af57"}
Sep 15 21:12:01.961412 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:01.961400    2584 scope.go:117] "RemoveContainer" containerID="c6de37eff28037fd2f5f68c50e70543e32d916c20e0b815192546436328560ed"
Sep 15 21:12:01.961541 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:01.961517    2584 scope.go:117] "RemoveContainer" containerID="e9da45f2232abea6d808bef7a74405c64465d963db0c20b732dafa256286af57"
Sep 15 21:12:01.961697 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:01.961674    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-4d7vv_openshift-console-operator(96e8185d-5c74-4462-88ef-a4c4e53f65f5)\"" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv" podUID="96e8185d-5c74-4462-88ef-a4c4e53f65f5"
Sep 15 21:12:02.965135 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:02.965109    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/1.log"
Sep 15 21:12:02.965520 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:02.965373    2584 scope.go:117] "RemoveContainer" containerID="e9da45f2232abea6d808bef7a74405c64465d963db0c20b732dafa256286af57"
Sep 15 21:12:02.965561 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:02.965523    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-4d7vv_openshift-console-operator(96e8185d-5c74-4462-88ef-a4c4e53f65f5)\"" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv" podUID="96e8185d-5c74-4462-88ef-a4c4e53f65f5"
Sep 15 21:12:03.027058 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.027025    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:12:03.027058 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.027058    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:12:03.027269 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.027121    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mfp8r\" (UID: \"306fd13e-6e53-43f4-9903-e0a886c375e8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:12:03.027269 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:03.027165    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 21:12:03.027269 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:03.027233    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs podName:63176ac2-9b27-4a17-88f4-0a934db86c97 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:19.027208899 +0000 UTC m=+106.889142695 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs") pod "router-default-547bb554b6-j9fkz" (UID: "63176ac2-9b27-4a17-88f4-0a934db86c97") : secret "router-metrics-certs-default" not found
Sep 15 21:12:03.027424 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:03.027300    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle podName:63176ac2-9b27-4a17-88f4-0a934db86c97 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:19.027282488 +0000 UTC m=+106.889216271 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle") pod "router-default-547bb554b6-j9fkz" (UID: "63176ac2-9b27-4a17-88f4-0a934db86c97") : configmap references non-existent config key: service-ca.crt
Sep 15 21:12:03.030701 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:03.027968    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 21:12:03.030701 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:03.028673    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls podName:306fd13e-6e53-43f4-9903-e0a886c375e8 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:19.028653725 +0000 UTC m=+106.890587513 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mfp8r" (UID: "306fd13e-6e53-43f4-9903-e0a886c375e8") : secret "cluster-monitoring-operator-tls" not found
Sep 15 21:12:03.729255 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.729223    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6x9kn"]
Sep 15 21:12:03.733364 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.733348    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-6x9kn"
Sep 15 21:12:03.735796 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.735762    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 15 21:12:03.735796 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.735787    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 15 21:12:03.736821 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.736797    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-72d58\""
Sep 15 21:12:03.736821 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.736818    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 15 21:12:03.736931 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.736823    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 15 21:12:03.739346 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.739324    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6x9kn"]
Sep 15 21:12:03.833216 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.833179    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9ab05b08-b7f6-433f-bc96-5ac79a32814d-signing-cabundle\") pod \"service-ca-568869fdc5-6x9kn\" (UID: \"9ab05b08-b7f6-433f-bc96-5ac79a32814d\") " pod="openshift-service-ca/service-ca-568869fdc5-6x9kn"
Sep 15 21:12:03.833216 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.833221    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9ab05b08-b7f6-433f-bc96-5ac79a32814d-signing-key\") pod \"service-ca-568869fdc5-6x9kn\" (UID: \"9ab05b08-b7f6-433f-bc96-5ac79a32814d\") " pod="openshift-service-ca/service-ca-568869fdc5-6x9kn"
Sep 15 21:12:03.833430 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.833241    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfn89\" (UniqueName: \"kubernetes.io/projected/9ab05b08-b7f6-433f-bc96-5ac79a32814d-kube-api-access-nfn89\") pod \"service-ca-568869fdc5-6x9kn\" (UID: \"9ab05b08-b7f6-433f-bc96-5ac79a32814d\") " pod="openshift-service-ca/service-ca-568869fdc5-6x9kn"
Sep 15 21:12:03.933902 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.933870    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9ab05b08-b7f6-433f-bc96-5ac79a32814d-signing-cabundle\") pod \"service-ca-568869fdc5-6x9kn\" (UID: \"9ab05b08-b7f6-433f-bc96-5ac79a32814d\") " pod="openshift-service-ca/service-ca-568869fdc5-6x9kn"
Sep 15 21:12:03.934035 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.933915    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9ab05b08-b7f6-433f-bc96-5ac79a32814d-signing-key\") pod \"service-ca-568869fdc5-6x9kn\" (UID: \"9ab05b08-b7f6-433f-bc96-5ac79a32814d\") " pod="openshift-service-ca/service-ca-568869fdc5-6x9kn"
Sep 15 21:12:03.934035 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.933935    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfn89\" (UniqueName: \"kubernetes.io/projected/9ab05b08-b7f6-433f-bc96-5ac79a32814d-kube-api-access-nfn89\") pod \"service-ca-568869fdc5-6x9kn\" (UID: \"9ab05b08-b7f6-433f-bc96-5ac79a32814d\") " pod="openshift-service-ca/service-ca-568869fdc5-6x9kn"
Sep 15 21:12:03.934522 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.934500    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9ab05b08-b7f6-433f-bc96-5ac79a32814d-signing-cabundle\") pod \"service-ca-568869fdc5-6x9kn\" (UID: \"9ab05b08-b7f6-433f-bc96-5ac79a32814d\") " pod="openshift-service-ca/service-ca-568869fdc5-6x9kn"
Sep 15 21:12:03.936198 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.936172    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9ab05b08-b7f6-433f-bc96-5ac79a32814d-signing-key\") pod \"service-ca-568869fdc5-6x9kn\" (UID: \"9ab05b08-b7f6-433f-bc96-5ac79a32814d\") " pod="openshift-service-ca/service-ca-568869fdc5-6x9kn"
Sep 15 21:12:03.941864 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:03.941844    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfn89\" (UniqueName: \"kubernetes.io/projected/9ab05b08-b7f6-433f-bc96-5ac79a32814d-kube-api-access-nfn89\") pod \"service-ca-568869fdc5-6x9kn\" (UID: \"9ab05b08-b7f6-433f-bc96-5ac79a32814d\") " pod="openshift-service-ca/service-ca-568869fdc5-6x9kn"
Sep 15 21:12:04.041738 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:04.041714    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-6x9kn"
Sep 15 21:12:04.307540 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:04.307463    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6x9kn"]
Sep 15 21:12:04.311743 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:12:04.311712    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab05b08_b7f6_433f_bc96_5ac79a32814d.slice/crio-072529808d8789a470a7fc6951bd4414ee2e3461c67dd30d19848eccc53b74f8 WatchSource:0}: Error finding container 072529808d8789a470a7fc6951bd4414ee2e3461c67dd30d19848eccc53b74f8: Status 404 returned error can't find the container with id 072529808d8789a470a7fc6951bd4414ee2e3461c67dd30d19848eccc53b74f8
Sep 15 21:12:04.942201 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:04.942141    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2kvmt\" (UID: \"36b7bbb9-15c6-47e8-af15-84b227e35766\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt"
Sep 15 21:12:04.942397 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:04.942252    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 21:12:04.942397 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:04.942344    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls podName:36b7bbb9-15c6-47e8-af15-84b227e35766 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:12.942323189 +0000 UTC m=+100.804256977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2kvmt" (UID: "36b7bbb9-15c6-47e8-af15-84b227e35766") : secret "samples-operator-tls" not found
Sep 15 21:12:04.972478 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:04.972444    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-6x9kn" event={"ID":"9ab05b08-b7f6-433f-bc96-5ac79a32814d","Type":"ContainerStarted","Data":"072529808d8789a470a7fc6951bd4414ee2e3461c67dd30d19848eccc53b74f8"}
Sep 15 21:12:06.455226 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:06.455134    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:12:06.455226 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:06.455211    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:12:06.455635 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:06.455282    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:06.455635 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:06.455347    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls podName:2a0964c1-a9bf-466a-a96e-c51abcdb59cc nodeName:}" failed. No retries permitted until 2026-09-15 21:12:22.455331084 +0000 UTC m=+110.317264866 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4mgr8" (UID: "2a0964c1-a9bf-466a-a96e-c51abcdb59cc") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:06.455781 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:06.455763    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:12:06.980476 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:06.980436    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-6x9kn" event={"ID":"9ab05b08-b7f6-433f-bc96-5ac79a32814d","Type":"ContainerStarted","Data":"defdfea30757b8fe2b0f0323704c98730111078287e19b8751914629504d7462"}
Sep 15 21:12:06.997227 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:06.997175    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-6x9kn" podStartSLOduration=2.250004307 podStartE2EDuration="3.997158571s" totalImagesPullingTime="1.747154264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:06.995761597 +0000 UTC m=+94.857695402" watchObservedRunningTime="2026-09-15 21:12:06.997158571 +0000 UTC m=+94.859092374"
Sep 15 21:12:07.607006 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:07.606962    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:12:07.607430 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:07.607067    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:12:07.607503 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:07.607454    2584 scope.go:117] "RemoveContainer" containerID="e9da45f2232abea6d808bef7a74405c64465d963db0c20b732dafa256286af57"
Sep 15 21:12:07.607626 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:07.607606    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-4d7vv_openshift-console-operator(96e8185d-5c74-4462-88ef-a4c4e53f65f5)\"" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv" podUID="96e8185d-5c74-4462-88ef-a4c4e53f65f5"
Sep 15 21:12:08.873860 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:08.873822    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:12:08.874205 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:08.873961    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:12:08.874205 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:08.874025    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls podName:62fa88e4-9983-484f-87b7-65a070871dfa nodeName:}" failed. No retries permitted until 2026-09-15 21:13:12.874008352 +0000 UTC m=+160.735942134 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls") pod "dns-default-vg2k9" (UID: "62fa88e4-9983-484f-87b7-65a070871dfa") : secret "dns-default-metrics-tls" not found
Sep 15 21:12:13.006643 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:13.006609    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2kvmt\" (UID: \"36b7bbb9-15c6-47e8-af15-84b227e35766\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt"
Sep 15 21:12:13.008941 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:13.008909    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/36b7bbb9-15c6-47e8-af15-84b227e35766-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2kvmt\" (UID: \"36b7bbb9-15c6-47e8-af15-84b227e35766\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt"
Sep 15 21:12:13.105090 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:13.105059    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt"
Sep 15 21:12:13.373961 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:13.373822    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt"]
Sep 15 21:12:14.003957 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:14.003924    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt" event={"ID":"36b7bbb9-15c6-47e8-af15-84b227e35766","Type":"ContainerStarted","Data":"d757f47d356a035eac36cf15b8a1881eed7cee6f135c130c02a106350498bc6a"}
Sep 15 21:12:14.914438 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:14.914406    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9dgmv"
Sep 15 21:12:16.012657 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:16.012588    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt" event={"ID":"36b7bbb9-15c6-47e8-af15-84b227e35766","Type":"ContainerStarted","Data":"6b235713dad326683f0de173033b51417678ad1c286acc64c90de7a0dc076443"}
Sep 15 21:12:16.012657 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:16.012638    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt" event={"ID":"36b7bbb9-15c6-47e8-af15-84b227e35766","Type":"ContainerStarted","Data":"77fd8ddb91b5015e9a89a16693bf2debcfc0aae28f20091c199e66edce6a34a4"}
Sep 15 21:12:16.029011 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:16.028971    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2kvmt" podStartSLOduration=16.87562138 podStartE2EDuration="19.028957349s" totalImagesPullingTime="2.153335969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:16.028299008 +0000 UTC m=+103.890232810" watchObservedRunningTime="2026-09-15 21:12:16.028957349 +0000 UTC m=+103.890891200"
Sep 15 21:12:19.053909 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:19.053865    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:12:19.054417 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:19.053964    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:12:19.054417 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:19.053997    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mfp8r\" (UID: \"306fd13e-6e53-43f4-9903-e0a886c375e8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:12:19.054606 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:19.054585    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63176ac2-9b27-4a17-88f4-0a934db86c97-service-ca-bundle\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:12:19.056098 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:19.056072    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63176ac2-9b27-4a17-88f4-0a934db86c97-metrics-certs\") pod \"router-default-547bb554b6-j9fkz\" (UID: \"63176ac2-9b27-4a17-88f4-0a934db86c97\") " pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:12:19.056177 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:19.056135    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/306fd13e-6e53-43f4-9903-e0a886c375e8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mfp8r\" (UID: \"306fd13e-6e53-43f4-9903-e0a886c375e8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:12:19.343293 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:19.343205    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"
Sep 15 21:12:19.348486 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:19.348462    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:12:19.624981 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:19.624905    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r"]
Sep 15 21:12:19.627975 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:12:19.627945    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306fd13e_6e53_43f4_9903_e0a886c375e8.slice/crio-2dde5a6db94cf803b781c533780bfe536143ba7f2233aae810a7103587cf5528 WatchSource:0}: Error finding container 2dde5a6db94cf803b781c533780bfe536143ba7f2233aae810a7103587cf5528: Status 404 returned error can't find the container with id 2dde5a6db94cf803b781c533780bfe536143ba7f2233aae810a7103587cf5528
Sep 15 21:12:19.628156 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:19.628138    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-547bb554b6-j9fkz"]
Sep 15 21:12:19.630684 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:12:19.630666    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63176ac2_9b27_4a17_88f4_0a934db86c97.slice/crio-1605e6cac4dbf334cc45967fb79697ce35a19ef1a3b445022069c0ec30000941 WatchSource:0}: Error finding container 1605e6cac4dbf334cc45967fb79697ce35a19ef1a3b445022069c0ec30000941: Status 404 returned error can't find the container with id 1605e6cac4dbf334cc45967fb79697ce35a19ef1a3b445022069c0ec30000941
Sep 15 21:12:20.027755 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:20.027721    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-547bb554b6-j9fkz" event={"ID":"63176ac2-9b27-4a17-88f4-0a934db86c97","Type":"ContainerStarted","Data":"7fcb3df1c6c3a890ec5c851216d5a74870003220c5a3d657f1c7aa0be6c49677"}
Sep 15 21:12:20.027919 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:20.027766    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-547bb554b6-j9fkz" event={"ID":"63176ac2-9b27-4a17-88f4-0a934db86c97","Type":"ContainerStarted","Data":"1605e6cac4dbf334cc45967fb79697ce35a19ef1a3b445022069c0ec30000941"}
Sep 15 21:12:20.028845 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:20.028821    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r" event={"ID":"306fd13e-6e53-43f4-9903-e0a886c375e8","Type":"ContainerStarted","Data":"2dde5a6db94cf803b781c533780bfe536143ba7f2233aae810a7103587cf5528"}
Sep 15 21:12:20.047411 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:20.047354    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-547bb554b6-j9fkz" podStartSLOduration=33.047341955 podStartE2EDuration="33.047341955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:20.046233255 +0000 UTC m=+107.908167056" watchObservedRunningTime="2026-09-15 21:12:20.047341955 +0000 UTC m=+107.909275758"
Sep 15 21:12:20.349320 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:20.349238    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:12:20.352304 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:20.352280    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:12:21.032245 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:21.032215    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:12:21.033498 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:21.033474    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-547bb554b6-j9fkz"
Sep 15 21:12:22.036253 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:22.036215    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r" event={"ID":"306fd13e-6e53-43f4-9903-e0a886c375e8","Type":"ContainerStarted","Data":"4d4c0bc88278c694ec8a8e38a5a72035e440bd792d0117db9fd1427327eb7bba"}
Sep 15 21:12:22.052688 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:22.052639    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mfp8r" podStartSLOduration=33.452306532 podStartE2EDuration="35.05262559s" totalImagesPullingTime="1.600319058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:22.051081044 +0000 UTC m=+109.913014847" watchObservedRunningTime="2026-09-15 21:12:22.05262559 +0000 UTC m=+109.914559393"
Sep 15 21:12:22.481576 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:22.481485    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:12:22.483551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:22.483532    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a0964c1-a9bf-466a-a96e-c51abcdb59cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mgr8\" (UID: \"2a0964c1-a9bf-466a-a96e-c51abcdb59cc\") " pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:12:22.667751 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:22.667723    2584 scope.go:117] "RemoveContainer" containerID="e9da45f2232abea6d808bef7a74405c64465d963db0c20b732dafa256286af57"
Sep 15 21:12:22.756654 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:22.756574    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4mgr8"
Sep 15 21:12:23.022190 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:23.022166    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4mgr8"]
Sep 15 21:12:23.025149 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:12:23.025114    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0964c1_a9bf_466a_a96e_c51abcdb59cc.slice/crio-490cb3a849b858340ef1921258c89979b2bf6294b9a7f45bb248a818aa22ee74 WatchSource:0}: Error finding container 490cb3a849b858340ef1921258c89979b2bf6294b9a7f45bb248a818aa22ee74: Status 404 returned error can't find the container with id 490cb3a849b858340ef1921258c89979b2bf6294b9a7f45bb248a818aa22ee74
Sep 15 21:12:23.040237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:23.040212    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4mgr8" event={"ID":"2a0964c1-a9bf-466a-a96e-c51abcdb59cc","Type":"ContainerStarted","Data":"490cb3a849b858340ef1921258c89979b2bf6294b9a7f45bb248a818aa22ee74"}
Sep 15 21:12:23.041698 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:23.041681    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:12:23.042058 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:23.042041    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/1.log"
Sep 15 21:12:23.042143 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:23.042071    2584 generic.go:356] "Generic (PLEG): container finished" podID="96e8185d-5c74-4462-88ef-a4c4e53f65f5" containerID="937cbbd4514ac7465913149ffa3d2090b2a406904a43d7d559e8cb3c19d3d70b" exitCode=255
Sep 15 21:12:23.042182 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:23.042143    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv" event={"ID":"96e8185d-5c74-4462-88ef-a4c4e53f65f5","Type":"ContainerDied","Data":"937cbbd4514ac7465913149ffa3d2090b2a406904a43d7d559e8cb3c19d3d70b"}
Sep 15 21:12:23.042222 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:23.042188    2584 scope.go:117] "RemoveContainer" containerID="e9da45f2232abea6d808bef7a74405c64465d963db0c20b732dafa256286af57"
Sep 15 21:12:23.042490 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:23.042473    2584 scope.go:117] "RemoveContainer" containerID="937cbbd4514ac7465913149ffa3d2090b2a406904a43d7d559e8cb3c19d3d70b"
Sep 15 21:12:23.042635 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:23.042613    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-4d7vv_openshift-console-operator(96e8185d-5c74-4462-88ef-a4c4e53f65f5)\"" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv" podUID="96e8185d-5c74-4462-88ef-a4c4e53f65f5"
Sep 15 21:12:24.046209 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:24.046177    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4mgr8" event={"ID":"2a0964c1-a9bf-466a-a96e-c51abcdb59cc","Type":"ContainerStarted","Data":"6209d16cad61c13bf6e78c4bd77f55cae3b8924be28669c14c28608db0eb67eb"}
Sep 15 21:12:24.047549 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:24.047531    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:12:25.052594 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:25.052554    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4mgr8" event={"ID":"2a0964c1-a9bf-466a-a96e-c51abcdb59cc","Type":"ContainerStarted","Data":"6005b00cafcb5a61af9a4d368067ee0a08bc0f41f4fd8cd7dd8375c1b60bc025"}
Sep 15 21:12:27.060689 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:27.060656    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4mgr8" event={"ID":"2a0964c1-a9bf-466a-a96e-c51abcdb59cc","Type":"ContainerStarted","Data":"769864dee54acd1cb9cd35be1d01269b68382ccefb11f7e79aefc7ed1ac07d30"}
Sep 15 21:12:27.078922 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:27.078878    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4mgr8" podStartSLOduration=34.18650701 podStartE2EDuration="37.078863555s" totalImagesPullingTime="2.892356545s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:11:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:27.078374209 +0000 UTC m=+114.940308012" watchObservedRunningTime="2026-09-15 21:12:27.078863555 +0000 UTC m=+114.940797358"
Sep 15 21:12:27.606693 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:27.606657    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:12:27.606872 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:27.606722    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:12:27.607069 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:27.607051    2584 scope.go:117] "RemoveContainer" containerID="937cbbd4514ac7465913149ffa3d2090b2a406904a43d7d559e8cb3c19d3d70b"
Sep 15 21:12:27.607251 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:27.607230    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-4d7vv_openshift-console-operator(96e8185d-5c74-4462-88ef-a4c4e53f65f5)\"" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv" podUID="96e8185d-5c74-4462-88ef-a4c4e53f65f5"
Sep 15 21:12:28.560072 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.560035    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69c485f746-vlgs4"]
Sep 15 21:12:28.563301 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.563278    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.565312 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.565295    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dnxk5\""
Sep 15 21:12:28.565514 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.565494    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 21:12:28.565724 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.565707    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 21:12:28.566063 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.566046    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 21:12:28.569731 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.569711    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 21:12:28.582097 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.582075    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69c485f746-vlgs4"]
Sep 15 21:12:28.726000 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.725969    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-registry-tls\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.726000 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.726000    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-ca-trust-extracted\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.726209 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.726040    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-trusted-ca\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.726209 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.726089    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hv4p\" (UniqueName: \"kubernetes.io/projected/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-kube-api-access-2hv4p\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.726209 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.726123    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-image-registry-private-configuration\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.726209 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.726151    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-installation-pull-secrets\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.726209 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.726172    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-registry-certificates\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.726431 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.726227    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-bound-sa-token\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.827498 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.827414    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-registry-tls\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.827498 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.827450    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-ca-trust-extracted\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.827686 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.827657    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-trusted-ca\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.827723 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.827693    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hv4p\" (UniqueName: \"kubernetes.io/projected/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-kube-api-access-2hv4p\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.827768 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.827720    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-image-registry-private-configuration\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.827768 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.827758    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-installation-pull-secrets\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.827859 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.827783    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-ca-trust-extracted\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.827859 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.827802    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-registry-certificates\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.827859 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.827829    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-bound-sa-token\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.828568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.828544    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-registry-certificates\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.828703 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.828689    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-trusted-ca\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.829927 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.829898    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-registry-tls\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.830526 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.830500    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-installation-pull-secrets\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.830648 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.830548    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-image-registry-private-configuration\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.835309 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.835282    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-bound-sa-token\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.836600 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.836578    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hv4p\" (UniqueName: \"kubernetes.io/projected/ddd07ea7-ab52-4446-a110-4d9ea6c2cc77-kube-api-access-2hv4p\") pod \"image-registry-69c485f746-vlgs4\" (UID: \"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77\") " pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:28.871657 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:28.871633    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:29.151532 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:12:29.151503    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd07ea7_ab52_4446_a110_4d9ea6c2cc77.slice/crio-b201772a8312e1f5886930750d1296d8b0079428f28c25ca89d30a27bcc80499 WatchSource:0}: Error finding container b201772a8312e1f5886930750d1296d8b0079428f28c25ca89d30a27bcc80499: Status 404 returned error can't find the container with id b201772a8312e1f5886930750d1296d8b0079428f28c25ca89d30a27bcc80499
Sep 15 21:12:29.157747 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:29.157721    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69c485f746-vlgs4"]
Sep 15 21:12:30.071934 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:30.071902    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c485f746-vlgs4" event={"ID":"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77","Type":"ContainerStarted","Data":"12ed7ab803503be6424f05cc21a1a80da3544b489dfc165de9c3ed44ed4e1da7"}
Sep 15 21:12:30.072339 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:30.071954    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c485f746-vlgs4" event={"ID":"ddd07ea7-ab52-4446-a110-4d9ea6c2cc77","Type":"ContainerStarted","Data":"b201772a8312e1f5886930750d1296d8b0079428f28c25ca89d30a27bcc80499"}
Sep 15 21:12:30.072339 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:30.071989    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:12:30.092862 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:30.092822    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69c485f746-vlgs4" podStartSLOduration=2.092806929 podStartE2EDuration="2.092806929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:30.091918254 +0000 UTC m=+117.953852061" watchObservedRunningTime="2026-09-15 21:12:30.092806929 +0000 UTC m=+117.954740741"
Sep 15 21:12:38.269079 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.269039    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5tvnc"]
Sep 15 21:12:38.272637 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.272619    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.276331 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.276301    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 21:12:38.278345 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.278324    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 21:12:38.278511 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.278416    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 21:12:38.278588 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.278519    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 21:12:38.279128 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.279113    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s6wpq\""
Sep 15 21:12:38.395104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.395075    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1913778-4e8f-4c93-9078-48d67f4884c0-root\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.395286 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.395114    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1913778-4e8f-4c93-9078-48d67f4884c0-node-exporter-textfile\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.395286 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.395147    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1913778-4e8f-4c93-9078-48d67f4884c0-metrics-client-ca\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.395286 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.395169    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1913778-4e8f-4c93-9078-48d67f4884c0-node-exporter-wtmp\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.395286 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.395190    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzz8z\" (UniqueName: \"kubernetes.io/projected/f1913778-4e8f-4c93-9078-48d67f4884c0-kube-api-access-rzz8z\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.395286 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.395248    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1913778-4e8f-4c93-9078-48d67f4884c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.395506 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.395311    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1913778-4e8f-4c93-9078-48d67f4884c0-node-exporter-tls\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.395506 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.395393    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1913778-4e8f-4c93-9078-48d67f4884c0-sys\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.395506 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.395426    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1913778-4e8f-4c93-9078-48d67f4884c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.496261 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.496228    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1913778-4e8f-4c93-9078-48d67f4884c0-sys\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.496477 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.496273    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1913778-4e8f-4c93-9078-48d67f4884c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.496477 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.496360    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1913778-4e8f-4c93-9078-48d67f4884c0-root\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.496477 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.496409    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1913778-4e8f-4c93-9078-48d67f4884c0-node-exporter-textfile\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.496665 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.496473    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1913778-4e8f-4c93-9078-48d67f4884c0-sys\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.496665 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.496485    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1913778-4e8f-4c93-9078-48d67f4884c0-root\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.496665 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.496522    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1913778-4e8f-4c93-9078-48d67f4884c0-metrics-client-ca\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.496665 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.496551    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1913778-4e8f-4c93-9078-48d67f4884c0-node-exporter-wtmp\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.496665 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.496575    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzz8z\" (UniqueName: \"kubernetes.io/projected/f1913778-4e8f-4c93-9078-48d67f4884c0-kube-api-access-rzz8z\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.496665 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.496631    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1913778-4e8f-4c93-9078-48d67f4884c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.496952 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.496669    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1913778-4e8f-4c93-9078-48d67f4884c0-node-exporter-tls\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.496952 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.496698    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1913778-4e8f-4c93-9078-48d67f4884c0-node-exporter-wtmp\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.496952 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.496771    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1913778-4e8f-4c93-9078-48d67f4884c0-node-exporter-textfile\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.497101 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.497059    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1913778-4e8f-4c93-9078-48d67f4884c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.497196 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.497173    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1913778-4e8f-4c93-9078-48d67f4884c0-metrics-client-ca\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.498985 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.498959    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1913778-4e8f-4c93-9078-48d67f4884c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.499092 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.498966    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1913778-4e8f-4c93-9078-48d67f4884c0-node-exporter-tls\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.505865 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.505839    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzz8z\" (UniqueName: \"kubernetes.io/projected/f1913778-4e8f-4c93-9078-48d67f4884c0-kube-api-access-rzz8z\") pod \"node-exporter-5tvnc\" (UID: \"f1913778-4e8f-4c93-9078-48d67f4884c0\") " pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.583497 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:38.583459    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5tvnc"
Sep 15 21:12:38.600822 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:12:38.600790    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1913778_4e8f_4c93_9078_48d67f4884c0.slice/crio-7e0a420906e7f35d69605b01a7f79cc5a41863008c76ef33c8487c48c8a5787d WatchSource:0}: Error finding container 7e0a420906e7f35d69605b01a7f79cc5a41863008c76ef33c8487c48c8a5787d: Status 404 returned error can't find the container with id 7e0a420906e7f35d69605b01a7f79cc5a41863008c76ef33c8487c48c8a5787d
Sep 15 21:12:39.103601 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:39.103565    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5tvnc" event={"ID":"f1913778-4e8f-4c93-9078-48d67f4884c0","Type":"ContainerStarted","Data":"7e0a420906e7f35d69605b01a7f79cc5a41863008c76ef33c8487c48c8a5787d"}
Sep 15 21:12:39.666932 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:39.666902    2584 scope.go:117] "RemoveContainer" containerID="937cbbd4514ac7465913149ffa3d2090b2a406904a43d7d559e8cb3c19d3d70b"
Sep 15 21:12:39.667227 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:12:39.667137    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-4d7vv_openshift-console-operator(96e8185d-5c74-4462-88ef-a4c4e53f65f5)\"" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv" podUID="96e8185d-5c74-4462-88ef-a4c4e53f65f5"
Sep 15 21:12:40.108643 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.108612    2584 generic.go:356] "Generic (PLEG): container finished" podID="f1913778-4e8f-4c93-9078-48d67f4884c0" containerID="1f6659187c79ddddca3b5bfbe49cfefaa2219c743a691221b64a15224347d130" exitCode=0
Sep 15 21:12:40.108816 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.108670    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5tvnc" event={"ID":"f1913778-4e8f-4c93-9078-48d67f4884c0","Type":"ContainerDied","Data":"1f6659187c79ddddca3b5bfbe49cfefaa2219c743a691221b64a15224347d130"}
Sep 15 21:12:40.249856 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.249829    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"]
Sep 15 21:12:40.253567 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.253546    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.256687 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.256645    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 15 21:12:40.257033 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.257006    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dhctg\""
Sep 15 21:12:40.257121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.257101    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 15 21:12:40.257121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.257110    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 15 21:12:40.257239 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.257118    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2ctq3nemglmik\""
Sep 15 21:12:40.257370 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.257355    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 15 21:12:40.257445 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.257354    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 15 21:12:40.264077 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.264056    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"]
Sep 15 21:12:40.412156 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.412053    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/701bffe8-9eff-4e17-943c-6e005cf74022-metrics-client-ca\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.412156 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.412100    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-thanos-querier-tls\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.412156 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.412126    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgwks\" (UniqueName: \"kubernetes.io/projected/701bffe8-9eff-4e17-943c-6e005cf74022-kube-api-access-zgwks\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.412156 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.412155    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.412447 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.412225    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.412447 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.412278    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.412447 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.412335    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.412447 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.412374    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-grpc-tls\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.513176 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.513133    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/701bffe8-9eff-4e17-943c-6e005cf74022-metrics-client-ca\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.513176 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.513174    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-thanos-querier-tls\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.513375 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.513202    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgwks\" (UniqueName: \"kubernetes.io/projected/701bffe8-9eff-4e17-943c-6e005cf74022-kube-api-access-zgwks\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.513375 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.513328    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.513375 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.513398    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.513542 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.513519    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.513604 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.513586    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.513658 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.513639    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-grpc-tls\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.514039 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.514005    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/701bffe8-9eff-4e17-943c-6e005cf74022-metrics-client-ca\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.516002 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.515976    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.516184 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.516161    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-grpc-tls\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.516267 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.516248    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-thanos-querier-tls\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.516330 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.516253    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.516330 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.516313    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.516442 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.516398    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/701bffe8-9eff-4e17-943c-6e005cf74022-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.525491 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.525472    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgwks\" (UniqueName: \"kubernetes.io/projected/701bffe8-9eff-4e17-943c-6e005cf74022-kube-api-access-zgwks\") pod \"thanos-querier-79ff6dcd-zr4mh\" (UID: \"701bffe8-9eff-4e17-943c-6e005cf74022\") " pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.563214 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.563177    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:40.852951 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:40.852926    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"]
Sep 15 21:12:40.854313 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:12:40.854284    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701bffe8_9eff_4e17_943c_6e005cf74022.slice/crio-58d9888ad618e25f079153e140dd7fe4bf40c3d6c69931fcaa63f7a1376d0f1f WatchSource:0}: Error finding container 58d9888ad618e25f079153e140dd7fe4bf40c3d6c69931fcaa63f7a1376d0f1f: Status 404 returned error can't find the container with id 58d9888ad618e25f079153e140dd7fe4bf40c3d6c69931fcaa63f7a1376d0f1f
Sep 15 21:12:41.115288 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:41.115202    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5tvnc" event={"ID":"f1913778-4e8f-4c93-9078-48d67f4884c0","Type":"ContainerStarted","Data":"8e0544a783e6344fce32c11e1f75dd0355a15d35775211fd6528328681460eee"}
Sep 15 21:12:41.115288 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:41.115266    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5tvnc" event={"ID":"f1913778-4e8f-4c93-9078-48d67f4884c0","Type":"ContainerStarted","Data":"63f24e5a87651d1e47f0306dea98a644fdfadb2886bb4debda4d8a14526a7c98"}
Sep 15 21:12:41.122496 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:41.122469    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh" event={"ID":"701bffe8-9eff-4e17-943c-6e005cf74022","Type":"ContainerStarted","Data":"58d9888ad618e25f079153e140dd7fe4bf40c3d6c69931fcaa63f7a1376d0f1f"}
Sep 15 21:12:41.140306 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:41.140263    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5tvnc" podStartSLOduration=2.076029696 podStartE2EDuration="3.140250662s" totalImagesPullingTime="1.001278561s" totalInitContainerRuntime="62.942405ms" isStatefulPod=true podCreationTimestamp="2026-09-15 21:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:41.139214978 +0000 UTC m=+129.001148784" watchObservedRunningTime="2026-09-15 21:12:41.140250662 +0000 UTC m=+129.002184466"
Sep 15 21:12:42.432911 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.432871    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs\") pod \"network-metrics-daemon-pt55j\" (UID: \"d8ed8d4e-9c80-4109-a61a-440ac53e69e6\") " pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:12:42.435611 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.435585    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8ed8d4e-9c80-4109-a61a-440ac53e69e6-metrics-certs\") pod \"network-metrics-daemon-pt55j\" (UID: \"d8ed8d4e-9c80-4109-a61a-440ac53e69e6\") " pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:12:42.675614 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.675584    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"]
Sep 15 21:12:42.678888 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.678866    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.682117 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.682090    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 15 21:12:42.682224 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.682116    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tnljp\""
Sep 15 21:12:42.682224 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.682139    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 15 21:12:42.682224 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.682158    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 15 21:12:42.682358 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.682106    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 15 21:12:42.682479 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.682457    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-hhbs6cmeoerl\""
Sep 15 21:12:42.687658 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.687423    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8fnh6\""
Sep 15 21:12:42.689636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.689588    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"]
Sep 15 21:12:42.695198 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.695183    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt55j"
Sep 15 21:12:42.836640 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.836618    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcxdk\" (UniqueName: \"kubernetes.io/projected/7bd6c5fd-299c-4553-aae7-173fb0142ad5-kube-api-access-jcxdk\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.836735 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.836668    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7bd6c5fd-299c-4553-aae7-173fb0142ad5-audit-log\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.836735 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.836692    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7bd6c5fd-299c-4553-aae7-173fb0142ad5-secret-metrics-server-tls\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.836803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.836750    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd6c5fd-299c-4553-aae7-173fb0142ad5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.836803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.836797    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7bd6c5fd-299c-4553-aae7-173fb0142ad5-secret-metrics-server-client-certs\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.836870 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.836814    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7bd6c5fd-299c-4553-aae7-173fb0142ad5-metrics-server-audit-profiles\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.836870 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.836831    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7bd6c5fd-299c-4553-aae7-173fb0142ad5-client-ca-bundle\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.925259 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.925233    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-j9smx"]
Sep 15 21:12:42.929424 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.929404    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j9smx"
Sep 15 21:12:42.931542 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.931487    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 15 21:12:42.931542 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.931523    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-kwh96\""
Sep 15 21:12:42.936478 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.936456    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-j9smx"]
Sep 15 21:12:42.937442 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.937363    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7bd6c5fd-299c-4553-aae7-173fb0142ad5-audit-log\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.937536 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.937424    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7bd6c5fd-299c-4553-aae7-173fb0142ad5-secret-metrics-server-tls\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.937536 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.937509    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd6c5fd-299c-4553-aae7-173fb0142ad5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.937652 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.937588    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7bd6c5fd-299c-4553-aae7-173fb0142ad5-secret-metrics-server-client-certs\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.937652 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.937616    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7bd6c5fd-299c-4553-aae7-173fb0142ad5-metrics-server-audit-profiles\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.937652 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.937648    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7bd6c5fd-299c-4553-aae7-173fb0142ad5-client-ca-bundle\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.937796 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.937680    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxdk\" (UniqueName: \"kubernetes.io/projected/7bd6c5fd-299c-4553-aae7-173fb0142ad5-kube-api-access-jcxdk\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.937796 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.937743    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7bd6c5fd-299c-4553-aae7-173fb0142ad5-audit-log\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.938933 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.938912    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7bd6c5fd-299c-4553-aae7-173fb0142ad5-metrics-server-audit-profiles\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.939071 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.939048    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd6c5fd-299c-4553-aae7-173fb0142ad5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.940715 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.940665    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7bd6c5fd-299c-4553-aae7-173fb0142ad5-client-ca-bundle\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.941729 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.941688    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7bd6c5fd-299c-4553-aae7-173fb0142ad5-secret-metrics-server-client-certs\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.941861 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.941840    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7bd6c5fd-299c-4553-aae7-173fb0142ad5-secret-metrics-server-tls\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.948181 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.948119    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxdk\" (UniqueName: \"kubernetes.io/projected/7bd6c5fd-299c-4553-aae7-173fb0142ad5-kube-api-access-jcxdk\") pod \"metrics-server-7d57bdd96c-4hnx6\" (UID: \"7bd6c5fd-299c-4553-aae7-173fb0142ad5\") " pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:42.990431 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:42.990412    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:12:43.024129 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:43.024096    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pt55j"]
Sep 15 21:12:43.038643 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:43.038616    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8h9q\" (UniqueName: \"kubernetes.io/projected/27b7e762-2ec9-4277-8c31-53910792ed85-kube-api-access-t8h9q\") pod \"monitoring-plugin-5d85778b9d-j9smx\" (UID: \"27b7e762-2ec9-4277-8c31-53910792ed85\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j9smx"
Sep 15 21:12:43.038733 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:43.038687    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/27b7e762-2ec9-4277-8c31-53910792ed85-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j9smx\" (UID: \"27b7e762-2ec9-4277-8c31-53910792ed85\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j9smx"
Sep 15 21:12:43.131231 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:43.131199    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh" event={"ID":"701bffe8-9eff-4e17-943c-6e005cf74022","Type":"ContainerStarted","Data":"750d993db564f110143714665d2711659c883e6dafb27acfed7e488b5fd6df35"}
Sep 15 21:12:43.131420 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:43.131282    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh" event={"ID":"701bffe8-9eff-4e17-943c-6e005cf74022","Type":"ContainerStarted","Data":"657fb0281dd27ba19503fcb498da94effad2aef515c9967e04a4de7c401c1960"}
Sep 15 21:12:43.131420 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:43.131315    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh" event={"ID":"701bffe8-9eff-4e17-943c-6e005cf74022","Type":"ContainerStarted","Data":"39b58060514222bdef5240fad4ca670ad4607852e9afffcd3baea4faf5df66b9"}
Sep 15 21:12:43.132238 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:43.132219    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pt55j" event={"ID":"d8ed8d4e-9c80-4109-a61a-440ac53e69e6","Type":"ContainerStarted","Data":"c8ab03386c580acfb0c02a7699dc44d098d6c7a3a253a73acf736d60c8db701f"}
Sep 15 21:12:43.139758 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:43.139737    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8h9q\" (UniqueName: \"kubernetes.io/projected/27b7e762-2ec9-4277-8c31-53910792ed85-kube-api-access-t8h9q\") pod \"monitoring-plugin-5d85778b9d-j9smx\" (UID: \"27b7e762-2ec9-4277-8c31-53910792ed85\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j9smx"
Sep 15 21:12:43.139845 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:43.139781    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/27b7e762-2ec9-4277-8c31-53910792ed85-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j9smx\" (UID: \"27b7e762-2ec9-4277-8c31-53910792ed85\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j9smx"
Sep 15 21:12:43.142104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:43.142083    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/27b7e762-2ec9-4277-8c31-53910792ed85-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j9smx\" (UID: \"27b7e762-2ec9-4277-8c31-53910792ed85\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j9smx"
Sep 15 21:12:43.148134 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:43.148111    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8h9q\" (UniqueName: \"kubernetes.io/projected/27b7e762-2ec9-4277-8c31-53910792ed85-kube-api-access-t8h9q\") pod \"monitoring-plugin-5d85778b9d-j9smx\" (UID: \"27b7e762-2ec9-4277-8c31-53910792ed85\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j9smx"
Sep 15 21:12:43.241106 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:43.241024    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j9smx"
Sep 15 21:12:43.266970 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:43.266940    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"]
Sep 15 21:12:43.269624 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:12:43.269599    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd6c5fd_299c_4553_aae7_173fb0142ad5.slice/crio-0f87ae35e2a22a0a07a7f7e5547477ea14c241e076e367a7adc1c6186fafb5be WatchSource:0}: Error finding container 0f87ae35e2a22a0a07a7f7e5547477ea14c241e076e367a7adc1c6186fafb5be: Status 404 returned error can't find the container with id 0f87ae35e2a22a0a07a7f7e5547477ea14c241e076e367a7adc1c6186fafb5be
Sep 15 21:12:43.514609 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:43.514163    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-j9smx"]
Sep 15 21:12:43.523063 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:12:43.523038    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b7e762_2ec9_4277_8c31_53910792ed85.slice/crio-5253f5a3d1edf695257f154aa76695df70e28be7a9c6447122218842fc6be89d WatchSource:0}: Error finding container 5253f5a3d1edf695257f154aa76695df70e28be7a9c6447122218842fc6be89d: Status 404 returned error can't find the container with id 5253f5a3d1edf695257f154aa76695df70e28be7a9c6447122218842fc6be89d
Sep 15 21:12:44.138975 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.138934    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6" event={"ID":"7bd6c5fd-299c-4553-aae7-173fb0142ad5","Type":"ContainerStarted","Data":"0f87ae35e2a22a0a07a7f7e5547477ea14c241e076e367a7adc1c6186fafb5be"}
Sep 15 21:12:44.140795 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.140718    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j9smx" event={"ID":"27b7e762-2ec9-4277-8c31-53910792ed85","Type":"ContainerStarted","Data":"5253f5a3d1edf695257f154aa76695df70e28be7a9c6447122218842fc6be89d"}
Sep 15 21:12:44.412889 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.412730    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:12:44.417220 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.417194    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.420058 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.420033    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 21:12:44.420173 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.420084    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 21:12:44.420173 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.420127    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8s4gw\""
Sep 15 21:12:44.420173 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.420084    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 21:12:44.420334 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.420235    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 21:12:44.420407 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.420358    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 21:12:44.420517 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.420498    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e1nv0l3t1knej\""
Sep 15 21:12:44.420578 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.420550    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 21:12:44.421507 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.421491    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 21:12:44.421594 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.421517    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 21:12:44.421594 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.421534    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 21:12:44.421949 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.421808    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 21:12:44.421949 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.421846    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 21:12:44.422105 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.421990    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 21:12:44.422593 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.422562    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 21:12:44.423762 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.423728    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 21:12:44.432104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.432082    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:12:44.552166 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552120    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552166 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552167    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552195    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552249    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8141059c-4eed-4a96-94b6-2e3c0664f4cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552280    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552307    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552327    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8141059c-4eed-4a96-94b6-2e3c0664f4cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552342    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552365    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552401    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552439    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-config\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552535    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552559    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552588    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpn8b\" (UniqueName: \"kubernetes.io/projected/8141059c-4eed-4a96-94b6-2e3c0664f4cd-kube-api-access-lpn8b\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552630    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552658    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552680    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.552732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552702    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.553524 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552717    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.553524 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.552774    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.654194 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654158    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.654194 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654197    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.654458 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654226    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.654458 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654267    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.654458 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654353    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.654458 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654410    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.654458 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654443    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8141059c-4eed-4a96-94b6-2e3c0664f4cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.654701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654469    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.654701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654505    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.654701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654536    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8141059c-4eed-4a96-94b6-2e3c0664f4cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.654701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654559    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.654701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654588    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.654701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654610    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.654701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654636    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-config\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.654701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654687    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.655084 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654723    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.655084 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654759    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpn8b\" (UniqueName: \"kubernetes.io/projected/8141059c-4eed-4a96-94b6-2e3c0664f4cd-kube-api-access-lpn8b\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.655084 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654806    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.655084 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654845    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.655084 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.654876    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.655084 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.655053    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.655084 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.655053    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.655459 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.655420    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.655514 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.655454    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.655795 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.655769    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.658336 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.657895    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.658336 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.657924    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.658336 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.658041    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.658336 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.658313    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8141059c-4eed-4a96-94b6-2e3c0664f4cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.658838 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.658712    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8141059c-4eed-4a96-94b6-2e3c0664f4cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.658838 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.658757    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-config\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.659184 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.659163    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.659666 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.659639    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.659973 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.659949    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.660679 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.660643    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.661309 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.661287    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.661498 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.661477    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.661925 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.661903    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.662176 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.662155    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.665867 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.665818    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpn8b\" (UniqueName: \"kubernetes.io/projected/8141059c-4eed-4a96-94b6-2e3c0664f4cd-kube-api-access-lpn8b\") pod \"prometheus-k8s-0\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:44.730241 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:44.730199    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:45.425857 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:45.425831    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:12:45.427115 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:12:45.427091    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8141059c_4eed_4a96_94b6_2e3c0664f4cd.slice/crio-d71ebd10c673dba6858c8102234811953c59750d4b67609f2bdf0988cc8a3752 WatchSource:0}: Error finding container d71ebd10c673dba6858c8102234811953c59750d4b67609f2bdf0988cc8a3752: Status 404 returned error can't find the container with id d71ebd10c673dba6858c8102234811953c59750d4b67609f2bdf0988cc8a3752
Sep 15 21:12:46.152090 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:46.152047    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j9smx" event={"ID":"27b7e762-2ec9-4277-8c31-53910792ed85","Type":"ContainerStarted","Data":"5b19a80a1424b604c894fee9558bea748e827db286ef13f287addf252fc04f98"}
Sep 15 21:12:46.152555 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:46.152162    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j9smx"
Sep 15 21:12:46.153446 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:46.153420    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8141059c-4eed-4a96-94b6-2e3c0664f4cd","Type":"ContainerStarted","Data":"d71ebd10c673dba6858c8102234811953c59750d4b67609f2bdf0988cc8a3752"}
Sep 15 21:12:46.156797 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:46.156764    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh" event={"ID":"701bffe8-9eff-4e17-943c-6e005cf74022","Type":"ContainerStarted","Data":"75206d012cc20fbb23e5de5522a494970ca6e944b6d9cd126c8fb941a36442dd"}
Sep 15 21:12:46.156904 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:46.156831    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh" event={"ID":"701bffe8-9eff-4e17-943c-6e005cf74022","Type":"ContainerStarted","Data":"8b6ec360dc02b8ca1d7f608320b25a52875a3d729d9c5233a0c7882d17cb8857"}
Sep 15 21:12:46.156904 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:46.156876    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh" event={"ID":"701bffe8-9eff-4e17-943c-6e005cf74022","Type":"ContainerStarted","Data":"3676c7569935b5a6a3d1528a363685803c8682f620a85b125bf1385cb6011581"}
Sep 15 21:12:46.157038 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:46.156923    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:46.157038 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:46.156988    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j9smx"
Sep 15 21:12:46.158306 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:46.158281    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6" event={"ID":"7bd6c5fd-299c-4553-aae7-173fb0142ad5","Type":"ContainerStarted","Data":"67ab70456086bd6f18185820d7b4c59b07ef7f3608c7b9da75ad42131c7ae572"}
Sep 15 21:12:46.160092 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:46.160072    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pt55j" event={"ID":"d8ed8d4e-9c80-4109-a61a-440ac53e69e6","Type":"ContainerStarted","Data":"f85093171063e8fb917d470ac618c8818c9e6a35b7ce5e7fb7b6a644f1263ccc"}
Sep 15 21:12:46.160191 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:46.160112    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pt55j" event={"ID":"d8ed8d4e-9c80-4109-a61a-440ac53e69e6","Type":"ContainerStarted","Data":"27e45b4d7f85453f58fd20c061d32b0484b804b8dfcbffd1ee84a63510fd035d"}
Sep 15 21:12:46.171951 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:46.171910    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j9smx" podStartSLOduration=2.559133927 podStartE2EDuration="4.171896838s" totalImagesPullingTime="1.612762911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:46.170803274 +0000 UTC m=+134.032737079" watchObservedRunningTime="2026-09-15 21:12:46.171896838 +0000 UTC m=+134.033830643"
Sep 15 21:12:46.195900 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:46.195862    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh" podStartSLOduration=2.123616422 podStartE2EDuration="6.19584834s" totalImagesPullingTime="4.072231918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:46.19517296 +0000 UTC m=+134.057106764" watchObservedRunningTime="2026-09-15 21:12:46.19584834 +0000 UTC m=+134.057782145"
Sep 15 21:12:46.212661 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:46.212619    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6" podStartSLOduration=2.348977199 podStartE2EDuration="4.212600337s" totalImagesPullingTime="1.863623138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:46.211253091 +0000 UTC m=+134.073186897" watchObservedRunningTime="2026-09-15 21:12:46.212600337 +0000 UTC m=+134.074534141"
Sep 15 21:12:46.243031 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:46.242991    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pt55j" podStartSLOduration=132.148842049 podStartE2EDuration="2m14.242976939s" totalImagesPullingTime="2.09413489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:46.241458271 +0000 UTC m=+134.103392077" watchObservedRunningTime="2026-09-15 21:12:46.242976939 +0000 UTC m=+134.104910744"
Sep 15 21:12:47.166666 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:47.166635    2584 generic.go:356] "Generic (PLEG): container finished" podID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerID="6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954" exitCode=0
Sep 15 21:12:47.167046 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:47.166729    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8141059c-4eed-4a96-94b6-2e3c0664f4cd","Type":"ContainerDied","Data":"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954"}
Sep 15 21:12:47.173931 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:47.173834    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-79ff6dcd-zr4mh"
Sep 15 21:12:48.876801 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:48.876759    2584 patch_prober.go:28] interesting pod/image-registry-69c485f746-vlgs4 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 21:12:48.877218 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:48.876827    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-69c485f746-vlgs4" podUID="ddd07ea7-ab52-4446-a110-4d9ea6c2cc77" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:12:50.666922 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:50.666889    2584 scope.go:117] "RemoveContainer" containerID="937cbbd4514ac7465913149ffa3d2090b2a406904a43d7d559e8cb3c19d3d70b"
Sep 15 21:12:51.080657 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:51.080622    2584 patch_prober.go:28] interesting pod/image-registry-69c485f746-vlgs4 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 21:12:51.080782 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:51.080686    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69c485f746-vlgs4" podUID="ddd07ea7-ab52-4446-a110-4d9ea6c2cc77" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:12:51.188479 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:51.188370    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8141059c-4eed-4a96-94b6-2e3c0664f4cd","Type":"ContainerStarted","Data":"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d"}
Sep 15 21:12:51.188479 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:51.188455    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8141059c-4eed-4a96-94b6-2e3c0664f4cd","Type":"ContainerStarted","Data":"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d"}
Sep 15 21:12:51.188669 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:51.188497    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8141059c-4eed-4a96-94b6-2e3c0664f4cd","Type":"ContainerStarted","Data":"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e"}
Sep 15 21:12:51.188669 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:51.188535    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8141059c-4eed-4a96-94b6-2e3c0664f4cd","Type":"ContainerStarted","Data":"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d"}
Sep 15 21:12:51.190412 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:51.190369    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:12:51.190493 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:51.190467    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv" event={"ID":"96e8185d-5c74-4462-88ef-a4c4e53f65f5","Type":"ContainerStarted","Data":"7b2a0bcbbdb148f284a4e9fbac2a9013cb5c54c60d071a694d4ece9dbea9fa96"}
Sep 15 21:12:51.190805 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:51.190768    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:12:51.209701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:51.209645    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv" podStartSLOduration=52.114315472 podStartE2EDuration="54.209619807s" totalImagesPullingTime="2.095304335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:51.208479158 +0000 UTC m=+139.070413032" watchObservedRunningTime="2026-09-15 21:12:51.209619807 +0000 UTC m=+139.071553613"
Sep 15 21:12:51.506166 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:51.506093    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-4d7vv"
Sep 15 21:12:52.197222 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:52.197183    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8141059c-4eed-4a96-94b6-2e3c0664f4cd","Type":"ContainerStarted","Data":"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194"}
Sep 15 21:12:52.197607 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:52.197249    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8141059c-4eed-4a96-94b6-2e3c0664f4cd","Type":"ContainerStarted","Data":"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55"}
Sep 15 21:12:52.225603 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:52.225542    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.030578407 podStartE2EDuration="8.225521523s" totalImagesPullingTime="5.176345815s" totalInitContainerRuntime="18.597301ms" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:52.223446321 +0000 UTC m=+140.085380134" watchObservedRunningTime="2026-09-15 21:12:52.225521523 +0000 UTC m=+140.087455330"
Sep 15 21:12:54.730447 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:54.730398    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:12:58.875938 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:58.875903    2584 patch_prober.go:28] interesting pod/image-registry-69c485f746-vlgs4 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 21:12:58.876368 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:12:58.875954    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-69c485f746-vlgs4" podUID="ddd07ea7-ab52-4446-a110-4d9ea6c2cc77" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:13:01.079161 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:01.079135    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69c485f746-vlgs4"
Sep 15 21:13:02.990713 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:02.990684    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:13:02.991165 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:02.990741    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:13:08.038310 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:13:08.038264    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vg2k9" podUID="62fa88e4-9983-484f-87b7-65a070871dfa"
Sep 15 21:13:08.256607 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:08.256578    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:13:12.892949 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:12.892913    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:13:12.895126 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:12.895106    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62fa88e4-9983-484f-87b7-65a070871dfa-metrics-tls\") pod \"dns-default-vg2k9\" (UID: \"62fa88e4-9983-484f-87b7-65a070871dfa\") " pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:13:13.059748 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:13.059720    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nfdpp\""
Sep 15 21:13:13.067458 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:13.067435    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:13:13.340198 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:13.340173    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vg2k9"]
Sep 15 21:13:13.342760 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:13:13.342735    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fa88e4_9983_484f_87b7_65a070871dfa.slice/crio-40d1cb4a6d4a1cd180bb068a297166d406cb1054054de10e9e27ffa7d349e4bd WatchSource:0}: Error finding container 40d1cb4a6d4a1cd180bb068a297166d406cb1054054de10e9e27ffa7d349e4bd: Status 404 returned error can't find the container with id 40d1cb4a6d4a1cd180bb068a297166d406cb1054054de10e9e27ffa7d349e4bd
Sep 15 21:13:14.282814 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:14.282780    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vg2k9" event={"ID":"62fa88e4-9983-484f-87b7-65a070871dfa","Type":"ContainerStarted","Data":"40d1cb4a6d4a1cd180bb068a297166d406cb1054054de10e9e27ffa7d349e4bd"}
Sep 15 21:13:23.003938 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:22.996969    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:13:23.008533 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:23.008505    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7d57bdd96c-4hnx6"
Sep 15 21:13:24.322483 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:24.322450    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vg2k9" event={"ID":"62fa88e4-9983-484f-87b7-65a070871dfa","Type":"ContainerStarted","Data":"c2059b6d90750030bf0d4f1ddbb630f579571f2698b04a979d8b7cf7f90efd43"}
Sep 15 21:13:24.322851 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:24.322502    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vg2k9" event={"ID":"62fa88e4-9983-484f-87b7-65a070871dfa","Type":"ContainerStarted","Data":"4bfb55205046416a9a54d6ea665323b6cd978226e8f199bc10a4f15352cac648"}
Sep 15 21:13:24.322851 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:24.322562    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:13:24.342095 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:24.342040    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vg2k9" podStartSLOduration=129.851744614 podStartE2EDuration="2m20.342024345s" totalImagesPullingTime="10.490279731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:24.341604868 +0000 UTC m=+172.203538674" watchObservedRunningTime="2026-09-15 21:13:24.342024345 +0000 UTC m=+172.203958150"
Sep 15 21:13:34.328587 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:34.328508    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vg2k9"
Sep 15 21:13:44.731330 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:44.731297    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:44.746459 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:44.746436    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:13:45.425411 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:13:45.425365    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:02.783131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:02.783096    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:14:02.783667 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:02.783358    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="prometheus" containerID="cri-o://cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d" gracePeriod=600
Sep 15 21:14:02.783667 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:02.783391    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="kube-rbac-proxy" containerID="cri-o://f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55" gracePeriod=600
Sep 15 21:14:02.783667 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:02.783444    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="kube-rbac-proxy-thanos" containerID="cri-o://45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194" gracePeriod=600
Sep 15 21:14:02.783667 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:02.783432    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="thanos-sidecar" containerID="cri-o://56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d" gracePeriod=600
Sep 15 21:14:02.783667 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:02.783445    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="kube-rbac-proxy-web" containerID="cri-o://195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d" gracePeriod=600
Sep 15 21:14:02.783667 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:02.783485    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="config-reloader" containerID="cri-o://29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e" gracePeriod=600
Sep 15 21:14:03.020717 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.020681    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.125543 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.125469    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-serving-certs-ca-bundle\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.125543 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.125519    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-web-config\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.125700 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.125566    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-tls\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.125700 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.125593    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-metrics-client-ca\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.125700 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.125626    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-thanos-prometheus-http-client-file\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.125838 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.125751    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-kubelet-serving-ca-bundle\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.125838 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.125799    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8141059c-4eed-4a96-94b6-2e3c0664f4cd-tls-assets\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.125922 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.125836    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-grpc-tls\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.125922 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.125876    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-metrics-client-certs\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.125922 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.125900    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpn8b\" (UniqueName: \"kubernetes.io/projected/8141059c-4eed-4a96-94b6-2e3c0664f4cd-kube-api-access-lpn8b\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.126067 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.125927    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-1\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.126067 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.125959    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-trusted-ca-bundle\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.126067 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.126008    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8141059c-4eed-4a96-94b6-2e3c0664f4cd-config-out\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.126067 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.126033    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.126067 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.126062    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-kube-rbac-proxy\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.126324 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.126091    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-config\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.126324 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.126116    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-2\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.126324 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.126132    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:03.126324 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.126143    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-db\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.126324 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.126182    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:03.126324 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.126218    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-0\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.126324 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.126249    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\" (UID: \"8141059c-4eed-4a96-94b6-2e3c0664f4cd\") "
Sep 15 21:14:03.126716 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.126623    2584 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-metrics-client-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.126716 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.126643    2584 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.126716 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.126640    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:03.126716 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.126658    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:03.127723 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.127476    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:14:03.128125 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.128100    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:03.128463 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.128436    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:03.129168 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.128691    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:03.129168 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.128708    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8141059c-4eed-4a96-94b6-2e3c0664f4cd-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:14:03.129168 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.128834    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:03.129168 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.128940    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:03.129168 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.129153    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:03.129512 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.129486    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:03.130451 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.130426    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:03.130667 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.130644    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8141059c-4eed-4a96-94b6-2e3c0664f4cd-kube-api-access-lpn8b" (OuterVolumeSpecName: "kube-api-access-lpn8b") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "kube-api-access-lpn8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:14:03.130667 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.130644    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8141059c-4eed-4a96-94b6-2e3c0664f4cd-config-out" (OuterVolumeSpecName: "config-out") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:14:03.130951 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.130927    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:03.131356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.131337    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:03.131444 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.131367    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-config" (OuterVolumeSpecName: "config") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:03.138683 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.138658    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-web-config" (OuterVolumeSpecName: "web-config") pod "8141059c-4eed-4a96-94b6-2e3c0664f4cd" (UID: "8141059c-4eed-4a96-94b6-2e3c0664f4cd"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:14:03.227198 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227170    2584 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8141059c-4eed-4a96-94b6-2e3c0664f4cd-tls-assets\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227198 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227193    2584 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-grpc-tls\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227198 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227202    2584 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-metrics-client-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227210    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpn8b\" (UniqueName: \"kubernetes.io/projected/8141059c-4eed-4a96-94b6-2e3c0664f4cd-kube-api-access-lpn8b\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227219    2584 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227227    2584 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227237    2584 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8141059c-4eed-4a96-94b6-2e3c0664f4cd-config-out\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227245    2584 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227254    2584 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-kube-rbac-proxy\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227263    2584 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-config\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227271    2584 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227279    2584 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-db\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227287    2584 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227296    2584 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227304    2584 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8141059c-4eed-4a96-94b6-2e3c0664f4cd-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227312    2584 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-web-config\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227320    2584 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.227416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.227328    2584 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8141059c-4eed-4a96-94b6-2e3c0664f4cd-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:14:03.481882 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.481799    2584 generic.go:356] "Generic (PLEG): container finished" podID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerID="45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194" exitCode=0
Sep 15 21:14:03.481882 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.481825    2584 generic.go:356] "Generic (PLEG): container finished" podID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerID="f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55" exitCode=0
Sep 15 21:14:03.481882 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.481831    2584 generic.go:356] "Generic (PLEG): container finished" podID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerID="195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d" exitCode=0
Sep 15 21:14:03.481882 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.481836    2584 generic.go:356] "Generic (PLEG): container finished" podID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerID="56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d" exitCode=0
Sep 15 21:14:03.481882 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.481841    2584 generic.go:356] "Generic (PLEG): container finished" podID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerID="29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e" exitCode=0
Sep 15 21:14:03.481882 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.481846    2584 generic.go:356] "Generic (PLEG): container finished" podID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerID="cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d" exitCode=0
Sep 15 21:14:03.482286 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.481883    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.482286 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.481882    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8141059c-4eed-4a96-94b6-2e3c0664f4cd","Type":"ContainerDied","Data":"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194"}
Sep 15 21:14:03.482286 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.481959    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8141059c-4eed-4a96-94b6-2e3c0664f4cd","Type":"ContainerDied","Data":"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55"}
Sep 15 21:14:03.482286 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.481988    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8141059c-4eed-4a96-94b6-2e3c0664f4cd","Type":"ContainerDied","Data":"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d"}
Sep 15 21:14:03.482286 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.482015    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8141059c-4eed-4a96-94b6-2e3c0664f4cd","Type":"ContainerDied","Data":"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d"}
Sep 15 21:14:03.482286 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.482028    2584 scope.go:117] "RemoveContainer" containerID="45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194"
Sep 15 21:14:03.482286 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.482054    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8141059c-4eed-4a96-94b6-2e3c0664f4cd","Type":"ContainerDied","Data":"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e"}
Sep 15 21:14:03.482286 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.482093    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8141059c-4eed-4a96-94b6-2e3c0664f4cd","Type":"ContainerDied","Data":"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d"}
Sep 15 21:14:03.482286 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.482135    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8141059c-4eed-4a96-94b6-2e3c0664f4cd","Type":"ContainerDied","Data":"d71ebd10c673dba6858c8102234811953c59750d4b67609f2bdf0988cc8a3752"}
Sep 15 21:14:03.490419 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.490398    2584 scope.go:117] "RemoveContainer" containerID="f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55"
Sep 15 21:14:03.499293 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.499271    2584 scope.go:117] "RemoveContainer" containerID="195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d"
Sep 15 21:14:03.505879 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.505859    2584 scope.go:117] "RemoveContainer" containerID="56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d"
Sep 15 21:14:03.509794 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.509774    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:14:03.513044 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.513030    2584 scope.go:117] "RemoveContainer" containerID="29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e"
Sep 15 21:14:03.516540 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.516520    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:14:03.520235 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.520220    2584 scope.go:117] "RemoveContainer" containerID="cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d"
Sep 15 21:14:03.526998 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.526980    2584 scope.go:117] "RemoveContainer" containerID="6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954"
Sep 15 21:14:03.533212 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.533196    2584 scope.go:117] "RemoveContainer" containerID="45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194"
Sep 15 21:14:03.533498 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:14:03.533472    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194\": container with ID starting with 45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194 not found: ID does not exist" containerID="45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194"
Sep 15 21:14:03.533562 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.533507    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194"} err="failed to get container status \"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194\": rpc error: code = NotFound desc = could not find container \"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194\": container with ID starting with 45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194 not found: ID does not exist"
Sep 15 21:14:03.533562 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.533539    2584 scope.go:117] "RemoveContainer" containerID="f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55"
Sep 15 21:14:03.533792 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:14:03.533774    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55\": container with ID starting with f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55 not found: ID does not exist" containerID="f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55"
Sep 15 21:14:03.533828 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.533798    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55"} err="failed to get container status \"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55\": rpc error: code = NotFound desc = could not find container \"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55\": container with ID starting with f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55 not found: ID does not exist"
Sep 15 21:14:03.533828 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.533813    2584 scope.go:117] "RemoveContainer" containerID="195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d"
Sep 15 21:14:03.534044 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:14:03.534028    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d\": container with ID starting with 195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d not found: ID does not exist" containerID="195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d"
Sep 15 21:14:03.534092 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.534047    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d"} err="failed to get container status \"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d\": rpc error: code = NotFound desc = could not find container \"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d\": container with ID starting with 195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d not found: ID does not exist"
Sep 15 21:14:03.534092 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.534061    2584 scope.go:117] "RemoveContainer" containerID="56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d"
Sep 15 21:14:03.534284 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:14:03.534266    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d\": container with ID starting with 56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d not found: ID does not exist" containerID="56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d"
Sep 15 21:14:03.534320 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.534288    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d"} err="failed to get container status \"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d\": rpc error: code = NotFound desc = could not find container \"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d\": container with ID starting with 56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d not found: ID does not exist"
Sep 15 21:14:03.534320 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.534302    2584 scope.go:117] "RemoveContainer" containerID="29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e"
Sep 15 21:14:03.534595 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:14:03.534580    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e\": container with ID starting with 29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e not found: ID does not exist" containerID="29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e"
Sep 15 21:14:03.534654 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.534597    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e"} err="failed to get container status \"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e\": rpc error: code = NotFound desc = could not find container \"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e\": container with ID starting with 29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e not found: ID does not exist"
Sep 15 21:14:03.534654 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.534608    2584 scope.go:117] "RemoveContainer" containerID="cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d"
Sep 15 21:14:03.534874 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:14:03.534855    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d\": container with ID starting with cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d not found: ID does not exist" containerID="cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d"
Sep 15 21:14:03.534952 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.534884    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d"} err="failed to get container status \"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d\": rpc error: code = NotFound desc = could not find container \"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d\": container with ID starting with cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d not found: ID does not exist"
Sep 15 21:14:03.534952 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.534904    2584 scope.go:117] "RemoveContainer" containerID="6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954"
Sep 15 21:14:03.535243 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:14:03.535226    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954\": container with ID starting with 6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954 not found: ID does not exist" containerID="6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954"
Sep 15 21:14:03.535300 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.535245    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954"} err="failed to get container status \"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954\": rpc error: code = NotFound desc = could not find container \"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954\": container with ID starting with 6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954 not found: ID does not exist"
Sep 15 21:14:03.535300 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.535272    2584 scope.go:117] "RemoveContainer" containerID="45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194"
Sep 15 21:14:03.535521 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.535506    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194"} err="failed to get container status \"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194\": rpc error: code = NotFound desc = could not find container \"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194\": container with ID starting with 45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194 not found: ID does not exist"
Sep 15 21:14:03.535571 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.535521    2584 scope.go:117] "RemoveContainer" containerID="f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55"
Sep 15 21:14:03.535743 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.535725    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55"} err="failed to get container status \"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55\": rpc error: code = NotFound desc = could not find container \"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55\": container with ID starting with f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55 not found: ID does not exist"
Sep 15 21:14:03.535809 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.535746    2584 scope.go:117] "RemoveContainer" containerID="195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d"
Sep 15 21:14:03.535983 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.535957    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d"} err="failed to get container status \"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d\": rpc error: code = NotFound desc = could not find container \"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d\": container with ID starting with 195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d not found: ID does not exist"
Sep 15 21:14:03.536056 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.535984    2584 scope.go:117] "RemoveContainer" containerID="56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d"
Sep 15 21:14:03.536206 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.536190    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d"} err="failed to get container status \"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d\": rpc error: code = NotFound desc = could not find container \"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d\": container with ID starting with 56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d not found: ID does not exist"
Sep 15 21:14:03.536257 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.536206    2584 scope.go:117] "RemoveContainer" containerID="29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e"
Sep 15 21:14:03.536499 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.536481    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e"} err="failed to get container status \"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e\": rpc error: code = NotFound desc = could not find container \"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e\": container with ID starting with 29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e not found: ID does not exist"
Sep 15 21:14:03.536575 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.536501    2584 scope.go:117] "RemoveContainer" containerID="cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d"
Sep 15 21:14:03.536775 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.536758    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d"} err="failed to get container status \"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d\": rpc error: code = NotFound desc = could not find container \"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d\": container with ID starting with cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d not found: ID does not exist"
Sep 15 21:14:03.536828 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.536775    2584 scope.go:117] "RemoveContainer" containerID="6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954"
Sep 15 21:14:03.537038 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.537021    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954"} err="failed to get container status \"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954\": rpc error: code = NotFound desc = could not find container \"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954\": container with ID starting with 6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954 not found: ID does not exist"
Sep 15 21:14:03.537112 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.537040    2584 scope.go:117] "RemoveContainer" containerID="45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194"
Sep 15 21:14:03.537303 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.537286    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194"} err="failed to get container status \"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194\": rpc error: code = NotFound desc = could not find container \"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194\": container with ID starting with 45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194 not found: ID does not exist"
Sep 15 21:14:03.537351 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.537303    2584 scope.go:117] "RemoveContainer" containerID="f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55"
Sep 15 21:14:03.537587 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.537569    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55"} err="failed to get container status \"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55\": rpc error: code = NotFound desc = could not find container \"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55\": container with ID starting with f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55 not found: ID does not exist"
Sep 15 21:14:03.537587 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.537587    2584 scope.go:117] "RemoveContainer" containerID="195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d"
Sep 15 21:14:03.537817 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.537802    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d"} err="failed to get container status \"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d\": rpc error: code = NotFound desc = could not find container \"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d\": container with ID starting with 195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d not found: ID does not exist"
Sep 15 21:14:03.537817 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.537817    2584 scope.go:117] "RemoveContainer" containerID="56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d"
Sep 15 21:14:03.538032 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.538016    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d"} err="failed to get container status \"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d\": rpc error: code = NotFound desc = could not find container \"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d\": container with ID starting with 56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d not found: ID does not exist"
Sep 15 21:14:03.538070 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.538032    2584 scope.go:117] "RemoveContainer" containerID="29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e"
Sep 15 21:14:03.538295 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.538280    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e"} err="failed to get container status \"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e\": rpc error: code = NotFound desc = could not find container \"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e\": container with ID starting with 29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e not found: ID does not exist"
Sep 15 21:14:03.538337 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.538296    2584 scope.go:117] "RemoveContainer" containerID="cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d"
Sep 15 21:14:03.538562 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.538544    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d"} err="failed to get container status \"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d\": rpc error: code = NotFound desc = could not find container \"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d\": container with ID starting with cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d not found: ID does not exist"
Sep 15 21:14:03.538629 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.538563    2584 scope.go:117] "RemoveContainer" containerID="6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954"
Sep 15 21:14:03.538858 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.538836    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954"} err="failed to get container status \"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954\": rpc error: code = NotFound desc = could not find container \"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954\": container with ID starting with 6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954 not found: ID does not exist"
Sep 15 21:14:03.538901 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.538861    2584 scope.go:117] "RemoveContainer" containerID="45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194"
Sep 15 21:14:03.539122 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.539103    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194"} err="failed to get container status \"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194\": rpc error: code = NotFound desc = could not find container \"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194\": container with ID starting with 45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194 not found: ID does not exist"
Sep 15 21:14:03.539168 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.539122    2584 scope.go:117] "RemoveContainer" containerID="f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55"
Sep 15 21:14:03.539367 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.539346    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55"} err="failed to get container status \"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55\": rpc error: code = NotFound desc = could not find container \"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55\": container with ID starting with f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55 not found: ID does not exist"
Sep 15 21:14:03.539441 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.539368    2584 scope.go:117] "RemoveContainer" containerID="195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d"
Sep 15 21:14:03.539628 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.539609    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d"} err="failed to get container status \"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d\": rpc error: code = NotFound desc = could not find container \"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d\": container with ID starting with 195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d not found: ID does not exist"
Sep 15 21:14:03.539693 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.539631    2584 scope.go:117] "RemoveContainer" containerID="56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d"
Sep 15 21:14:03.539878 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.539852    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d"} err="failed to get container status \"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d\": rpc error: code = NotFound desc = could not find container \"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d\": container with ID starting with 56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d not found: ID does not exist"
Sep 15 21:14:03.539878 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.539874    2584 scope.go:117] "RemoveContainer" containerID="29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e"
Sep 15 21:14:03.540147 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.540126    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e"} err="failed to get container status \"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e\": rpc error: code = NotFound desc = could not find container \"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e\": container with ID starting with 29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e not found: ID does not exist"
Sep 15 21:14:03.540147 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.540146    2584 scope.go:117] "RemoveContainer" containerID="cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d"
Sep 15 21:14:03.540440 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.540424    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d"} err="failed to get container status \"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d\": rpc error: code = NotFound desc = could not find container \"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d\": container with ID starting with cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d not found: ID does not exist"
Sep 15 21:14:03.540440 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.540440    2584 scope.go:117] "RemoveContainer" containerID="6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954"
Sep 15 21:14:03.540695 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.540674    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954"} err="failed to get container status \"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954\": rpc error: code = NotFound desc = could not find container \"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954\": container with ID starting with 6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954 not found: ID does not exist"
Sep 15 21:14:03.540759 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.540698    2584 scope.go:117] "RemoveContainer" containerID="45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194"
Sep 15 21:14:03.540945 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.540922    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194"} err="failed to get container status \"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194\": rpc error: code = NotFound desc = could not find container \"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194\": container with ID starting with 45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194 not found: ID does not exist"
Sep 15 21:14:03.540945 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.540940    2584 scope.go:117] "RemoveContainer" containerID="f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55"
Sep 15 21:14:03.541176 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.541157    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55"} err="failed to get container status \"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55\": rpc error: code = NotFound desc = could not find container \"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55\": container with ID starting with f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55 not found: ID does not exist"
Sep 15 21:14:03.541255 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.541178    2584 scope.go:117] "RemoveContainer" containerID="195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d"
Sep 15 21:14:03.541431 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.541414    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d"} err="failed to get container status \"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d\": rpc error: code = NotFound desc = could not find container \"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d\": container with ID starting with 195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d not found: ID does not exist"
Sep 15 21:14:03.541514 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.541432    2584 scope.go:117] "RemoveContainer" containerID="56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d"
Sep 15 21:14:03.541680 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.541663    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d"} err="failed to get container status \"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d\": rpc error: code = NotFound desc = could not find container \"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d\": container with ID starting with 56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d not found: ID does not exist"
Sep 15 21:14:03.541726 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.541680    2584 scope.go:117] "RemoveContainer" containerID="29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e"
Sep 15 21:14:03.541931 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.541910    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e"} err="failed to get container status \"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e\": rpc error: code = NotFound desc = could not find container \"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e\": container with ID starting with 29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e not found: ID does not exist"
Sep 15 21:14:03.541979 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.541933    2584 scope.go:117] "RemoveContainer" containerID="cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d"
Sep 15 21:14:03.542167 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.542150    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d"} err="failed to get container status \"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d\": rpc error: code = NotFound desc = could not find container \"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d\": container with ID starting with cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d not found: ID does not exist"
Sep 15 21:14:03.542245 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.542169    2584 scope.go:117] "RemoveContainer" containerID="6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954"
Sep 15 21:14:03.542647 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.542553    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954"} err="failed to get container status \"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954\": rpc error: code = NotFound desc = could not find container \"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954\": container with ID starting with 6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954 not found: ID does not exist"
Sep 15 21:14:03.542647 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.542581    2584 scope.go:117] "RemoveContainer" containerID="45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194"
Sep 15 21:14:03.542956 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.542932    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194"} err="failed to get container status \"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194\": rpc error: code = NotFound desc = could not find container \"45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194\": container with ID starting with 45393d8ada74933db90dd8965a2e1f086997dbcc8a081185360bdd68b322e194 not found: ID does not exist"
Sep 15 21:14:03.543057 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.542956    2584 scope.go:117] "RemoveContainer" containerID="f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55"
Sep 15 21:14:03.543407 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.543366    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55"} err="failed to get container status \"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55\": rpc error: code = NotFound desc = could not find container \"f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55\": container with ID starting with f979e10d58f2b66dcd602edd9edfa699aa2b01229289a7f04a542405f87d8d55 not found: ID does not exist"
Sep 15 21:14:03.543520 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.543408    2584 scope.go:117] "RemoveContainer" containerID="195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d"
Sep 15 21:14:03.543764 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.543740    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d"} err="failed to get container status \"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d\": rpc error: code = NotFound desc = could not find container \"195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d\": container with ID starting with 195a6d4f940dcb0b79b543692cea36e7ec5b5f24885e49bfa9519ebf81018a1d not found: ID does not exist"
Sep 15 21:14:03.543764 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.543764    2584 scope.go:117] "RemoveContainer" containerID="56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d"
Sep 15 21:14:03.544059 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.544038    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d"} err="failed to get container status \"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d\": rpc error: code = NotFound desc = could not find container \"56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d\": container with ID starting with 56d376bd0e39d42bda9966cbbee1c5002f41f9046987e173e82c2dda0806ba5d not found: ID does not exist"
Sep 15 21:14:03.544059 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.544058    2584 scope.go:117] "RemoveContainer" containerID="29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e"
Sep 15 21:14:03.544317 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.544299    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e"} err="failed to get container status \"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e\": rpc error: code = NotFound desc = could not find container \"29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e\": container with ID starting with 29266d3d6ba070e894bf4169202d996f29e111623daefd1ea423b41f279d446e not found: ID does not exist"
Sep 15 21:14:03.544445 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.544318    2584 scope.go:117] "RemoveContainer" containerID="cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d"
Sep 15 21:14:03.544558 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.544533    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:14:03.544671 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.544558    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d"} err="failed to get container status \"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d\": rpc error: code = NotFound desc = could not find container \"cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d\": container with ID starting with cd781f5cdd3c7ffe8de78a1323bea69419dcddc0111eb594c11c4aeb2ef9472d not found: ID does not exist"
Sep 15 21:14:03.544671 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.544572    2584 scope.go:117] "RemoveContainer" containerID="6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954"
Sep 15 21:14:03.544859 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.544828    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954"} err="failed to get container status \"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954\": rpc error: code = NotFound desc = could not find container \"6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954\": container with ID starting with 6e13c712573e25700cb14fdd9bd9ea6f61e9de7cb4dcd6aed373603e77fb6954 not found: ID does not exist"
Sep 15 21:14:03.544958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.544941    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="kube-rbac-proxy"
Sep 15 21:14:03.545017 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.544962    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="kube-rbac-proxy"
Sep 15 21:14:03.545017 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.544978    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="init-config-reloader"
Sep 15 21:14:03.545017 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.544987    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="init-config-reloader"
Sep 15 21:14:03.545017 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.544996    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="prometheus"
Sep 15 21:14:03.545017 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.545005    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="prometheus"
Sep 15 21:14:03.545237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.545018    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="config-reloader"
Sep 15 21:14:03.545237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.545026    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="config-reloader"
Sep 15 21:14:03.545237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.545039    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="thanos-sidecar"
Sep 15 21:14:03.545237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.545048    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="thanos-sidecar"
Sep 15 21:14:03.545237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.545060    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="kube-rbac-proxy-web"
Sep 15 21:14:03.545237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.545068    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="kube-rbac-proxy-web"
Sep 15 21:14:03.545237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.545079    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="kube-rbac-proxy-thanos"
Sep 15 21:14:03.545237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.545088    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="kube-rbac-proxy-thanos"
Sep 15 21:14:03.545237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.545175    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="thanos-sidecar"
Sep 15 21:14:03.545237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.545185    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="kube-rbac-proxy-web"
Sep 15 21:14:03.545237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.545195    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="prometheus"
Sep 15 21:14:03.545237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.545204    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="config-reloader"
Sep 15 21:14:03.545237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.545212    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="kube-rbac-proxy"
Sep 15 21:14:03.545237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.545222    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" containerName="kube-rbac-proxy-thanos"
Sep 15 21:14:03.550710 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.550692    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.553044 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.553026    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 21:14:03.553244 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.553223    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 21:14:03.553316 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.553227    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 21:14:03.553519 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.553499    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 21:14:03.553613 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.553534    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 21:14:03.553613 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.553547    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 21:14:03.553613 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.553507    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 21:14:03.553613 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.553549    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 21:14:03.553812 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.553507    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8s4gw\""
Sep 15 21:14:03.553812 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.553502    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 21:14:03.553916 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.553841    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 21:14:03.553916 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.553848    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 21:14:03.553916 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.553895    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e1nv0l3t1knej\""
Sep 15 21:14:03.553916 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.553900    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 21:14:03.556815 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.556793    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 21:14:03.560283 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.560263    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 21:14:03.568797 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.568774    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:14:03.631032 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.630999    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drzpz\" (UniqueName: \"kubernetes.io/projected/6de4ec41-0eb5-4e60-b967-c56afe68e625-kube-api-access-drzpz\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631165 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631039    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631165 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631054    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631165 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631106    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6de4ec41-0eb5-4e60-b967-c56afe68e625-config-out\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631165 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631125    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631165 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631140    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631165 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631159    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631351 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631181    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631351 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631198    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-web-config\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631351 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631214    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631351 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631230    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6de4ec41-0eb5-4e60-b967-c56afe68e625-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631351 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631287    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631351 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631311    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-config\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631351 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631329    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631351 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631343    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631622 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631360    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6de4ec41-0eb5-4e60-b967-c56afe68e625-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631622 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631402    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631622 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631416    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631622 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631446    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.631622 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.631481    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732278 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732183    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drzpz\" (UniqueName: \"kubernetes.io/projected/6de4ec41-0eb5-4e60-b967-c56afe68e625-kube-api-access-drzpz\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732278 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732232    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732278 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732254    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732574 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732283    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6de4ec41-0eb5-4e60-b967-c56afe68e625-config-out\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732574 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732306    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732574 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732330    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732574 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732354    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732574 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732516    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732822 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732560    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-web-config\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732822 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732608    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732822 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732640    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6de4ec41-0eb5-4e60-b967-c56afe68e625-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732822 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732680    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732822 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732705    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-config\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732822 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732730    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732822 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732753    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732822 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732776    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6de4ec41-0eb5-4e60-b967-c56afe68e625-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.732822 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.733245 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732831    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.733245 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732857    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.733245 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.732894    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.733245 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.733090    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.733245 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.733108    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.733245 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.733176    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.733915 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.733888    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.734020 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.733913    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.735490 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.735464    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.735596 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.735508    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.735977 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.735900    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6de4ec41-0eb5-4e60-b967-c56afe68e625-config-out\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.735977 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.735943    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6de4ec41-0eb5-4e60-b967-c56afe68e625-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.736422 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.736399    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.736602 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.736535    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.736602 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.736470    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.736602 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.736541    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6de4ec41-0eb5-4e60-b967-c56afe68e625-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.736783 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.736705    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.736783 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.736782    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6de4ec41-0eb5-4e60-b967-c56afe68e625-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.738037 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.738014    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.738111 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.738043    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-web-config\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.738184 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.738162    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-config\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.738506 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.738490    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6de4ec41-0eb5-4e60-b967-c56afe68e625-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.741198 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.741181    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drzpz\" (UniqueName: \"kubernetes.io/projected/6de4ec41-0eb5-4e60-b967-c56afe68e625-kube-api-access-drzpz\") pod \"prometheus-k8s-0\" (UID: \"6de4ec41-0eb5-4e60-b967-c56afe68e625\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:03.861035 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:03.861002    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:14:04.149478 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:04.149452    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:14:04.151309 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:14:04.151280    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de4ec41_0eb5_4e60_b967_c56afe68e625.slice/crio-ca6cabe887b02dda6447718c8613d1f053e4be713abbd6f81da51bcb3161a861 WatchSource:0}: Error finding container ca6cabe887b02dda6447718c8613d1f053e4be713abbd6f81da51bcb3161a861: Status 404 returned error can't find the container with id ca6cabe887b02dda6447718c8613d1f053e4be713abbd6f81da51bcb3161a861
Sep 15 21:14:04.488465 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:04.488360    2584 generic.go:356] "Generic (PLEG): container finished" podID="6de4ec41-0eb5-4e60-b967-c56afe68e625" containerID="c234b856a2f5f41c76d553c339df6583b8039eba76a77ffe719194002fbb6711" exitCode=0
Sep 15 21:14:04.488465 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:04.488445    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6de4ec41-0eb5-4e60-b967-c56afe68e625","Type":"ContainerDied","Data":"c234b856a2f5f41c76d553c339df6583b8039eba76a77ffe719194002fbb6711"}
Sep 15 21:14:04.488766 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:04.488501    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6de4ec41-0eb5-4e60-b967-c56afe68e625","Type":"ContainerStarted","Data":"ca6cabe887b02dda6447718c8613d1f053e4be713abbd6f81da51bcb3161a861"}
Sep 15 21:14:04.671136 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:04.671109    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8141059c-4eed-4a96-94b6-2e3c0664f4cd" path="/var/lib/kubelet/pods/8141059c-4eed-4a96-94b6-2e3c0664f4cd/volumes"
Sep 15 21:14:05.495553 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:05.495518    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6de4ec41-0eb5-4e60-b967-c56afe68e625","Type":"ContainerStarted","Data":"bc045c0e5923e84f155aadceb1df9ebaaa1010aeaf4d99743d9c6b60e220918e"}
Sep 15 21:14:05.495919 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:05.495607    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6de4ec41-0eb5-4e60-b967-c56afe68e625","Type":"ContainerStarted","Data":"19c1cdf3978abadee654a3894244435404ed7f775ee96bf68c06799522cc66d7"}
Sep 15 21:14:05.495919 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:05.495637    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6de4ec41-0eb5-4e60-b967-c56afe68e625","Type":"ContainerStarted","Data":"a03f193b0c43cab5f463844afe98cf36041096a39c231858e1bb76e7e305ffb0"}
Sep 15 21:14:05.495919 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:05.495662    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6de4ec41-0eb5-4e60-b967-c56afe68e625","Type":"ContainerStarted","Data":"64e420d65334ffc6c9760f5a0973d8f3da6a53c56cff61fe206507d14477bf95"}
Sep 15 21:14:05.495919 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:05.495684    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6de4ec41-0eb5-4e60-b967-c56afe68e625","Type":"ContainerStarted","Data":"6beabd0f19e31c5792ce096228d3e501d65c2471ee05a80d2621d6f2ee67bcce"}
Sep 15 21:14:05.495919 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:05.495705    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6de4ec41-0eb5-4e60-b967-c56afe68e625","Type":"ContainerStarted","Data":"aaa0ed696471a606fd71a9165f6391c63066bdfe28a9ac906ed7729ee5852344"}
Sep 15 21:14:05.526243 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:05.526187    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.508268689 podStartE2EDuration="2.526165057s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.896368ms" isStatefulPod=false podCreationTimestamp="2026-09-15 21:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:14:05.523370727 +0000 UTC m=+213.385304546" watchObservedRunningTime="2026-09-15 21:14:05.526165057 +0000 UTC m=+213.388098863"
Sep 15 21:14:08.861241 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:14:08.861205    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:15:03.861487 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:15:03.861397    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:15:03.876720 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:15:03.876696    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:15:04.743604 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:15:04.743578    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:15:32.481339 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:15:32.481303    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:15:32.485277 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:15:32.485252    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:15:32.492953 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:15:32.492931    2584 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 21:16:23.881536 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:23.881505    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"]
Sep 15 21:16:23.885111 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:23.885091    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:23.888876 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:23.888853    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 21:16:23.889348 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:23.889332    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 21:16:23.889874 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:23.889859    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 21:16:23.892031 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:23.892012    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sp8bs\""
Sep 15 21:16:23.892229 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:23.892215    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 21:16:23.904164 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:23.904139    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"]
Sep 15 21:16:23.973469 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:23.973436    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff759f48-2d5b-4c93-809e-8b363ce8bb6d-webhook-cert\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-ptqvt\" (UID: \"ff759f48-2d5b-4c93-809e-8b363ce8bb6d\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:23.973668 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:23.973478    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff759f48-2d5b-4c93-809e-8b363ce8bb6d-manifests\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-ptqvt\" (UID: \"ff759f48-2d5b-4c93-809e-8b363ce8bb6d\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:23.973668 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:23.973537    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ff759f48-2d5b-4c93-809e-8b363ce8bb6d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-ptqvt\" (UID: \"ff759f48-2d5b-4c93-809e-8b363ce8bb6d\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:23.973668 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:23.973564    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff759f48-2d5b-4c93-809e-8b363ce8bb6d-tmp\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-ptqvt\" (UID: \"ff759f48-2d5b-4c93-809e-8b363ce8bb6d\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:23.973668 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:23.973592    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rbvm\" (UniqueName: \"kubernetes.io/projected/ff759f48-2d5b-4c93-809e-8b363ce8bb6d-kube-api-access-8rbvm\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-ptqvt\" (UID: \"ff759f48-2d5b-4c93-809e-8b363ce8bb6d\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:24.074775 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:24.074737    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ff759f48-2d5b-4c93-809e-8b363ce8bb6d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-ptqvt\" (UID: \"ff759f48-2d5b-4c93-809e-8b363ce8bb6d\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:24.074775 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:24.074778    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff759f48-2d5b-4c93-809e-8b363ce8bb6d-tmp\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-ptqvt\" (UID: \"ff759f48-2d5b-4c93-809e-8b363ce8bb6d\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:24.075032 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:24.074813    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rbvm\" (UniqueName: \"kubernetes.io/projected/ff759f48-2d5b-4c93-809e-8b363ce8bb6d-kube-api-access-8rbvm\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-ptqvt\" (UID: \"ff759f48-2d5b-4c93-809e-8b363ce8bb6d\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:24.075032 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:24.074893    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff759f48-2d5b-4c93-809e-8b363ce8bb6d-webhook-cert\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-ptqvt\" (UID: \"ff759f48-2d5b-4c93-809e-8b363ce8bb6d\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:24.075133 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:24.075064    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff759f48-2d5b-4c93-809e-8b363ce8bb6d-manifests\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-ptqvt\" (UID: \"ff759f48-2d5b-4c93-809e-8b363ce8bb6d\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:24.075455 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:24.075429    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff759f48-2d5b-4c93-809e-8b363ce8bb6d-manifests\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-ptqvt\" (UID: \"ff759f48-2d5b-4c93-809e-8b363ce8bb6d\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:24.077202 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:24.077174    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff759f48-2d5b-4c93-809e-8b363ce8bb6d-tmp\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-ptqvt\" (UID: \"ff759f48-2d5b-4c93-809e-8b363ce8bb6d\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:24.077331 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:24.077312    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ff759f48-2d5b-4c93-809e-8b363ce8bb6d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-ptqvt\" (UID: \"ff759f48-2d5b-4c93-809e-8b363ce8bb6d\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:24.077522 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:24.077504    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff759f48-2d5b-4c93-809e-8b363ce8bb6d-webhook-cert\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-ptqvt\" (UID: \"ff759f48-2d5b-4c93-809e-8b363ce8bb6d\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:24.086580 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:24.086557    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rbvm\" (UniqueName: \"kubernetes.io/projected/ff759f48-2d5b-4c93-809e-8b363ce8bb6d-kube-api-access-8rbvm\") pod \"opendatahub-operator-controller-manager-568b8bf8c4-ptqvt\" (UID: \"ff759f48-2d5b-4c93-809e-8b363ce8bb6d\") " pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:24.195452 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:24.195322    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:24.474460 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:24.473897    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"]
Sep 15 21:16:24.480826 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:16:24.480799    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff759f48_2d5b_4c93_809e_8b363ce8bb6d.slice/crio-9e9c0538be3d36baa493c870d11ec7d64ded8c121ab9990de526e2953e33a99b WatchSource:0}: Error finding container 9e9c0538be3d36baa493c870d11ec7d64ded8c121ab9990de526e2953e33a99b: Status 404 returned error can't find the container with id 9e9c0538be3d36baa493c870d11ec7d64ded8c121ab9990de526e2953e33a99b
Sep 15 21:16:24.482533 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:24.482518    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:16:25.036192 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:25.036153    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt" event={"ID":"ff759f48-2d5b-4c93-809e-8b363ce8bb6d","Type":"ContainerStarted","Data":"9e9c0538be3d36baa493c870d11ec7d64ded8c121ab9990de526e2953e33a99b"}
Sep 15 21:16:29.057702 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:29.057660    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt" event={"ID":"ff759f48-2d5b-4c93-809e-8b363ce8bb6d","Type":"ContainerStarted","Data":"42016a03580d6fd607d357ad1520d43a11366eee174908d6431c531eceea095b"}
Sep 15 21:16:30.063641 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:30.063607    2584 generic.go:356] "Generic (PLEG): container finished" podID="ff759f48-2d5b-4c93-809e-8b363ce8bb6d" containerID="42016a03580d6fd607d357ad1520d43a11366eee174908d6431c531eceea095b" exitCode=0
Sep 15 21:16:30.064032 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:30.063693    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt" event={"ID":"ff759f48-2d5b-4c93-809e-8b363ce8bb6d","Type":"ContainerDied","Data":"42016a03580d6fd607d357ad1520d43a11366eee174908d6431c531eceea095b"}
Sep 15 21:16:31.069807 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:31.069762    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt" event={"ID":"ff759f48-2d5b-4c93-809e-8b363ce8bb6d","Type":"ContainerStarted","Data":"0913e572ccd323f8fdb4bd349934a1150002a94af90a6a171d517f6431bd46a7"}
Sep 15 21:16:31.070277 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:31.069912    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:31.091691 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:31.091627    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt" podStartSLOduration=2.934441832 podStartE2EDuration="8.091608019s" totalImagesPullingTime="4.09651742s" totalInitContainerRuntime="1.060648767s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:31.090073129 +0000 UTC m=+358.952006933" watchObservedRunningTime="2026-09-15 21:16:31.091608019 +0000 UTC m=+358.953541826"
Sep 15 21:16:42.076300 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:42.076267    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-568b8bf8c4-ptqvt"
Sep 15 21:16:49.260448 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:49.260412    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb"]
Sep 15 21:16:49.271199 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:49.271172    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb"
Sep 15 21:16:49.274074 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:49.273879    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 21:16:49.275796 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:49.275769    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb"]
Sep 15 21:16:49.275961 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:49.275939    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 21:16:49.391398 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:49.391346    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72956236-ecac-4bda-af44-f973911d4235-tmp\") pod \"kube-auth-proxy-cbb5b4bf7-z86xb\" (UID: \"72956236-ecac-4bda-af44-f973911d4235\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb"
Sep 15 21:16:49.391577 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:49.391466    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4xrw\" (UniqueName: \"kubernetes.io/projected/72956236-ecac-4bda-af44-f973911d4235-kube-api-access-t4xrw\") pod \"kube-auth-proxy-cbb5b4bf7-z86xb\" (UID: \"72956236-ecac-4bda-af44-f973911d4235\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb"
Sep 15 21:16:49.391577 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:49.391497    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72956236-ecac-4bda-af44-f973911d4235-tls-certs\") pod \"kube-auth-proxy-cbb5b4bf7-z86xb\" (UID: \"72956236-ecac-4bda-af44-f973911d4235\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb"
Sep 15 21:16:49.492546 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:49.492506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4xrw\" (UniqueName: \"kubernetes.io/projected/72956236-ecac-4bda-af44-f973911d4235-kube-api-access-t4xrw\") pod \"kube-auth-proxy-cbb5b4bf7-z86xb\" (UID: \"72956236-ecac-4bda-af44-f973911d4235\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb"
Sep 15 21:16:49.492725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:49.492554    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72956236-ecac-4bda-af44-f973911d4235-tls-certs\") pod \"kube-auth-proxy-cbb5b4bf7-z86xb\" (UID: \"72956236-ecac-4bda-af44-f973911d4235\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb"
Sep 15 21:16:49.492725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:49.492623    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72956236-ecac-4bda-af44-f973911d4235-tmp\") pod \"kube-auth-proxy-cbb5b4bf7-z86xb\" (UID: \"72956236-ecac-4bda-af44-f973911d4235\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb"
Sep 15 21:16:49.494853 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:49.494825    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72956236-ecac-4bda-af44-f973911d4235-tmp\") pod \"kube-auth-proxy-cbb5b4bf7-z86xb\" (UID: \"72956236-ecac-4bda-af44-f973911d4235\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb"
Sep 15 21:16:49.495194 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:49.495169    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72956236-ecac-4bda-af44-f973911d4235-tls-certs\") pod \"kube-auth-proxy-cbb5b4bf7-z86xb\" (UID: \"72956236-ecac-4bda-af44-f973911d4235\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb"
Sep 15 21:16:49.501586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:49.501561    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4xrw\" (UniqueName: \"kubernetes.io/projected/72956236-ecac-4bda-af44-f973911d4235-kube-api-access-t4xrw\") pod \"kube-auth-proxy-cbb5b4bf7-z86xb\" (UID: \"72956236-ecac-4bda-af44-f973911d4235\") " pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb"
Sep 15 21:16:49.582856 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:49.582822    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb"
Sep 15 21:16:49.855580 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:49.855540    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb"]
Sep 15 21:16:49.857912 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:16:49.857886    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72956236_ecac_4bda_af44_f973911d4235.slice/crio-8c6ba7955ec5beffd395003377945c509c7a4d5c1424d3d4eba58d2a1ce4ba4b WatchSource:0}: Error finding container 8c6ba7955ec5beffd395003377945c509c7a4d5c1424d3d4eba58d2a1ce4ba4b: Status 404 returned error can't find the container with id 8c6ba7955ec5beffd395003377945c509c7a4d5c1424d3d4eba58d2a1ce4ba4b
Sep 15 21:16:50.149178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.149093    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb" event={"ID":"72956236-ecac-4bda-af44-f973911d4235","Type":"ContainerStarted","Data":"8c6ba7955ec5beffd395003377945c509c7a4d5c1424d3d4eba58d2a1ce4ba4b"}
Sep 15 21:16:50.432953 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.432877    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"]
Sep 15 21:16:50.436134 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.436118    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.438487 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.438459    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 15 21:16:50.438619 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.438600    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 21:16:50.438683 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.438652    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 15 21:16:50.438746 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.438705    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 15 21:16:50.438800 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.438770    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ddskr\""
Sep 15 21:16:50.448156 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.448135    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"]
Sep 15 21:16:50.500877 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.500844    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.500877 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.500879    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.501100 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.500899    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e028f7d-c019-4cac-8613-ec7f25beef71-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.501100 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.500970    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.501100 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.501030    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.501100 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.501066    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.501100 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.501084    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfrw7\" (UniqueName: \"kubernetes.io/projected/4e028f7d-c019-4cac-8613-ec7f25beef71-kube-api-access-rfrw7\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.601846 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.601804    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.602038 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.601851    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.602038 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.601879    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e028f7d-c019-4cac-8613-ec7f25beef71-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.602038 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.601917    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.602038 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.601968    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.602038 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.602013    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.602288 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.602037    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrw7\" (UniqueName: \"kubernetes.io/projected/4e028f7d-c019-4cac-8613-ec7f25beef71-kube-api-access-rfrw7\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.602693 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.602664    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.604442 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.604417    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.604527 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.604457    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e028f7d-c019-4cac-8613-ec7f25beef71-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.604527 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.604510    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.604615 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.604571    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.610933 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.610906    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.611031 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.610966    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfrw7\" (UniqueName: \"kubernetes.io/projected/4e028f7d-c019-4cac-8613-ec7f25beef71-kube-api-access-rfrw7\") pod \"istiod-openshift-gateway-7b7995d5cd-d542c\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:50.745659 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:50.745563    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:51.039872 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:51.039836    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"]
Sep 15 21:16:51.044738 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:16:51.044695    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e028f7d_c019_4cac_8613_ec7f25beef71.slice/crio-2b67d9a1702230758708f4d3aab5b49b2c9ecf2c20c185ca4761c97654ba3f92 WatchSource:0}: Error finding container 2b67d9a1702230758708f4d3aab5b49b2c9ecf2c20c185ca4761c97654ba3f92: Status 404 returned error can't find the container with id 2b67d9a1702230758708f4d3aab5b49b2c9ecf2c20c185ca4761c97654ba3f92
Sep 15 21:16:51.157213 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:51.157172    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c" event={"ID":"4e028f7d-c019-4cac-8613-ec7f25beef71","Type":"ContainerStarted","Data":"2b67d9a1702230758708f4d3aab5b49b2c9ecf2c20c185ca4761c97654ba3f92"}
Sep 15 21:16:53.167595 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:53.167511    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb" event={"ID":"72956236-ecac-4bda-af44-f973911d4235","Type":"ContainerStarted","Data":"8cbc2c8817c44fe056308f96bd94ed4af6298c1fd8100f533197978fce3b7343"}
Sep 15 21:16:53.186521 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:53.186472    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-cbb5b4bf7-z86xb" podStartSLOduration=1.207533555 podStartE2EDuration="4.186455242s" totalImagesPullingTime="2.978921687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:53.184868411 +0000 UTC m=+381.046802229" watchObservedRunningTime="2026-09-15 21:16:53.186455242 +0000 UTC m=+381.048389092"
Sep 15 21:16:55.957319 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:55.957283    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:16:55.957603 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:55.957358    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:16:56.182847 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.182799    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c" event={"ID":"4e028f7d-c019-4cac-8613-ec7f25beef71","Type":"ContainerStarted","Data":"76a33014381b094f043815f8d14b02760ee67f41f8b500f7e10b18526eec0c9e"}
Sep 15 21:16:56.183014 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.182879    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:56.204835 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.204776    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c" podStartSLOduration=1.295741076 podStartE2EDuration="6.204760714s" totalImagesPullingTime="4.909019638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:56.203158338 +0000 UTC m=+384.065092145" watchObservedRunningTime="2026-09-15 21:16:56.204760714 +0000 UTC m=+384.066694538"
Sep 15 21:16:56.642925 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.642885    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"]
Sep 15 21:16:56.646324 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.646304    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.648528 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.648497    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-t5cqb\""
Sep 15 21:16:56.664847 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.664817    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"]
Sep 15 21:16:56.763530 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.763488    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3a7bee-7136-4630-ae69-61061cf917d1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.763530 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.763532    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3a7bee-7136-4630-ae69-61061cf917d1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.763744 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.763550    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be3a7bee-7136-4630-ae69-61061cf917d1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.763744 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.763579    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3a7bee-7136-4630-ae69-61061cf917d1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.763744 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.763634    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3a7bee-7136-4630-ae69-61061cf917d1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.763744 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.763667    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhh42\" (UniqueName: \"kubernetes.io/projected/be3a7bee-7136-4630-ae69-61061cf917d1-kube-api-access-dhh42\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.763744 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.763684    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3a7bee-7136-4630-ae69-61061cf917d1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.763744 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.763726    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3a7bee-7136-4630-ae69-61061cf917d1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.763927 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.763748    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3a7bee-7136-4630-ae69-61061cf917d1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.864866 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.864820    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3a7bee-7136-4630-ae69-61061cf917d1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.865037 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.864874    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhh42\" (UniqueName: \"kubernetes.io/projected/be3a7bee-7136-4630-ae69-61061cf917d1-kube-api-access-dhh42\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.865037 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.864902    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3a7bee-7136-4630-ae69-61061cf917d1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.865113 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.865042    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3a7bee-7136-4630-ae69-61061cf917d1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.865113 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.865080    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3a7bee-7136-4630-ae69-61061cf917d1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.865173 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.865137    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3a7bee-7136-4630-ae69-61061cf917d1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.865173 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.865163    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3a7bee-7136-4630-ae69-61061cf917d1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.865266 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.865189    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be3a7bee-7136-4630-ae69-61061cf917d1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.865266 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.865222    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3a7bee-7136-4630-ae69-61061cf917d1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.865266 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.865232    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3a7bee-7136-4630-ae69-61061cf917d1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.865541 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.865517    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3a7bee-7136-4630-ae69-61061cf917d1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.865636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.865602    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3a7bee-7136-4630-ae69-61061cf917d1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.865688 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.865676    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3a7bee-7136-4630-ae69-61061cf917d1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.865725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.865696    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3a7bee-7136-4630-ae69-61061cf917d1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.867503 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.867485    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3a7bee-7136-4630-ae69-61061cf917d1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.867659 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.867645    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3a7bee-7136-4630-ae69-61061cf917d1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.874180 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.874154    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be3a7bee-7136-4630-ae69-61061cf917d1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.874495 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.874468    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhh42\" (UniqueName: \"kubernetes.io/projected/be3a7bee-7136-4630-ae69-61061cf917d1-kube-api-access-dhh42\") pod \"data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw\" (UID: \"be3a7bee-7136-4630-ae69-61061cf917d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:56.957026 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:56.956924    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:16:57.189280 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:57.189254    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:16:57.234335 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:57.234302    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"]
Sep 15 21:16:57.237051 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:16:57.237013    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3a7bee_7136_4630_ae69_61061cf917d1.slice/crio-a4151dcd3f76865f9f40d0d8557d7bd02e9b5c0046281c69c7a0d200d24b4840 WatchSource:0}: Error finding container a4151dcd3f76865f9f40d0d8557d7bd02e9b5c0046281c69c7a0d200d24b4840: Status 404 returned error can't find the container with id a4151dcd3f76865f9f40d0d8557d7bd02e9b5c0046281c69c7a0d200d24b4840
Sep 15 21:16:58.194684 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:58.194641    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw" event={"ID":"be3a7bee-7136-4630-ae69-61061cf917d1","Type":"ContainerStarted","Data":"a4151dcd3f76865f9f40d0d8557d7bd02e9b5c0046281c69c7a0d200d24b4840"}
Sep 15 21:16:59.797057 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:59.797019    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:16:59.797325 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:59.797100    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:16:59.797325 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:16:59.797130    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:17:00.206863 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:00.206776    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw" event={"ID":"be3a7bee-7136-4630-ae69-61061cf917d1","Type":"ContainerStarted","Data":"a144d87c656b1d5b52152d00315492e9573499a3674649c223e930a2eca10a9c"}
Sep 15 21:17:00.227866 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:00.227813    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw" podStartSLOduration=1.670157302 podStartE2EDuration="4.227796896s" totalImagesPullingTime="2.557639594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:00.2265605 +0000 UTC m=+388.088494340" watchObservedRunningTime="2026-09-15 21:17:00.227796896 +0000 UTC m=+388.089730700"
Sep 15 21:17:00.957289 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:00.957257    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:17:00.961860 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:00.961832    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:17:01.211492 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:01.211409    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:17:01.212340 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:01.212322    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw"
Sep 15 21:17:04.203504 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.203472    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8sdqj"]
Sep 15 21:17:04.206962 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.206938    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8sdqj"
Sep 15 21:17:04.209534 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.209512    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wtctq\""
Sep 15 21:17:04.209661 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.209642    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 15 21:17:04.218629 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.218606    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8sdqj"]
Sep 15 21:17:04.221332 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.221308    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9rss4"]
Sep 15 21:17:04.225166 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.225146    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9rss4"
Sep 15 21:17:04.227537 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.227517    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 15 21:17:04.227537 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.227531    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-v9bhd\""
Sep 15 21:17:04.250139 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.250103    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9rss4"]
Sep 15 21:17:04.349534 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.349498    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c4dh\" (UniqueName: \"kubernetes.io/projected/425cc59c-7511-4a61-9ef3-9021837f32f6-kube-api-access-6c4dh\") pod \"kserve-controller-manager-d6f4b65b8-8sdqj\" (UID: \"425cc59c-7511-4a61-9ef3-9021837f32f6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8sdqj"
Sep 15 21:17:04.349715 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.349558    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/425cc59c-7511-4a61-9ef3-9021837f32f6-cert\") pod \"kserve-controller-manager-d6f4b65b8-8sdqj\" (UID: \"425cc59c-7511-4a61-9ef3-9021837f32f6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8sdqj"
Sep 15 21:17:04.349715 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.349650    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcjx9\" (UniqueName: \"kubernetes.io/projected/9877a946-ae67-4460-a1f9-8b50498d4332-kube-api-access-gcjx9\") pod \"llmisvc-controller-manager-777c6bd7d5-9rss4\" (UID: \"9877a946-ae67-4460-a1f9-8b50498d4332\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9rss4"
Sep 15 21:17:04.349715 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.349693    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9877a946-ae67-4460-a1f9-8b50498d4332-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9rss4\" (UID: \"9877a946-ae67-4460-a1f9-8b50498d4332\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9rss4"
Sep 15 21:17:04.450351 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.450317    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/425cc59c-7511-4a61-9ef3-9021837f32f6-cert\") pod \"kserve-controller-manager-d6f4b65b8-8sdqj\" (UID: \"425cc59c-7511-4a61-9ef3-9021837f32f6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8sdqj"
Sep 15 21:17:04.450351 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.450361    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcjx9\" (UniqueName: \"kubernetes.io/projected/9877a946-ae67-4460-a1f9-8b50498d4332-kube-api-access-gcjx9\") pod \"llmisvc-controller-manager-777c6bd7d5-9rss4\" (UID: \"9877a946-ae67-4460-a1f9-8b50498d4332\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9rss4"
Sep 15 21:17:04.450612 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.450413    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9877a946-ae67-4460-a1f9-8b50498d4332-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9rss4\" (UID: \"9877a946-ae67-4460-a1f9-8b50498d4332\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9rss4"
Sep 15 21:17:04.450612 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.450480    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4dh\" (UniqueName: \"kubernetes.io/projected/425cc59c-7511-4a61-9ef3-9021837f32f6-kube-api-access-6c4dh\") pod \"kserve-controller-manager-d6f4b65b8-8sdqj\" (UID: \"425cc59c-7511-4a61-9ef3-9021837f32f6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8sdqj"
Sep 15 21:17:04.452859 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.452831    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9877a946-ae67-4460-a1f9-8b50498d4332-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9rss4\" (UID: \"9877a946-ae67-4460-a1f9-8b50498d4332\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9rss4"
Sep 15 21:17:04.452958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.452867    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/425cc59c-7511-4a61-9ef3-9021837f32f6-cert\") pod \"kserve-controller-manager-d6f4b65b8-8sdqj\" (UID: \"425cc59c-7511-4a61-9ef3-9021837f32f6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8sdqj"
Sep 15 21:17:04.464339 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.464283    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4dh\" (UniqueName: \"kubernetes.io/projected/425cc59c-7511-4a61-9ef3-9021837f32f6-kube-api-access-6c4dh\") pod \"kserve-controller-manager-d6f4b65b8-8sdqj\" (UID: \"425cc59c-7511-4a61-9ef3-9021837f32f6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8sdqj"
Sep 15 21:17:04.464677 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.464658    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcjx9\" (UniqueName: \"kubernetes.io/projected/9877a946-ae67-4460-a1f9-8b50498d4332-kube-api-access-gcjx9\") pod \"llmisvc-controller-manager-777c6bd7d5-9rss4\" (UID: \"9877a946-ae67-4460-a1f9-8b50498d4332\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9rss4"
Sep 15 21:17:04.518558 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.518521    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8sdqj"
Sep 15 21:17:04.535848 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.535814    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9rss4"
Sep 15 21:17:04.798366 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:17:04.798324    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425cc59c_7511_4a61_9ef3_9021837f32f6.slice/crio-00c170d2bed4baf8028c73480c8076662cfeeec0b37f080e57b2c171fca47f28 WatchSource:0}: Error finding container 00c170d2bed4baf8028c73480c8076662cfeeec0b37f080e57b2c171fca47f28: Status 404 returned error can't find the container with id 00c170d2bed4baf8028c73480c8076662cfeeec0b37f080e57b2c171fca47f28
Sep 15 21:17:04.801875 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.801849    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8sdqj"]
Sep 15 21:17:04.819808 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:04.819789    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9rss4"]
Sep 15 21:17:04.821947 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:17:04.821923    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9877a946_ae67_4460_a1f9_8b50498d4332.slice/crio-b477c3132cf646f1e3a5ea735e7dcdd0bad30ebd1b65c44c6bb2b8f59c4d3b4e WatchSource:0}: Error finding container b477c3132cf646f1e3a5ea735e7dcdd0bad30ebd1b65c44c6bb2b8f59c4d3b4e: Status 404 returned error can't find the container with id b477c3132cf646f1e3a5ea735e7dcdd0bad30ebd1b65c44c6bb2b8f59c4d3b4e
Sep 15 21:17:05.230625 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:05.230546    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8sdqj" event={"ID":"425cc59c-7511-4a61-9ef3-9021837f32f6","Type":"ContainerStarted","Data":"00c170d2bed4baf8028c73480c8076662cfeeec0b37f080e57b2c171fca47f28"}
Sep 15 21:17:05.231634 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:05.231611    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9rss4" event={"ID":"9877a946-ae67-4460-a1f9-8b50498d4332","Type":"ContainerStarted","Data":"b477c3132cf646f1e3a5ea735e7dcdd0bad30ebd1b65c44c6bb2b8f59c4d3b4e"}
Sep 15 21:17:06.564731 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.564693    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"]
Sep 15 21:17:06.565164 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.564901    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c" podUID="4e028f7d-c019-4cac-8613-ec7f25beef71" containerName="discovery" containerID="cri-o://76a33014381b094f043815f8d14b02760ee67f41f8b500f7e10b18526eec0c9e" gracePeriod=30
Sep 15 21:17:06.813575 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.813551    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:17:06.872094 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.872015    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-token\") pod \"4e028f7d-c019-4cac-8613-ec7f25beef71\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") "
Sep 15 21:17:06.872094 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.872062    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-cacerts\") pod \"4e028f7d-c019-4cac-8613-ec7f25beef71\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") "
Sep 15 21:17:06.872094 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.872088    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfrw7\" (UniqueName: \"kubernetes.io/projected/4e028f7d-c019-4cac-8613-ec7f25beef71-kube-api-access-rfrw7\") pod \"4e028f7d-c019-4cac-8613-ec7f25beef71\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") "
Sep 15 21:17:06.872422 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.872134    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-csr-ca-configmap\") pod \"4e028f7d-c019-4cac-8613-ec7f25beef71\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") "
Sep 15 21:17:06.872422 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.872168    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-kubeconfig\") pod \"4e028f7d-c019-4cac-8613-ec7f25beef71\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") "
Sep 15 21:17:06.872422 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.872207    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e028f7d-c019-4cac-8613-ec7f25beef71-local-certs\") pod \"4e028f7d-c019-4cac-8613-ec7f25beef71\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") "
Sep 15 21:17:06.872422 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.872248    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-csr-dns-cert\") pod \"4e028f7d-c019-4cac-8613-ec7f25beef71\" (UID: \"4e028f7d-c019-4cac-8613-ec7f25beef71\") "
Sep 15 21:17:06.872689 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.872651    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "4e028f7d-c019-4cac-8613-ec7f25beef71" (UID: "4e028f7d-c019-4cac-8613-ec7f25beef71"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:17:06.875003 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.874976    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-token" (OuterVolumeSpecName: "istio-token") pod "4e028f7d-c019-4cac-8613-ec7f25beef71" (UID: "4e028f7d-c019-4cac-8613-ec7f25beef71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:17:06.875123 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.875037    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e028f7d-c019-4cac-8613-ec7f25beef71-kube-api-access-rfrw7" (OuterVolumeSpecName: "kube-api-access-rfrw7") pod "4e028f7d-c019-4cac-8613-ec7f25beef71" (UID: "4e028f7d-c019-4cac-8613-ec7f25beef71"). InnerVolumeSpecName "kube-api-access-rfrw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:17:06.875123 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.875043    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "4e028f7d-c019-4cac-8613-ec7f25beef71" (UID: "4e028f7d-c019-4cac-8613-ec7f25beef71"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:17:06.875404 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.875353    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e028f7d-c019-4cac-8613-ec7f25beef71-local-certs" (OuterVolumeSpecName: "local-certs") pod "4e028f7d-c019-4cac-8613-ec7f25beef71" (UID: "4e028f7d-c019-4cac-8613-ec7f25beef71"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:17:06.875657 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.875632    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-cacerts" (OuterVolumeSpecName: "cacerts") pod "4e028f7d-c019-4cac-8613-ec7f25beef71" (UID: "4e028f7d-c019-4cac-8613-ec7f25beef71"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:17:06.875759 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.875742    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "4e028f7d-c019-4cac-8613-ec7f25beef71" (UID: "4e028f7d-c019-4cac-8613-ec7f25beef71"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:17:06.973668 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.973638    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-token\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:17:06.973668 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.973667    2584 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-cacerts\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:17:06.973873 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.973680    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfrw7\" (UniqueName: \"kubernetes.io/projected/4e028f7d-c019-4cac-8613-ec7f25beef71-kube-api-access-rfrw7\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:17:06.973873 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.973696    2584 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-csr-ca-configmap\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:17:06.973873 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.973710    2584 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-kubeconfig\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:17:06.973873 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.973721    2584 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e028f7d-c019-4cac-8613-ec7f25beef71-local-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:17:06.973873 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:06.973732    2584 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4e028f7d-c019-4cac-8613-ec7f25beef71-istio-csr-dns-cert\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:17:07.245679 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:07.245587    2584 generic.go:356] "Generic (PLEG): container finished" podID="4e028f7d-c019-4cac-8613-ec7f25beef71" containerID="76a33014381b094f043815f8d14b02760ee67f41f8b500f7e10b18526eec0c9e" exitCode=0
Sep 15 21:17:07.245679 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:07.245642    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"
Sep 15 21:17:07.245911 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:07.245672    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c" event={"ID":"4e028f7d-c019-4cac-8613-ec7f25beef71","Type":"ContainerDied","Data":"76a33014381b094f043815f8d14b02760ee67f41f8b500f7e10b18526eec0c9e"}
Sep 15 21:17:07.245911 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:07.245745    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c" event={"ID":"4e028f7d-c019-4cac-8613-ec7f25beef71","Type":"ContainerDied","Data":"2b67d9a1702230758708f4d3aab5b49b2c9ecf2c20c185ca4761c97654ba3f92"}
Sep 15 21:17:07.245911 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:07.245775    2584 scope.go:117] "RemoveContainer" containerID="76a33014381b094f043815f8d14b02760ee67f41f8b500f7e10b18526eec0c9e"
Sep 15 21:17:07.269351 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:07.269322    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"]
Sep 15 21:17:07.274019 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:07.273995    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d542c"]
Sep 15 21:17:07.570984 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:07.570814    2584 scope.go:117] "RemoveContainer" containerID="76a33014381b094f043815f8d14b02760ee67f41f8b500f7e10b18526eec0c9e"
Sep 15 21:17:07.571222 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:17:07.571193    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a33014381b094f043815f8d14b02760ee67f41f8b500f7e10b18526eec0c9e\": container with ID starting with 76a33014381b094f043815f8d14b02760ee67f41f8b500f7e10b18526eec0c9e not found: ID does not exist" containerID="76a33014381b094f043815f8d14b02760ee67f41f8b500f7e10b18526eec0c9e"
Sep 15 21:17:07.571313 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:07.571230    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a33014381b094f043815f8d14b02760ee67f41f8b500f7e10b18526eec0c9e"} err="failed to get container status \"76a33014381b094f043815f8d14b02760ee67f41f8b500f7e10b18526eec0c9e\": rpc error: code = NotFound desc = could not find container \"76a33014381b094f043815f8d14b02760ee67f41f8b500f7e10b18526eec0c9e\": container with ID starting with 76a33014381b094f043815f8d14b02760ee67f41f8b500f7e10b18526eec0c9e not found: ID does not exist"
Sep 15 21:17:08.674035 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:08.673965    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e028f7d-c019-4cac-8613-ec7f25beef71" path="/var/lib/kubelet/pods/4e028f7d-c019-4cac-8613-ec7f25beef71/volumes"
Sep 15 21:17:09.258518 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:09.258479    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9rss4" event={"ID":"9877a946-ae67-4460-a1f9-8b50498d4332","Type":"ContainerStarted","Data":"bfddfe4f9e3dd4e575ca86b330b72ecb6a7e1d9a6a07534a24609dab443f1576"}
Sep 15 21:17:09.258710 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:09.258644    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9rss4"
Sep 15 21:17:09.259997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:09.259973    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8sdqj" event={"ID":"425cc59c-7511-4a61-9ef3-9021837f32f6","Type":"ContainerStarted","Data":"c6be63823e2f1a173cfcb594ad358c254219b9d9ef3295575d946b2e9da38c0c"}
Sep 15 21:17:09.260107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:09.260095    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8sdqj"
Sep 15 21:17:09.277554 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:09.277503    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9rss4" podStartSLOduration=1.359848538 podStartE2EDuration="5.277490479s" totalImagesPullingTime="3.917641941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:09.276040511 +0000 UTC m=+397.137974317" watchObservedRunningTime="2026-09-15 21:17:09.277490479 +0000 UTC m=+397.139424284"
Sep 15 21:17:09.294247 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:09.294201    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8sdqj" podStartSLOduration=2.1698891 podStartE2EDuration="5.29419073s" totalImagesPullingTime="3.12430163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:09.292498915 +0000 UTC m=+397.154432719" watchObservedRunningTime="2026-09-15 21:17:09.29419073 +0000 UTC m=+397.156124534"
Sep 15 21:17:40.267159 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:40.267124    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9rss4"
Sep 15 21:17:40.270636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:17:40.270615    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8sdqj"
Sep 15 21:18:13.380199 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:13.380162    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"]
Sep 15 21:18:13.380675 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:13.380539    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e028f7d-c019-4cac-8613-ec7f25beef71" containerName="discovery"
Sep 15 21:18:13.380675 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:13.380553    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e028f7d-c019-4cac-8613-ec7f25beef71" containerName="discovery"
Sep 15 21:18:13.380675 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:13.380628    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e028f7d-c019-4cac-8613-ec7f25beef71" containerName="discovery"
Sep 15 21:18:13.383805 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:13.383787    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:13.386190 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:13.386167    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 21:18:13.386878 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:13.386857    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nqdfj\""
Sep 15 21:18:13.386984 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:13.386891    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 21:18:13.405207 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:13.405183    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"]
Sep 15 21:18:13.468097 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:13.468056    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2jtm\" (UniqueName: \"kubernetes.io/projected/0b5185ed-b846-4b0f-a67c-55af274532aa-kube-api-access-s2jtm\") pod \"limitador-operator-controller-manager-74495c69f8-b5ffm\" (UID: \"0b5185ed-b846-4b0f-a67c-55af274532aa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:13.568754 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:13.568706    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2jtm\" (UniqueName: \"kubernetes.io/projected/0b5185ed-b846-4b0f-a67c-55af274532aa-kube-api-access-s2jtm\") pod \"limitador-operator-controller-manager-74495c69f8-b5ffm\" (UID: \"0b5185ed-b846-4b0f-a67c-55af274532aa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:13.579839 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:13.579807    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2jtm\" (UniqueName: \"kubernetes.io/projected/0b5185ed-b846-4b0f-a67c-55af274532aa-kube-api-access-s2jtm\") pod \"limitador-operator-controller-manager-74495c69f8-b5ffm\" (UID: \"0b5185ed-b846-4b0f-a67c-55af274532aa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:13.694244 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:13.694141    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:13.967328 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:13.967291    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"]
Sep 15 21:18:13.969562 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:18:13.969530    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5185ed_b846_4b0f_a67c_55af274532aa.slice/crio-a7103088dea6924ac472a08d4d05ed791d99730e22490a164d233fccd5687de2 WatchSource:0}: Error finding container a7103088dea6924ac472a08d4d05ed791d99730e22490a164d233fccd5687de2: Status 404 returned error can't find the container with id a7103088dea6924ac472a08d4d05ed791d99730e22490a164d233fccd5687de2
Sep 15 21:18:14.546295 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:14.546261    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm" event={"ID":"0b5185ed-b846-4b0f-a67c-55af274532aa","Type":"ContainerStarted","Data":"a7103088dea6924ac472a08d4d05ed791d99730e22490a164d233fccd5687de2"}
Sep 15 21:18:15.502019 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:15.501986    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n8nhk"]
Sep 15 21:18:15.509088 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:15.509056    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n8nhk"
Sep 15 21:18:15.511588 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:15.511564    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-h5smd\""
Sep 15 21:18:15.511723 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:15.511568    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 15 21:18:15.517526 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:15.517498    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n8nhk"]
Sep 15 21:18:15.588355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:15.588316    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52hpw\" (UniqueName: \"kubernetes.io/projected/c9dee725-b3fb-445e-bb74-7703ce75d73f-kube-api-access-52hpw\") pod \"dns-operator-controller-manager-7b84cd7c9c-n8nhk\" (UID: \"c9dee725-b3fb-445e-bb74-7703ce75d73f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n8nhk"
Sep 15 21:18:15.689668 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:15.689633    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52hpw\" (UniqueName: \"kubernetes.io/projected/c9dee725-b3fb-445e-bb74-7703ce75d73f-kube-api-access-52hpw\") pod \"dns-operator-controller-manager-7b84cd7c9c-n8nhk\" (UID: \"c9dee725-b3fb-445e-bb74-7703ce75d73f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n8nhk"
Sep 15 21:18:15.703416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:15.703359    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52hpw\" (UniqueName: \"kubernetes.io/projected/c9dee725-b3fb-445e-bb74-7703ce75d73f-kube-api-access-52hpw\") pod \"dns-operator-controller-manager-7b84cd7c9c-n8nhk\" (UID: \"c9dee725-b3fb-445e-bb74-7703ce75d73f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n8nhk"
Sep 15 21:18:15.822591 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:15.822550    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n8nhk"
Sep 15 21:18:16.560827 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:16.560783    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm" event={"ID":"0b5185ed-b846-4b0f-a67c-55af274532aa","Type":"ContainerStarted","Data":"474e953e0ae9fe492f48f1f0ead805c8622769d6cdb1f523defef5c9d96539c4"}
Sep 15 21:18:16.561000 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:16.560923    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:16.590172 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:16.590118    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm" podStartSLOduration=1.134636081 podStartE2EDuration="3.59010226s" totalImagesPullingTime="2.455466179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:16.589334807 +0000 UTC m=+464.451268611" watchObservedRunningTime="2026-09-15 21:18:16.59010226 +0000 UTC m=+464.452036065"
Sep 15 21:18:16.626350 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:16.626265    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n8nhk"]
Sep 15 21:18:16.629545 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:18:16.629516    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9dee725_b3fb_445e_bb74_7703ce75d73f.slice/crio-062fbef69e00d60464f5efc9a17dc01d5e67608dcaf9f83453f580a5d8abf262 WatchSource:0}: Error finding container 062fbef69e00d60464f5efc9a17dc01d5e67608dcaf9f83453f580a5d8abf262: Status 404 returned error can't find the container with id 062fbef69e00d60464f5efc9a17dc01d5e67608dcaf9f83453f580a5d8abf262
Sep 15 21:18:17.568117 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:17.568072    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n8nhk" event={"ID":"c9dee725-b3fb-445e-bb74-7703ce75d73f","Type":"ContainerStarted","Data":"062fbef69e00d60464f5efc9a17dc01d5e67608dcaf9f83453f580a5d8abf262"}
Sep 15 21:18:18.301046 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:18.301010    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-fglmp"]
Sep 15 21:18:18.304508 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:18.304489    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-fglmp"
Sep 15 21:18:18.308283 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:18.308265    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-twpvj\""
Sep 15 21:18:18.326940 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:18.326911    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-fglmp"]
Sep 15 21:18:18.418216 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:18.418118    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dqws\" (UniqueName: \"kubernetes.io/projected/a6d036da-41d2-4ef0-a98d-f439302aa413-kube-api-access-8dqws\") pod \"authorino-operator-76b478bd78-fglmp\" (UID: \"a6d036da-41d2-4ef0-a98d-f439302aa413\") " pod="kuadrant-system/authorino-operator-76b478bd78-fglmp"
Sep 15 21:18:18.519444 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:18.519398    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dqws\" (UniqueName: \"kubernetes.io/projected/a6d036da-41d2-4ef0-a98d-f439302aa413-kube-api-access-8dqws\") pod \"authorino-operator-76b478bd78-fglmp\" (UID: \"a6d036da-41d2-4ef0-a98d-f439302aa413\") " pod="kuadrant-system/authorino-operator-76b478bd78-fglmp"
Sep 15 21:18:18.538977 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:18.538950    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dqws\" (UniqueName: \"kubernetes.io/projected/a6d036da-41d2-4ef0-a98d-f439302aa413-kube-api-access-8dqws\") pod \"authorino-operator-76b478bd78-fglmp\" (UID: \"a6d036da-41d2-4ef0-a98d-f439302aa413\") " pod="kuadrant-system/authorino-operator-76b478bd78-fglmp"
Sep 15 21:18:18.574707 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:18.574672    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n8nhk" event={"ID":"c9dee725-b3fb-445e-bb74-7703ce75d73f","Type":"ContainerStarted","Data":"9ed46e800cb83176469867cd9e9e58c843004a45e424f642a1f196211769e37d"}
Sep 15 21:18:18.574887 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:18.574774    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n8nhk"
Sep 15 21:18:18.604185 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:18.604131    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n8nhk" podStartSLOduration=2.076657109 podStartE2EDuration="3.60411689s" totalImagesPullingTime="1.527459781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:18.602509575 +0000 UTC m=+466.464443381" watchObservedRunningTime="2026-09-15 21:18:18.60411689 +0000 UTC m=+466.466050695"
Sep 15 21:18:18.614603 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:18.614575    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-fglmp"
Sep 15 21:18:18.887702 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:18.887678    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-fglmp"]
Sep 15 21:18:18.889707 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:18:18.889672    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d036da_41d2_4ef0_a98d_f439302aa413.slice/crio-e4eb03d216ae6d7095ce751ca1da8f8163c0222c596daf45f0384bc3e2b48efc WatchSource:0}: Error finding container e4eb03d216ae6d7095ce751ca1da8f8163c0222c596daf45f0384bc3e2b48efc: Status 404 returned error can't find the container with id e4eb03d216ae6d7095ce751ca1da8f8163c0222c596daf45f0384bc3e2b48efc
Sep 15 21:18:19.580327 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:19.580296    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-fglmp" event={"ID":"a6d036da-41d2-4ef0-a98d-f439302aa413","Type":"ContainerStarted","Data":"e4eb03d216ae6d7095ce751ca1da8f8163c0222c596daf45f0384bc3e2b48efc"}
Sep 15 21:18:20.586610 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:20.586522    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-fglmp" event={"ID":"a6d036da-41d2-4ef0-a98d-f439302aa413","Type":"ContainerStarted","Data":"108167679d91a84dd12f52914b070f556d74df8b872519d40e56f918cc01616b"}
Sep 15 21:18:20.586610 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:20.586598    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-fglmp"
Sep 15 21:18:20.605114 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:20.605051    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-fglmp" podStartSLOduration=1.2168637900000001 podStartE2EDuration="2.605033134s" totalImagesPullingTime="1.388169344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:20.60408794 +0000 UTC m=+468.466021745" watchObservedRunningTime="2026-09-15 21:18:20.605033134 +0000 UTC m=+468.466966940"
Sep 15 21:18:22.446292 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:22.446261    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"]
Sep 15 21:18:22.449635 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:22.449617    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"
Sep 15 21:18:22.453543 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:22.453515    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6z224\""
Sep 15 21:18:22.468279 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:22.468246    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"]
Sep 15 21:18:22.557060 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:22.557013    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75f86aa6-32e5-441d-8ccd-a857e99f22bc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-92gb9\" (UID: \"75f86aa6-32e5-441d-8ccd-a857e99f22bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"
Sep 15 21:18:22.557060 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:22.557058    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n28g2\" (UniqueName: \"kubernetes.io/projected/75f86aa6-32e5-441d-8ccd-a857e99f22bc-kube-api-access-n28g2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-92gb9\" (UID: \"75f86aa6-32e5-441d-8ccd-a857e99f22bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"
Sep 15 21:18:22.658573 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:22.658535    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75f86aa6-32e5-441d-8ccd-a857e99f22bc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-92gb9\" (UID: \"75f86aa6-32e5-441d-8ccd-a857e99f22bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"
Sep 15 21:18:22.658573 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:22.658575    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n28g2\" (UniqueName: \"kubernetes.io/projected/75f86aa6-32e5-441d-8ccd-a857e99f22bc-kube-api-access-n28g2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-92gb9\" (UID: \"75f86aa6-32e5-441d-8ccd-a857e99f22bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"
Sep 15 21:18:22.658934 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:22.658915    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75f86aa6-32e5-441d-8ccd-a857e99f22bc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-92gb9\" (UID: \"75f86aa6-32e5-441d-8ccd-a857e99f22bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"
Sep 15 21:18:22.672906 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:22.672877    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n28g2\" (UniqueName: \"kubernetes.io/projected/75f86aa6-32e5-441d-8ccd-a857e99f22bc-kube-api-access-n28g2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-92gb9\" (UID: \"75f86aa6-32e5-441d-8ccd-a857e99f22bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"
Sep 15 21:18:22.759931 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:22.759853    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"
Sep 15 21:18:23.040750 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:23.040689    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"]
Sep 15 21:18:23.043049 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:18:23.043015    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f86aa6_32e5_441d_8ccd_a857e99f22bc.slice/crio-5028270d9bac61a48c9415a773c359cbe03d614cb9576406ebb54c06eb62053e WatchSource:0}: Error finding container 5028270d9bac61a48c9415a773c359cbe03d614cb9576406ebb54c06eb62053e: Status 404 returned error can't find the container with id 5028270d9bac61a48c9415a773c359cbe03d614cb9576406ebb54c06eb62053e
Sep 15 21:18:23.602949 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:23.602911    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9" event={"ID":"75f86aa6-32e5-441d-8ccd-a857e99f22bc","Type":"ContainerStarted","Data":"5028270d9bac61a48c9415a773c359cbe03d614cb9576406ebb54c06eb62053e"}
Sep 15 21:18:27.570560 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:27.570529    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:27.625765 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:27.625719    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9" event={"ID":"75f86aa6-32e5-441d-8ccd-a857e99f22bc","Type":"ContainerStarted","Data":"7a3c6c02efa24d241a586c799ce0aa07b0513d7cad882d37d864868968b262ff"}
Sep 15 21:18:27.625933 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:27.625809    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"
Sep 15 21:18:27.647347 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:27.647290    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9" podStartSLOduration=1.900505839 podStartE2EDuration="5.647271625s" totalImagesPullingTime="3.746765786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:27.646054936 +0000 UTC m=+475.507988741" watchObservedRunningTime="2026-09-15 21:18:27.647271625 +0000 UTC m=+475.509205431"
Sep 15 21:18:29.412879 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.412838    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8"]
Sep 15 21:18:29.417040 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.417015    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8"
Sep 15 21:18:29.419282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.419263    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 15 21:18:29.419605 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.419589    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 15 21:18:29.419694 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.419610    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gg6k6\""
Sep 15 21:18:29.423234 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.423209    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8"]
Sep 15 21:18:29.519442 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.519407    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfrfj\" (UniqueName: \"kubernetes.io/projected/44d32b0f-59dd-44d9-936c-3ec66bbdbc18-kube-api-access-bfrfj\") pod \"kuadrant-console-plugin-fbdd4f445-65mm8\" (UID: \"44d32b0f-59dd-44d9-936c-3ec66bbdbc18\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8"
Sep 15 21:18:29.519626 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.519464    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/44d32b0f-59dd-44d9-936c-3ec66bbdbc18-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-65mm8\" (UID: \"44d32b0f-59dd-44d9-936c-3ec66bbdbc18\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8"
Sep 15 21:18:29.519669 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.519649    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/44d32b0f-59dd-44d9-936c-3ec66bbdbc18-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-65mm8\" (UID: \"44d32b0f-59dd-44d9-936c-3ec66bbdbc18\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8"
Sep 15 21:18:29.582903 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.582874    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n8nhk"
Sep 15 21:18:29.621113 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.621080    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/44d32b0f-59dd-44d9-936c-3ec66bbdbc18-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-65mm8\" (UID: \"44d32b0f-59dd-44d9-936c-3ec66bbdbc18\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8"
Sep 15 21:18:29.621113 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.621120    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfrfj\" (UniqueName: \"kubernetes.io/projected/44d32b0f-59dd-44d9-936c-3ec66bbdbc18-kube-api-access-bfrfj\") pod \"kuadrant-console-plugin-fbdd4f445-65mm8\" (UID: \"44d32b0f-59dd-44d9-936c-3ec66bbdbc18\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8"
Sep 15 21:18:29.621372 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.621139    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/44d32b0f-59dd-44d9-936c-3ec66bbdbc18-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-65mm8\" (UID: \"44d32b0f-59dd-44d9-936c-3ec66bbdbc18\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8"
Sep 15 21:18:29.621929 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.621903    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/44d32b0f-59dd-44d9-936c-3ec66bbdbc18-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-65mm8\" (UID: \"44d32b0f-59dd-44d9-936c-3ec66bbdbc18\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8"
Sep 15 21:18:29.623891 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.623871    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/44d32b0f-59dd-44d9-936c-3ec66bbdbc18-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-65mm8\" (UID: \"44d32b0f-59dd-44d9-936c-3ec66bbdbc18\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8"
Sep 15 21:18:29.630427 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.630401    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfrfj\" (UniqueName: \"kubernetes.io/projected/44d32b0f-59dd-44d9-936c-3ec66bbdbc18-kube-api-access-bfrfj\") pod \"kuadrant-console-plugin-fbdd4f445-65mm8\" (UID: \"44d32b0f-59dd-44d9-936c-3ec66bbdbc18\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8"
Sep 15 21:18:29.734221 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:29.734141    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8"
Sep 15 21:18:30.016416 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:30.016390    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8"]
Sep 15 21:18:30.018338 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:18:30.018312    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44d32b0f_59dd_44d9_936c_3ec66bbdbc18.slice/crio-affb2246b412eca6656d18fb7922c3b4361613ead733138067aa77c5c288e21a WatchSource:0}: Error finding container affb2246b412eca6656d18fb7922c3b4361613ead733138067aa77c5c288e21a: Status 404 returned error can't find the container with id affb2246b412eca6656d18fb7922c3b4361613ead733138067aa77c5c288e21a
Sep 15 21:18:30.647313 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:30.647249    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8" event={"ID":"44d32b0f-59dd-44d9-936c-3ec66bbdbc18","Type":"ContainerStarted","Data":"affb2246b412eca6656d18fb7922c3b4361613ead733138067aa77c5c288e21a"}
Sep 15 21:18:31.594486 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:31.594454    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-fglmp"
Sep 15 21:18:34.672065 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:34.671977    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8" event={"ID":"44d32b0f-59dd-44d9-936c-3ec66bbdbc18","Type":"ContainerStarted","Data":"50c1a65cf3a41c07c4aa201527d66d70bcdad9effb7567ae7d7b5fa059f95f7e"}
Sep 15 21:18:34.689538 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:34.689479    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-65mm8" podStartSLOduration=1.417500943 podStartE2EDuration="5.689459081s" totalImagesPullingTime="4.271958138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:34.689336604 +0000 UTC m=+482.551270409" watchObservedRunningTime="2026-09-15 21:18:34.689459081 +0000 UTC m=+482.551392887"
Sep 15 21:18:38.633669 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:38.633631    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"
Sep 15 21:18:40.650897 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.650859    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"]
Sep 15 21:18:40.651488 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.651084    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9" podUID="75f86aa6-32e5-441d-8ccd-a857e99f22bc" containerName="manager" containerID="cri-o://7a3c6c02efa24d241a586c799ce0aa07b0513d7cad882d37d864868968b262ff" gracePeriod=2
Sep 15 21:18:40.656535 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.656508    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"]
Sep 15 21:18:40.673393 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.673347    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"]
Sep 15 21:18:40.673643 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.673602    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm" podUID="0b5185ed-b846-4b0f-a67c-55af274532aa" containerName="manager" containerID="cri-o://474e953e0ae9fe492f48f1f0ead805c8622769d6cdb1f523defef5c9d96539c4" gracePeriod=2
Sep 15 21:18:40.675792 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:18:40.675754    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-b5ffm\" is forbidden: User \"system:node:ip-10-0-139-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-184.ec2.internal' and this object" podUID="0b5185ed-b846-4b0f-a67c-55af274532aa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:40.678213 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.678180    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"]
Sep 15 21:18:40.681720 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.681698    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"]
Sep 15 21:18:40.682282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.682201    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5185ed-b846-4b0f-a67c-55af274532aa" containerName="manager"
Sep 15 21:18:40.682282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.682223    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5185ed-b846-4b0f-a67c-55af274532aa" containerName="manager"
Sep 15 21:18:40.682282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.682237    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f86aa6-32e5-441d-8ccd-a857e99f22bc" containerName="manager"
Sep 15 21:18:40.682282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.682246    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f86aa6-32e5-441d-8ccd-a857e99f22bc" containerName="manager"
Sep 15 21:18:40.682641 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.682333    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f86aa6-32e5-441d-8ccd-a857e99f22bc" containerName="manager"
Sep 15 21:18:40.682641 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.682347    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b5185ed-b846-4b0f-a67c-55af274532aa" containerName="manager"
Sep 15 21:18:40.685762 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.685743    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"
Sep 15 21:18:40.704851 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.704813    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-twq57"]
Sep 15 21:18:40.708545 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.708522    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"]
Sep 15 21:18:40.708685 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.708621    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-twq57"
Sep 15 21:18:40.719369 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.719336    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-twq57"]
Sep 15 21:18:40.722419 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.722373    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jswdv\" (UniqueName: \"kubernetes.io/projected/53d1a588-9ec9-4684-8fd3-1dec67fdf2b9-kube-api-access-jswdv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qvq5d\" (UID: \"53d1a588-9ec9-4684-8fd3-1dec67fdf2b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"
Sep 15 21:18:40.722597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.722490    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53d1a588-9ec9-4684-8fd3-1dec67fdf2b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qvq5d\" (UID: \"53d1a588-9ec9-4684-8fd3-1dec67fdf2b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"
Sep 15 21:18:40.722597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.722531    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-858cd\" (UniqueName: \"kubernetes.io/projected/49d2501c-80ab-40c0-bc11-df6414168bd7-kube-api-access-858cd\") pod \"limitador-operator-controller-manager-74495c69f8-twq57\" (UID: \"49d2501c-80ab-40c0-bc11-df6414168bd7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-twq57"
Sep 15 21:18:40.745659 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:18:40.745628    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-b5ffm\" is forbidden: User \"system:node:ip-10-0-139-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-184.ec2.internal' and this object" podUID="0b5185ed-b846-4b0f-a67c-55af274532aa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:40.747890 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:18:40.747862    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-b5ffm\" is forbidden: User \"system:node:ip-10-0-139-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-184.ec2.internal' and this object" podUID="0b5185ed-b846-4b0f-a67c-55af274532aa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:40.823299 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.823263    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jswdv\" (UniqueName: \"kubernetes.io/projected/53d1a588-9ec9-4684-8fd3-1dec67fdf2b9-kube-api-access-jswdv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qvq5d\" (UID: \"53d1a588-9ec9-4684-8fd3-1dec67fdf2b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"
Sep 15 21:18:40.823510 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.823323    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53d1a588-9ec9-4684-8fd3-1dec67fdf2b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qvq5d\" (UID: \"53d1a588-9ec9-4684-8fd3-1dec67fdf2b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"
Sep 15 21:18:40.823510 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.823350    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-858cd\" (UniqueName: \"kubernetes.io/projected/49d2501c-80ab-40c0-bc11-df6414168bd7-kube-api-access-858cd\") pod \"limitador-operator-controller-manager-74495c69f8-twq57\" (UID: \"49d2501c-80ab-40c0-bc11-df6414168bd7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-twq57"
Sep 15 21:18:40.823701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.823679    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53d1a588-9ec9-4684-8fd3-1dec67fdf2b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qvq5d\" (UID: \"53d1a588-9ec9-4684-8fd3-1dec67fdf2b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"
Sep 15 21:18:40.833424 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.832968    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jswdv\" (UniqueName: \"kubernetes.io/projected/53d1a588-9ec9-4684-8fd3-1dec67fdf2b9-kube-api-access-jswdv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qvq5d\" (UID: \"53d1a588-9ec9-4684-8fd3-1dec67fdf2b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"
Sep 15 21:18:40.833424 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.833047    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-858cd\" (UniqueName: \"kubernetes.io/projected/49d2501c-80ab-40c0-bc11-df6414168bd7-kube-api-access-858cd\") pod \"limitador-operator-controller-manager-74495c69f8-twq57\" (UID: \"49d2501c-80ab-40c0-bc11-df6414168bd7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-twq57"
Sep 15 21:18:40.906223 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.906192    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"
Sep 15 21:18:40.908295 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:18:40.908263    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-b5ffm\" is forbidden: User \"system:node:ip-10-0-139-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-184.ec2.internal' and this object" podUID="0b5185ed-b846-4b0f-a67c-55af274532aa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:40.911821 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.911794    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:40.914505 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:18:40.914469    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-b5ffm\" is forbidden: User \"system:node:ip-10-0-139-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-184.ec2.internal' and this object" podUID="0b5185ed-b846-4b0f-a67c-55af274532aa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:40.923970 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.923946    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n28g2\" (UniqueName: \"kubernetes.io/projected/75f86aa6-32e5-441d-8ccd-a857e99f22bc-kube-api-access-n28g2\") pod \"75f86aa6-32e5-441d-8ccd-a857e99f22bc\" (UID: \"75f86aa6-32e5-441d-8ccd-a857e99f22bc\") "
Sep 15 21:18:40.924118 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.923979    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2jtm\" (UniqueName: \"kubernetes.io/projected/0b5185ed-b846-4b0f-a67c-55af274532aa-kube-api-access-s2jtm\") pod \"0b5185ed-b846-4b0f-a67c-55af274532aa\" (UID: \"0b5185ed-b846-4b0f-a67c-55af274532aa\") "
Sep 15 21:18:40.924118 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.924068    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75f86aa6-32e5-441d-8ccd-a857e99f22bc-extensions-socket-volume\") pod \"75f86aa6-32e5-441d-8ccd-a857e99f22bc\" (UID: \"75f86aa6-32e5-441d-8ccd-a857e99f22bc\") "
Sep 15 21:18:40.924504 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.924478    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f86aa6-32e5-441d-8ccd-a857e99f22bc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "75f86aa6-32e5-441d-8ccd-a857e99f22bc" (UID: "75f86aa6-32e5-441d-8ccd-a857e99f22bc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:40.926178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.926154    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5185ed-b846-4b0f-a67c-55af274532aa-kube-api-access-s2jtm" (OuterVolumeSpecName: "kube-api-access-s2jtm") pod "0b5185ed-b846-4b0f-a67c-55af274532aa" (UID: "0b5185ed-b846-4b0f-a67c-55af274532aa"). InnerVolumeSpecName "kube-api-access-s2jtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:40.926178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:40.926164    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f86aa6-32e5-441d-8ccd-a857e99f22bc-kube-api-access-n28g2" (OuterVolumeSpecName: "kube-api-access-n28g2") pod "75f86aa6-32e5-441d-8ccd-a857e99f22bc" (UID: "75f86aa6-32e5-441d-8ccd-a857e99f22bc"). InnerVolumeSpecName "kube-api-access-n28g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:41.025638 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.025601    2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75f86aa6-32e5-441d-8ccd-a857e99f22bc-extensions-socket-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:18:41.025638 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.025634    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n28g2\" (UniqueName: \"kubernetes.io/projected/75f86aa6-32e5-441d-8ccd-a857e99f22bc-kube-api-access-n28g2\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:18:41.025638 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.025644    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2jtm\" (UniqueName: \"kubernetes.io/projected/0b5185ed-b846-4b0f-a67c-55af274532aa-kube-api-access-s2jtm\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:18:41.100708 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.100670    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"
Sep 15 21:18:41.106805 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.106783    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-twq57"
Sep 15 21:18:41.390815 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.390699    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"]
Sep 15 21:18:41.392700 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:18:41.392627    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d1a588_9ec9_4684_8fd3_1dec67fdf2b9.slice/crio-a7fd8fb3ee3644ef70624817fbd40253b4f7b561076ebb010372068520d37145 WatchSource:0}: Error finding container a7fd8fb3ee3644ef70624817fbd40253b4f7b561076ebb010372068520d37145: Status 404 returned error can't find the container with id a7fd8fb3ee3644ef70624817fbd40253b4f7b561076ebb010372068520d37145
Sep 15 21:18:41.392798 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.392756    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-twq57"]
Sep 15 21:18:41.394982 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:18:41.394892    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d2501c_80ab_40c0_bc11_df6414168bd7.slice/crio-816166d62708542a00b4eb7cc472a4619b9c74d8ca6615fbb15911f9ceca204b WatchSource:0}: Error finding container 816166d62708542a00b4eb7cc472a4619b9c74d8ca6615fbb15911f9ceca204b: Status 404 returned error can't find the container with id 816166d62708542a00b4eb7cc472a4619b9c74d8ca6615fbb15911f9ceca204b
Sep 15 21:18:41.706840 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.706801    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d" event={"ID":"53d1a588-9ec9-4684-8fd3-1dec67fdf2b9","Type":"ContainerStarted","Data":"26f631196aa7562f6acb5b38122b53f18d4fcab0564cbb1dcb2e9ff108c2667f"}
Sep 15 21:18:41.707278 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.706853    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d" event={"ID":"53d1a588-9ec9-4684-8fd3-1dec67fdf2b9","Type":"ContainerStarted","Data":"a7fd8fb3ee3644ef70624817fbd40253b4f7b561076ebb010372068520d37145"}
Sep 15 21:18:41.707278 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.706934    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"
Sep 15 21:18:41.708368 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.708345    2584 generic.go:356] "Generic (PLEG): container finished" podID="75f86aa6-32e5-441d-8ccd-a857e99f22bc" containerID="7a3c6c02efa24d241a586c799ce0aa07b0513d7cad882d37d864868968b262ff" exitCode=0
Sep 15 21:18:41.708497 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.708421    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-92gb9"
Sep 15 21:18:41.708497 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.708434    2584 scope.go:117] "RemoveContainer" containerID="7a3c6c02efa24d241a586c799ce0aa07b0513d7cad882d37d864868968b262ff"
Sep 15 21:18:41.710238 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.710209    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-twq57" event={"ID":"49d2501c-80ab-40c0-bc11-df6414168bd7","Type":"ContainerStarted","Data":"8f815b0d62895eb0f86de1afb57aab987abb2e6cbf1a33fa03801bba5c4bd936"}
Sep 15 21:18:41.710345 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.710259    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-twq57" event={"ID":"49d2501c-80ab-40c0-bc11-df6414168bd7","Type":"ContainerStarted","Data":"816166d62708542a00b4eb7cc472a4619b9c74d8ca6615fbb15911f9ceca204b"}
Sep 15 21:18:41.710420 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.710357    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-twq57"
Sep 15 21:18:41.711727 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.711701    2584 generic.go:356] "Generic (PLEG): container finished" podID="0b5185ed-b846-4b0f-a67c-55af274532aa" containerID="474e953e0ae9fe492f48f1f0ead805c8622769d6cdb1f523defef5c9d96539c4" exitCode=0
Sep 15 21:18:41.711862 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.711769    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:41.718624 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.718528    2584 scope.go:117] "RemoveContainer" containerID="7a3c6c02efa24d241a586c799ce0aa07b0513d7cad882d37d864868968b262ff"
Sep 15 21:18:41.718909 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:18:41.718888    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3c6c02efa24d241a586c799ce0aa07b0513d7cad882d37d864868968b262ff\": container with ID starting with 7a3c6c02efa24d241a586c799ce0aa07b0513d7cad882d37d864868968b262ff not found: ID does not exist" containerID="7a3c6c02efa24d241a586c799ce0aa07b0513d7cad882d37d864868968b262ff"
Sep 15 21:18:41.718994 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.718915    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3c6c02efa24d241a586c799ce0aa07b0513d7cad882d37d864868968b262ff"} err="failed to get container status \"7a3c6c02efa24d241a586c799ce0aa07b0513d7cad882d37d864868968b262ff\": rpc error: code = NotFound desc = could not find container \"7a3c6c02efa24d241a586c799ce0aa07b0513d7cad882d37d864868968b262ff\": container with ID starting with 7a3c6c02efa24d241a586c799ce0aa07b0513d7cad882d37d864868968b262ff not found: ID does not exist"
Sep 15 21:18:41.718994 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.718932    2584 scope.go:117] "RemoveContainer" containerID="474e953e0ae9fe492f48f1f0ead805c8622769d6cdb1f523defef5c9d96539c4"
Sep 15 21:18:41.726859 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.726843    2584 scope.go:117] "RemoveContainer" containerID="474e953e0ae9fe492f48f1f0ead805c8622769d6cdb1f523defef5c9d96539c4"
Sep 15 21:18:41.727123 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:18:41.727102    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474e953e0ae9fe492f48f1f0ead805c8622769d6cdb1f523defef5c9d96539c4\": container with ID starting with 474e953e0ae9fe492f48f1f0ead805c8622769d6cdb1f523defef5c9d96539c4 not found: ID does not exist" containerID="474e953e0ae9fe492f48f1f0ead805c8622769d6cdb1f523defef5c9d96539c4"
Sep 15 21:18:41.727172 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.727129    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474e953e0ae9fe492f48f1f0ead805c8622769d6cdb1f523defef5c9d96539c4"} err="failed to get container status \"474e953e0ae9fe492f48f1f0ead805c8622769d6cdb1f523defef5c9d96539c4\": rpc error: code = NotFound desc = could not find container \"474e953e0ae9fe492f48f1f0ead805c8622769d6cdb1f523defef5c9d96539c4\": container with ID starting with 474e953e0ae9fe492f48f1f0ead805c8622769d6cdb1f523defef5c9d96539c4 not found: ID does not exist"
Sep 15 21:18:41.731780 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.731738    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d" podStartSLOduration=1.7317260829999999 podStartE2EDuration="1.731726083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:41.730041647 +0000 UTC m=+489.591975446" watchObservedRunningTime="2026-09-15 21:18:41.731726083 +0000 UTC m=+489.593659888"
Sep 15 21:18:41.732015 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:18:41.731994    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-b5ffm\" is forbidden: User \"system:node:ip-10-0-139-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-184.ec2.internal' and this object" podUID="0b5185ed-b846-4b0f-a67c-55af274532aa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:41.733661 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:18:41.733641    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-b5ffm\" is forbidden: User \"system:node:ip-10-0-139-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-184.ec2.internal' and this object" podUID="0b5185ed-b846-4b0f-a67c-55af274532aa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:41.754152 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:41.754094    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-twq57" podStartSLOduration=1.754074159 podStartE2EDuration="1.754074159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:41.752770548 +0000 UTC m=+489.614704353" watchObservedRunningTime="2026-09-15 21:18:41.754074159 +0000 UTC m=+489.616007966"
Sep 15 21:18:42.670343 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:18:42.670308    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-b5ffm\" is forbidden: User \"system:node:ip-10-0-139-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-184.ec2.internal' and this object" podUID="0b5185ed-b846-4b0f-a67c-55af274532aa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b5ffm"
Sep 15 21:18:42.672239 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:42.672216    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b5185ed-b846-4b0f-a67c-55af274532aa" path="/var/lib/kubelet/pods/0b5185ed-b846-4b0f-a67c-55af274532aa/volumes"
Sep 15 21:18:42.672629 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:42.672615    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75f86aa6-32e5-441d-8ccd-a857e99f22bc" path="/var/lib/kubelet/pods/75f86aa6-32e5-441d-8ccd-a857e99f22bc/volumes"
Sep 15 21:18:52.720593 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:52.720554    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-twq57"
Sep 15 21:18:52.721105 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:52.720637    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"
Sep 15 21:18:55.985518 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:55.985473    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"]
Sep 15 21:18:55.985965 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:55.985702    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d" podUID="53d1a588-9ec9-4684-8fd3-1dec67fdf2b9" containerName="manager" containerID="cri-o://26f631196aa7562f6acb5b38122b53f18d4fcab0564cbb1dcb2e9ff108c2667f" gracePeriod=10
Sep 15 21:18:56.235594 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.235536    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"
Sep 15 21:18:56.246741 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.246714    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"]
Sep 15 21:18:56.247101 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.247086    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53d1a588-9ec9-4684-8fd3-1dec67fdf2b9" containerName="manager"
Sep 15 21:18:56.247157 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.247105    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="53d1a588-9ec9-4684-8fd3-1dec67fdf2b9" containerName="manager"
Sep 15 21:18:56.247192 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.247184    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="53d1a588-9ec9-4684-8fd3-1dec67fdf2b9" containerName="manager"
Sep 15 21:18:56.249256 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.249235    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"
Sep 15 21:18:56.270527 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.270498    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"]
Sep 15 21:18:56.363779 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.363745    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53d1a588-9ec9-4684-8fd3-1dec67fdf2b9-extensions-socket-volume\") pod \"53d1a588-9ec9-4684-8fd3-1dec67fdf2b9\" (UID: \"53d1a588-9ec9-4684-8fd3-1dec67fdf2b9\") "
Sep 15 21:18:56.363953 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.363797    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jswdv\" (UniqueName: \"kubernetes.io/projected/53d1a588-9ec9-4684-8fd3-1dec67fdf2b9-kube-api-access-jswdv\") pod \"53d1a588-9ec9-4684-8fd3-1dec67fdf2b9\" (UID: \"53d1a588-9ec9-4684-8fd3-1dec67fdf2b9\") "
Sep 15 21:18:56.364095 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.364061    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe5a30ee-14a2-4dec-a16b-2a44bd005c6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gjvxg\" (UID: \"fe5a30ee-14a2-4dec-a16b-2a44bd005c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"
Sep 15 21:18:56.364162 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.364116    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d1a588-9ec9-4684-8fd3-1dec67fdf2b9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "53d1a588-9ec9-4684-8fd3-1dec67fdf2b9" (UID: "53d1a588-9ec9-4684-8fd3-1dec67fdf2b9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:56.364221 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.364202    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92q6h\" (UniqueName: \"kubernetes.io/projected/fe5a30ee-14a2-4dec-a16b-2a44bd005c6b-kube-api-access-92q6h\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gjvxg\" (UID: \"fe5a30ee-14a2-4dec-a16b-2a44bd005c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"
Sep 15 21:18:56.364278 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.364266    2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53d1a588-9ec9-4684-8fd3-1dec67fdf2b9-extensions-socket-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:18:56.365911 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.365886    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d1a588-9ec9-4684-8fd3-1dec67fdf2b9-kube-api-access-jswdv" (OuterVolumeSpecName: "kube-api-access-jswdv") pod "53d1a588-9ec9-4684-8fd3-1dec67fdf2b9" (UID: "53d1a588-9ec9-4684-8fd3-1dec67fdf2b9"). InnerVolumeSpecName "kube-api-access-jswdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:56.464789 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.464751    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92q6h\" (UniqueName: \"kubernetes.io/projected/fe5a30ee-14a2-4dec-a16b-2a44bd005c6b-kube-api-access-92q6h\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gjvxg\" (UID: \"fe5a30ee-14a2-4dec-a16b-2a44bd005c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"
Sep 15 21:18:56.464972 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.464821    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe5a30ee-14a2-4dec-a16b-2a44bd005c6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gjvxg\" (UID: \"fe5a30ee-14a2-4dec-a16b-2a44bd005c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"
Sep 15 21:18:56.464972 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.464870    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jswdv\" (UniqueName: \"kubernetes.io/projected/53d1a588-9ec9-4684-8fd3-1dec67fdf2b9-kube-api-access-jswdv\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:18:56.465175 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.465149    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe5a30ee-14a2-4dec-a16b-2a44bd005c6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gjvxg\" (UID: \"fe5a30ee-14a2-4dec-a16b-2a44bd005c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"
Sep 15 21:18:56.475295 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.475274    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92q6h\" (UniqueName: \"kubernetes.io/projected/fe5a30ee-14a2-4dec-a16b-2a44bd005c6b-kube-api-access-92q6h\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gjvxg\" (UID: \"fe5a30ee-14a2-4dec-a16b-2a44bd005c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"
Sep 15 21:18:56.559506 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.559475    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"
Sep 15 21:18:56.789828 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.789796    2584 generic.go:356] "Generic (PLEG): container finished" podID="53d1a588-9ec9-4684-8fd3-1dec67fdf2b9" containerID="26f631196aa7562f6acb5b38122b53f18d4fcab0564cbb1dcb2e9ff108c2667f" exitCode=0
Sep 15 21:18:56.790022 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.789852    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d" event={"ID":"53d1a588-9ec9-4684-8fd3-1dec67fdf2b9","Type":"ContainerDied","Data":"26f631196aa7562f6acb5b38122b53f18d4fcab0564cbb1dcb2e9ff108c2667f"}
Sep 15 21:18:56.790022 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.789889    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d" event={"ID":"53d1a588-9ec9-4684-8fd3-1dec67fdf2b9","Type":"ContainerDied","Data":"a7fd8fb3ee3644ef70624817fbd40253b4f7b561076ebb010372068520d37145"}
Sep 15 21:18:56.790022 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.789911    2584 scope.go:117] "RemoveContainer" containerID="26f631196aa7562f6acb5b38122b53f18d4fcab0564cbb1dcb2e9ff108c2667f"
Sep 15 21:18:56.790022 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.789852    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"
Sep 15 21:18:56.798352 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.798333    2584 scope.go:117] "RemoveContainer" containerID="26f631196aa7562f6acb5b38122b53f18d4fcab0564cbb1dcb2e9ff108c2667f"
Sep 15 21:18:56.798656 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:18:56.798639    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f631196aa7562f6acb5b38122b53f18d4fcab0564cbb1dcb2e9ff108c2667f\": container with ID starting with 26f631196aa7562f6acb5b38122b53f18d4fcab0564cbb1dcb2e9ff108c2667f not found: ID does not exist" containerID="26f631196aa7562f6acb5b38122b53f18d4fcab0564cbb1dcb2e9ff108c2667f"
Sep 15 21:18:56.798709 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.798666    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f631196aa7562f6acb5b38122b53f18d4fcab0564cbb1dcb2e9ff108c2667f"} err="failed to get container status \"26f631196aa7562f6acb5b38122b53f18d4fcab0564cbb1dcb2e9ff108c2667f\": rpc error: code = NotFound desc = could not find container \"26f631196aa7562f6acb5b38122b53f18d4fcab0564cbb1dcb2e9ff108c2667f\": container with ID starting with 26f631196aa7562f6acb5b38122b53f18d4fcab0564cbb1dcb2e9ff108c2667f not found: ID does not exist"
Sep 15 21:18:56.817392 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.817309    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"]
Sep 15 21:18:56.840538 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.840508    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qvq5d"]
Sep 15 21:18:56.845691 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:56.845669    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"]
Sep 15 21:18:56.848043 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:18:56.848019    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5a30ee_14a2_4dec_a16b_2a44bd005c6b.slice/crio-c76547ff3b1ab85d864ceb08cb4448219cd309e560d8f8cc1360cc7f1f9f45b3 WatchSource:0}: Error finding container c76547ff3b1ab85d864ceb08cb4448219cd309e560d8f8cc1360cc7f1f9f45b3: Status 404 returned error can't find the container with id c76547ff3b1ab85d864ceb08cb4448219cd309e560d8f8cc1360cc7f1f9f45b3
Sep 15 21:18:57.798043 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:57.798011    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg" event={"ID":"fe5a30ee-14a2-4dec-a16b-2a44bd005c6b","Type":"ContainerStarted","Data":"5acdc19d015248d4a85932fa6d46c5088cad6b7f1e9f9c373cdba0d220b9c7ce"}
Sep 15 21:18:57.798464 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:57.798058    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg" event={"ID":"fe5a30ee-14a2-4dec-a16b-2a44bd005c6b","Type":"ContainerStarted","Data":"c76547ff3b1ab85d864ceb08cb4448219cd309e560d8f8cc1360cc7f1f9f45b3"}
Sep 15 21:18:57.798464 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:57.798193    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"
Sep 15 21:18:57.829680 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:57.829599    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg" podStartSLOduration=1.829583211 podStartE2EDuration="1.829583211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:57.827884395 +0000 UTC m=+505.689818199" watchObservedRunningTime="2026-09-15 21:18:57.829583211 +0000 UTC m=+505.691517015"
Sep 15 21:18:58.672106 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:18:58.672072    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53d1a588-9ec9-4684-8fd3-1dec67fdf2b9" path="/var/lib/kubelet/pods/53d1a588-9ec9-4684-8fd3-1dec67fdf2b9/volumes"
Sep 15 21:19:08.805832 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:08.805803    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"
Sep 15 21:19:12.292063 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.292030    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"]
Sep 15 21:19:12.295304 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.295282    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.297617 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.297593    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5qtd6\""
Sep 15 21:19:12.308256 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.308233    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"]
Sep 15 21:19:12.405395 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.405350    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-istio-token\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.405569 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.405403    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-workload-socket\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.405569 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.405506    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.405569 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.405544    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8x48\" (UniqueName: \"kubernetes.io/projected/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-kube-api-access-s8x48\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.405769 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.405609    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-credential-socket\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.405769 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.405663    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.405769 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.405738    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-istio-envoy\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.405904 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.405770    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-istio-data\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.405904 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.405810    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-workload-certs\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.507148 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.507105    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-workload-certs\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.507332 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.507167    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-istio-token\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.507332 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.507198    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-workload-socket\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.507332 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.507223    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.507332 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.507247    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8x48\" (UniqueName: \"kubernetes.io/projected/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-kube-api-access-s8x48\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.507332 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.507293    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-credential-socket\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.507663 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.507348    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.507663 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.507451    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-istio-envoy\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.507663 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.507490    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-istio-data\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.507663 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.507611    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-workload-certs\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.507882 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.507666    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-workload-socket\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.507882 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.507851    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-istio-data\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.507987 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.507878    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-credential-socket\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.508082 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.508060    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.509712 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.509682    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-istio-envoy\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.510135 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.510087    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.520746 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.520723    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-istio-token\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.520853 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.520833    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8x48\" (UniqueName: \"kubernetes.io/projected/fd952d22-e6de-4c5d-acdb-8da8fb46ffb1-kube-api-access-s8x48\") pod \"maas-default-gateway-openshift-default-c747fcbf8-jcbs4\" (UID: \"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.606810 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.606718    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:12.887428 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.887402    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"]
Sep 15 21:19:12.889868 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:19:12.889842    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd952d22_e6de_4c5d_acdb_8da8fb46ffb1.slice/crio-7b859c66a11e1d223bb56dcd66153d71e0f08bcb684169116390cdc9aae98f56 WatchSource:0}: Error finding container 7b859c66a11e1d223bb56dcd66153d71e0f08bcb684169116390cdc9aae98f56: Status 404 returned error can't find the container with id 7b859c66a11e1d223bb56dcd66153d71e0f08bcb684169116390cdc9aae98f56
Sep 15 21:19:12.892202 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.892166    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:19:12.892285 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.892239    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:19:12.892285 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:12.892267    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:19:13.878958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:13.878926    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4" event={"ID":"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1","Type":"ContainerStarted","Data":"07404b73ce54b265a0ad433b2a747c6f916a517a09c0b5d3cd812e265d9aa18f"}
Sep 15 21:19:13.879334 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:13.878978    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4" event={"ID":"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1","Type":"ContainerStarted","Data":"7b859c66a11e1d223bb56dcd66153d71e0f08bcb684169116390cdc9aae98f56"}
Sep 15 21:19:13.912553 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:13.912504    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4" podStartSLOduration=1.912489045 podStartE2EDuration="1.912489045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:19:13.909585064 +0000 UTC m=+521.771518868" watchObservedRunningTime="2026-09-15 21:19:13.912489045 +0000 UTC m=+521.774422850"
Sep 15 21:19:14.607467 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:14.607422    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:14.612326 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:14.612299    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:14.884636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:14.884548    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:14.885576 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:14.885557    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:19:24.594046 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:24.594008    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lwjdd"]
Sep 15 21:19:24.597299 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:24.597277    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lwjdd"
Sep 15 21:19:24.600244 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:24.600223    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-467nr\""
Sep 15 21:19:24.604476 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:24.604452    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lwjdd"]
Sep 15 21:19:24.619251 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:24.619223    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncvlq\" (UniqueName: \"kubernetes.io/projected/439c2336-ae5e-47a9-ae84-4bf283f15ea6-kube-api-access-ncvlq\") pod \"authorino-7bcfd8f9cd-lwjdd\" (UID: \"439c2336-ae5e-47a9-ae84-4bf283f15ea6\") " pod="kuadrant-system/authorino-7bcfd8f9cd-lwjdd"
Sep 15 21:19:24.719846 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:24.719801    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncvlq\" (UniqueName: \"kubernetes.io/projected/439c2336-ae5e-47a9-ae84-4bf283f15ea6-kube-api-access-ncvlq\") pod \"authorino-7bcfd8f9cd-lwjdd\" (UID: \"439c2336-ae5e-47a9-ae84-4bf283f15ea6\") " pod="kuadrant-system/authorino-7bcfd8f9cd-lwjdd"
Sep 15 21:19:24.729705 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:24.729675    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncvlq\" (UniqueName: \"kubernetes.io/projected/439c2336-ae5e-47a9-ae84-4bf283f15ea6-kube-api-access-ncvlq\") pod \"authorino-7bcfd8f9cd-lwjdd\" (UID: \"439c2336-ae5e-47a9-ae84-4bf283f15ea6\") " pod="kuadrant-system/authorino-7bcfd8f9cd-lwjdd"
Sep 15 21:19:24.907807 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:24.907709    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lwjdd"
Sep 15 21:19:25.179895 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:25.179871    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lwjdd"]
Sep 15 21:19:25.182233 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:19:25.182207    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod439c2336_ae5e_47a9_ae84_4bf283f15ea6.slice/crio-6df2ca559665a306a4d27b4776ccf4b00658468bdc9e4d44cc04bdd0c042ae94 WatchSource:0}: Error finding container 6df2ca559665a306a4d27b4776ccf4b00658468bdc9e4d44cc04bdd0c042ae94: Status 404 returned error can't find the container with id 6df2ca559665a306a4d27b4776ccf4b00658468bdc9e4d44cc04bdd0c042ae94
Sep 15 21:19:25.943534 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:25.943497    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lwjdd" event={"ID":"439c2336-ae5e-47a9-ae84-4bf283f15ea6","Type":"ContainerStarted","Data":"6df2ca559665a306a4d27b4776ccf4b00658468bdc9e4d44cc04bdd0c042ae94"}
Sep 15 21:19:26.951412 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:26.951306    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lwjdd" event={"ID":"439c2336-ae5e-47a9-ae84-4bf283f15ea6","Type":"ContainerStarted","Data":"d8eb3be5d11ca5b10f48a44b7abb779c5a94a886c0c4f855b955e03e1e9dd74f"}
Sep 15 21:19:26.967692 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:26.967640    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-lwjdd" podStartSLOduration=1.466151599 podStartE2EDuration="2.967624076s" totalImagesPullingTime="1.501472477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:19:26.966196108 +0000 UTC m=+534.828129916" watchObservedRunningTime="2026-09-15 21:19:26.967624076 +0000 UTC m=+534.829557880"
Sep 15 21:19:28.232455 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:28.232420    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lwjdd"]
Sep 15 21:19:28.962087 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:28.962040    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-lwjdd" podUID="439c2336-ae5e-47a9-ae84-4bf283f15ea6" containerName="authorino" containerID="cri-o://d8eb3be5d11ca5b10f48a44b7abb779c5a94a886c0c4f855b955e03e1e9dd74f" gracePeriod=30
Sep 15 21:19:29.219795 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:29.219735    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lwjdd"
Sep 15 21:19:29.259731 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:29.259701    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncvlq\" (UniqueName: \"kubernetes.io/projected/439c2336-ae5e-47a9-ae84-4bf283f15ea6-kube-api-access-ncvlq\") pod \"439c2336-ae5e-47a9-ae84-4bf283f15ea6\" (UID: \"439c2336-ae5e-47a9-ae84-4bf283f15ea6\") "
Sep 15 21:19:29.261845 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:29.261814    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439c2336-ae5e-47a9-ae84-4bf283f15ea6-kube-api-access-ncvlq" (OuterVolumeSpecName: "kube-api-access-ncvlq") pod "439c2336-ae5e-47a9-ae84-4bf283f15ea6" (UID: "439c2336-ae5e-47a9-ae84-4bf283f15ea6"). InnerVolumeSpecName "kube-api-access-ncvlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:19:29.360909 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:29.360876    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncvlq\" (UniqueName: \"kubernetes.io/projected/439c2336-ae5e-47a9-ae84-4bf283f15ea6-kube-api-access-ncvlq\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:19:29.968103 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:29.968067    2584 generic.go:356] "Generic (PLEG): container finished" podID="439c2336-ae5e-47a9-ae84-4bf283f15ea6" containerID="d8eb3be5d11ca5b10f48a44b7abb779c5a94a886c0c4f855b955e03e1e9dd74f" exitCode=0
Sep 15 21:19:29.968103 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:29.968103    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lwjdd"
Sep 15 21:19:29.968312 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:29.968152    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lwjdd" event={"ID":"439c2336-ae5e-47a9-ae84-4bf283f15ea6","Type":"ContainerDied","Data":"d8eb3be5d11ca5b10f48a44b7abb779c5a94a886c0c4f855b955e03e1e9dd74f"}
Sep 15 21:19:29.968312 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:29.968190    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lwjdd" event={"ID":"439c2336-ae5e-47a9-ae84-4bf283f15ea6","Type":"ContainerDied","Data":"6df2ca559665a306a4d27b4776ccf4b00658468bdc9e4d44cc04bdd0c042ae94"}
Sep 15 21:19:29.968312 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:29.968210    2584 scope.go:117] "RemoveContainer" containerID="d8eb3be5d11ca5b10f48a44b7abb779c5a94a886c0c4f855b955e03e1e9dd74f"
Sep 15 21:19:29.977009 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:29.976994    2584 scope.go:117] "RemoveContainer" containerID="d8eb3be5d11ca5b10f48a44b7abb779c5a94a886c0c4f855b955e03e1e9dd74f"
Sep 15 21:19:29.977277 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:19:29.977260    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8eb3be5d11ca5b10f48a44b7abb779c5a94a886c0c4f855b955e03e1e9dd74f\": container with ID starting with d8eb3be5d11ca5b10f48a44b7abb779c5a94a886c0c4f855b955e03e1e9dd74f not found: ID does not exist" containerID="d8eb3be5d11ca5b10f48a44b7abb779c5a94a886c0c4f855b955e03e1e9dd74f"
Sep 15 21:19:29.977352 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:29.977284    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8eb3be5d11ca5b10f48a44b7abb779c5a94a886c0c4f855b955e03e1e9dd74f"} err="failed to get container status \"d8eb3be5d11ca5b10f48a44b7abb779c5a94a886c0c4f855b955e03e1e9dd74f\": rpc error: code = NotFound desc = could not find container \"d8eb3be5d11ca5b10f48a44b7abb779c5a94a886c0c4f855b955e03e1e9dd74f\": container with ID starting with d8eb3be5d11ca5b10f48a44b7abb779c5a94a886c0c4f855b955e03e1e9dd74f not found: ID does not exist"
Sep 15 21:19:29.989229 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:29.989203    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lwjdd"]
Sep 15 21:19:29.995835 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:29.995813    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lwjdd"]
Sep 15 21:19:30.673442 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:30.673404    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="439c2336-ae5e-47a9-ae84-4bf283f15ea6" path="/var/lib/kubelet/pods/439c2336-ae5e-47a9-ae84-4bf283f15ea6/volumes"
Sep 15 21:19:53.917808 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:53.917776    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-56fq7"]
Sep 15 21:19:53.918258 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:53.918156    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="439c2336-ae5e-47a9-ae84-4bf283f15ea6" containerName="authorino"
Sep 15 21:19:53.918258 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:53.918168    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="439c2336-ae5e-47a9-ae84-4bf283f15ea6" containerName="authorino"
Sep 15 21:19:53.918258 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:53.918223    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="439c2336-ae5e-47a9-ae84-4bf283f15ea6" containerName="authorino"
Sep 15 21:19:53.923017 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:53.922998    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-56fq7"
Sep 15 21:19:53.925259 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:53.925232    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-467nr\""
Sep 15 21:19:53.927853 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:53.927829    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-56fq7"]
Sep 15 21:19:53.986536 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:53.986500    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hv9r\" (UniqueName: \"kubernetes.io/projected/f56574c2-3b3a-4e31-8994-0a960b75c59e-kube-api-access-7hv9r\") pod \"authorino-557f876f69-56fq7\" (UID: \"f56574c2-3b3a-4e31-8994-0a960b75c59e\") " pod="kuadrant-system/authorino-557f876f69-56fq7"
Sep 15 21:19:54.037448 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.037408    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-56fq7"]
Sep 15 21:19:54.037723 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:19:54.037704    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7hv9r], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-56fq7" podUID="f56574c2-3b3a-4e31-8994-0a960b75c59e"
Sep 15 21:19:54.087510 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.087470    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hv9r\" (UniqueName: \"kubernetes.io/projected/f56574c2-3b3a-4e31-8994-0a960b75c59e-kube-api-access-7hv9r\") pod \"authorino-557f876f69-56fq7\" (UID: \"f56574c2-3b3a-4e31-8994-0a960b75c59e\") " pod="kuadrant-system/authorino-557f876f69-56fq7"
Sep 15 21:19:54.088310 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.088290    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-56fq7"
Sep 15 21:19:54.096158 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.096132    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-56fq7"
Sep 15 21:19:54.096296 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.096227    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hv9r\" (UniqueName: \"kubernetes.io/projected/f56574c2-3b3a-4e31-8994-0a960b75c59e-kube-api-access-7hv9r\") pod \"authorino-557f876f69-56fq7\" (UID: \"f56574c2-3b3a-4e31-8994-0a960b75c59e\") " pod="kuadrant-system/authorino-557f876f69-56fq7"
Sep 15 21:19:54.180991 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.180905    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-757c596f4c-9lkjt"]
Sep 15 21:19:54.184538 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.184514    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-757c596f4c-9lkjt"
Sep 15 21:19:54.186684 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.186661    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 21:19:54.187934 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.187917    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hv9r\" (UniqueName: \"kubernetes.io/projected/f56574c2-3b3a-4e31-8994-0a960b75c59e-kube-api-access-7hv9r\") pod \"f56574c2-3b3a-4e31-8994-0a960b75c59e\" (UID: \"f56574c2-3b3a-4e31-8994-0a960b75c59e\") "
Sep 15 21:19:54.190094 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.190070    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56574c2-3b3a-4e31-8994-0a960b75c59e-kube-api-access-7hv9r" (OuterVolumeSpecName: "kube-api-access-7hv9r") pod "f56574c2-3b3a-4e31-8994-0a960b75c59e" (UID: "f56574c2-3b3a-4e31-8994-0a960b75c59e"). InnerVolumeSpecName "kube-api-access-7hv9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:19:54.194493 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.194470    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-757c596f4c-9lkjt"]
Sep 15 21:19:54.289263 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.289228    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzxr\" (UniqueName: \"kubernetes.io/projected/0d7b0e0d-2538-48f1-9354-ef8c430c8e38-kube-api-access-mnzxr\") pod \"authorino-757c596f4c-9lkjt\" (UID: \"0d7b0e0d-2538-48f1-9354-ef8c430c8e38\") " pod="kuadrant-system/authorino-757c596f4c-9lkjt"
Sep 15 21:19:54.289470 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.289336    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d7b0e0d-2538-48f1-9354-ef8c430c8e38-tls-cert\") pod \"authorino-757c596f4c-9lkjt\" (UID: \"0d7b0e0d-2538-48f1-9354-ef8c430c8e38\") " pod="kuadrant-system/authorino-757c596f4c-9lkjt"
Sep 15 21:19:54.289470 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.289444    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hv9r\" (UniqueName: \"kubernetes.io/projected/f56574c2-3b3a-4e31-8994-0a960b75c59e-kube-api-access-7hv9r\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:19:54.390882 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.390843    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d7b0e0d-2538-48f1-9354-ef8c430c8e38-tls-cert\") pod \"authorino-757c596f4c-9lkjt\" (UID: \"0d7b0e0d-2538-48f1-9354-ef8c430c8e38\") " pod="kuadrant-system/authorino-757c596f4c-9lkjt"
Sep 15 21:19:54.391055 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.390923    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzxr\" (UniqueName: \"kubernetes.io/projected/0d7b0e0d-2538-48f1-9354-ef8c430c8e38-kube-api-access-mnzxr\") pod \"authorino-757c596f4c-9lkjt\" (UID: \"0d7b0e0d-2538-48f1-9354-ef8c430c8e38\") " pod="kuadrant-system/authorino-757c596f4c-9lkjt"
Sep 15 21:19:54.393342 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.393311    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d7b0e0d-2538-48f1-9354-ef8c430c8e38-tls-cert\") pod \"authorino-757c596f4c-9lkjt\" (UID: \"0d7b0e0d-2538-48f1-9354-ef8c430c8e38\") " pod="kuadrant-system/authorino-757c596f4c-9lkjt"
Sep 15 21:19:54.399486 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.399464    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzxr\" (UniqueName: \"kubernetes.io/projected/0d7b0e0d-2538-48f1-9354-ef8c430c8e38-kube-api-access-mnzxr\") pod \"authorino-757c596f4c-9lkjt\" (UID: \"0d7b0e0d-2538-48f1-9354-ef8c430c8e38\") " pod="kuadrant-system/authorino-757c596f4c-9lkjt"
Sep 15 21:19:54.501868 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.501775    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-757c596f4c-9lkjt"
Sep 15 21:19:54.783484 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:54.783411    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-757c596f4c-9lkjt"]
Sep 15 21:19:54.785775 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:19:54.785747    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d7b0e0d_2538_48f1_9354_ef8c430c8e38.slice/crio-fc04cda3666785161ff3f362c96273d43a24ebf71d71f42699bdb5a741caeb33 WatchSource:0}: Error finding container fc04cda3666785161ff3f362c96273d43a24ebf71d71f42699bdb5a741caeb33: Status 404 returned error can't find the container with id fc04cda3666785161ff3f362c96273d43a24ebf71d71f42699bdb5a741caeb33
Sep 15 21:19:55.094814 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:55.094722    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-757c596f4c-9lkjt" event={"ID":"0d7b0e0d-2538-48f1-9354-ef8c430c8e38","Type":"ContainerStarted","Data":"fc04cda3666785161ff3f362c96273d43a24ebf71d71f42699bdb5a741caeb33"}
Sep 15 21:19:55.094814 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:55.094780    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-56fq7"
Sep 15 21:19:55.127924 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:55.127895    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-56fq7"]
Sep 15 21:19:55.134441 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:55.134412    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-56fq7"]
Sep 15 21:19:56.100823 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:56.100788    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-757c596f4c-9lkjt" event={"ID":"0d7b0e0d-2538-48f1-9354-ef8c430c8e38","Type":"ContainerStarted","Data":"2b4f25ee7c2364c0d86ab410656f1be0395fe73a905d820ce44088bfa9cf3966"}
Sep 15 21:19:56.120273 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:56.120217    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-757c596f4c-9lkjt" podStartSLOduration=1.628593345 podStartE2EDuration="2.120199532s" totalImagesPullingTime="491.606187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:19:56.118010843 +0000 UTC m=+563.979944648" watchObservedRunningTime="2026-09-15 21:19:56.120199532 +0000 UTC m=+563.982133337"
Sep 15 21:19:56.672187 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:19:56.672157    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f56574c2-3b3a-4e31-8994-0a960b75c59e" path="/var/lib/kubelet/pods/f56574c2-3b3a-4e31-8994-0a960b75c59e/volumes"
Sep 15 21:20:32.513432 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:32.513405    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:20:32.515830 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:32.515808    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:20:38.872911 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:38.872878    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-5wksw"]
Sep 15 21:20:38.876501 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:38.876474    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5wksw"
Sep 15 21:20:38.879129 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:38.878904    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 15 21:20:38.879129 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:38.878929    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tx2kg\""
Sep 15 21:20:38.884959 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:38.884911    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-5wksw"]
Sep 15 21:20:38.995093 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:38.995058    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa501378-f7fd-4481-a046-98e4bfc9bb6d-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-5wksw\" (UID: \"fa501378-f7fd-4481-a046-98e4bfc9bb6d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5wksw"
Sep 15 21:20:38.995093 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:38.995090    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8fl5\" (UniqueName: \"kubernetes.io/projected/fa501378-f7fd-4481-a046-98e4bfc9bb6d-kube-api-access-q8fl5\") pod \"payload-pre-processing-59fbc47bf8-5wksw\" (UID: \"fa501378-f7fd-4481-a046-98e4bfc9bb6d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5wksw"
Sep 15 21:20:39.096474 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:39.096429    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa501378-f7fd-4481-a046-98e4bfc9bb6d-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-5wksw\" (UID: \"fa501378-f7fd-4481-a046-98e4bfc9bb6d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5wksw"
Sep 15 21:20:39.096657 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:39.096481    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8fl5\" (UniqueName: \"kubernetes.io/projected/fa501378-f7fd-4481-a046-98e4bfc9bb6d-kube-api-access-q8fl5\") pod \"payload-pre-processing-59fbc47bf8-5wksw\" (UID: \"fa501378-f7fd-4481-a046-98e4bfc9bb6d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5wksw"
Sep 15 21:20:39.097172 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:39.097153    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa501378-f7fd-4481-a046-98e4bfc9bb6d-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-5wksw\" (UID: \"fa501378-f7fd-4481-a046-98e4bfc9bb6d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5wksw"
Sep 15 21:20:39.117863 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:39.117829    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8fl5\" (UniqueName: \"kubernetes.io/projected/fa501378-f7fd-4481-a046-98e4bfc9bb6d-kube-api-access-q8fl5\") pod \"payload-pre-processing-59fbc47bf8-5wksw\" (UID: \"fa501378-f7fd-4481-a046-98e4bfc9bb6d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5wksw"
Sep 15 21:20:39.190809 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:39.190727    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5wksw"
Sep 15 21:20:39.475363 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:39.475202    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-5wksw"]
Sep 15 21:20:39.478323 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:20:39.478297    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa501378_f7fd_4481_a046_98e4bfc9bb6d.slice/crio-deb8ba2774a3ee5abe4524957e1a2d15502004240246b4351616d9a55e8fdde7 WatchSource:0}: Error finding container deb8ba2774a3ee5abe4524957e1a2d15502004240246b4351616d9a55e8fdde7: Status 404 returned error can't find the container with id deb8ba2774a3ee5abe4524957e1a2d15502004240246b4351616d9a55e8fdde7
Sep 15 21:20:40.318496 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:40.318415    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5wksw" event={"ID":"fa501378-f7fd-4481-a046-98e4bfc9bb6d","Type":"ContainerStarted","Data":"deb8ba2774a3ee5abe4524957e1a2d15502004240246b4351616d9a55e8fdde7"}
Sep 15 21:20:43.336034 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:43.335994    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5wksw" event={"ID":"fa501378-f7fd-4481-a046-98e4bfc9bb6d","Type":"ContainerStarted","Data":"d36693edded97fcc3ef06b44889bdd11da23d68201825521a99bd30b2654097f"}
Sep 15 21:20:43.336451 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:43.336065    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5wksw"
Sep 15 21:20:43.354240 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:43.354172    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5wksw" podStartSLOduration=2.5247692109999997 podStartE2EDuration="5.35414898s" totalImagesPullingTime="2.829379769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:20:43.353302144 +0000 UTC m=+611.215235952" watchObservedRunningTime="2026-09-15 21:20:43.35414898 +0000 UTC m=+611.216082786"
Sep 15 21:20:48.362257 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:48.362173    2584 generic.go:356] "Generic (PLEG): container finished" podID="fd952d22-e6de-4c5d-acdb-8da8fb46ffb1" containerID="07404b73ce54b265a0ad433b2a747c6f916a517a09c0b5d3cd812e265d9aa18f" exitCode=0
Sep 15 21:20:48.362679 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:48.362248    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4" event={"ID":"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1","Type":"ContainerDied","Data":"07404b73ce54b265a0ad433b2a747c6f916a517a09c0b5d3cd812e265d9aa18f"}
Sep 15 21:20:48.362679 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:48.362658    2584 scope.go:117] "RemoveContainer" containerID="07404b73ce54b265a0ad433b2a747c6f916a517a09c0b5d3cd812e265d9aa18f"
Sep 15 21:20:48.363662 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:48.363638    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:20:48.363718 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:48.363706    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:20:48.363754 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:48.363739    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:20:49.373096 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:49.373046    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4" event={"ID":"fd952d22-e6de-4c5d-acdb-8da8fb46ffb1","Type":"ContainerStarted","Data":"260e75e5d170c4aba65a7c809a85f9c3f521d599d0879767d23ca6f1090035ca"}
Sep 15 21:20:49.607407 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:49.607332    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:20:49.607580 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:49.607554    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-c747fcbf8-jcbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 15 21:20:49.607627 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:49.607599    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4" podUID="fd952d22-e6de-4c5d-acdb-8da8fb46ffb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 15 21:20:50.607346 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:50.607305    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-c747fcbf8-jcbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 15 21:20:50.607824 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:50.607410    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4" podUID="fd952d22-e6de-4c5d-acdb-8da8fb46ffb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 15 21:20:51.607527 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:51.607483    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-c747fcbf8-jcbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 15 21:20:51.607924 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:51.607553    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4" podUID="fd952d22-e6de-4c5d-acdb-8da8fb46ffb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 15 21:20:52.607472 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:52.607439    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-c747fcbf8-jcbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 15 21:20:52.607636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:52.607496    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4" podUID="fd952d22-e6de-4c5d-acdb-8da8fb46ffb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 15 21:20:53.607511 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:53.607474    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-c747fcbf8-jcbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 15 21:20:53.607692 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:53.607544    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4" podUID="fd952d22-e6de-4c5d-acdb-8da8fb46ffb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 15 21:20:54.343154 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:54.343126    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5wksw"
Sep 15 21:20:54.607448 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:54.607350    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-c747fcbf8-jcbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 15 21:20:54.607606 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:54.607441    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4" podUID="fd952d22-e6de-4c5d-acdb-8da8fb46ffb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 15 21:20:55.611655 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:55.611580    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:20:55.612031 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:55.611957    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:20:55.612746 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:20:55.612727    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-jcbs4"
Sep 15 21:21:06.121659 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.121623    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"]
Sep 15 21:21:06.125045 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.125026    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.127274 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.127248    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 21:21:06.127430 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.127323    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 21:21:06.127430 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.127335    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mjjfq\""
Sep 15 21:21:06.127430 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.127399    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 15 21:21:06.132746 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.132724    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"]
Sep 15 21:21:06.150120 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.150086    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa4b4-5064-4610-9b10-d005889f006c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.150287 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.150169    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa4b4-5064-4610-9b10-d005889f006c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.150287 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.150241    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa4b4-5064-4610-9b10-d005889f006c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.150365 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.150325    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa4b4-5064-4610-9b10-d005889f006c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.150426 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.150363    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac3aa4b4-5064-4610-9b10-d005889f006c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.150462 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.150449    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbk24\" (UniqueName: \"kubernetes.io/projected/ac3aa4b4-5064-4610-9b10-d005889f006c-kube-api-access-pbk24\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.250935 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.250897    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa4b4-5064-4610-9b10-d005889f006c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.251126 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.250989    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa4b4-5064-4610-9b10-d005889f006c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.251126 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.251024    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac3aa4b4-5064-4610-9b10-d005889f006c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.251126 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.251079    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbk24\" (UniqueName: \"kubernetes.io/projected/ac3aa4b4-5064-4610-9b10-d005889f006c-kube-api-access-pbk24\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.251271 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.251125    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa4b4-5064-4610-9b10-d005889f006c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.251271 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.251170    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa4b4-5064-4610-9b10-d005889f006c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.251369 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.251330    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa4b4-5064-4610-9b10-d005889f006c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.251536 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.251501    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa4b4-5064-4610-9b10-d005889f006c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.251672 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.251553    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa4b4-5064-4610-9b10-d005889f006c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.253278 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.253259    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa4b4-5064-4610-9b10-d005889f006c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.253807 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.253790    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac3aa4b4-5064-4610-9b10-d005889f006c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.260715 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.260692    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbk24\" (UniqueName: \"kubernetes.io/projected/ac3aa4b4-5064-4610-9b10-d005889f006c-kube-api-access-pbk24\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz\" (UID: \"ac3aa4b4-5064-4610-9b10-d005889f006c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.437209 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.437102    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:06.723477 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:06.723354    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"]
Sep 15 21:21:06.726450 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:21:06.726415    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3aa4b4_5064_4610_9b10_d005889f006c.slice/crio-7f93a751be297448bcc93556f46661925f8a38ec71f091157d7bf35eb257e5da WatchSource:0}: Error finding container 7f93a751be297448bcc93556f46661925f8a38ec71f091157d7bf35eb257e5da: Status 404 returned error can't find the container with id 7f93a751be297448bcc93556f46661925f8a38ec71f091157d7bf35eb257e5da
Sep 15 21:21:07.467996 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:07.467953    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz" event={"ID":"ac3aa4b4-5064-4610-9b10-d005889f006c","Type":"ContainerStarted","Data":"7f93a751be297448bcc93556f46661925f8a38ec71f091157d7bf35eb257e5da"}
Sep 15 21:21:08.475314 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:08.475278    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz" event={"ID":"ac3aa4b4-5064-4610-9b10-d005889f006c","Type":"ContainerStarted","Data":"137e5cd1a2e334d6cfd628f4b12ea2ba254139bea03f8ee38debd4a4f44ba2ba"}
Sep 15 21:21:08.496219 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:08.496168    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz" podStartSLOduration=1.429138641 podStartE2EDuration="2.496152012s" totalImagesPullingTime="1.067013371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:21:08.49490543 +0000 UTC m=+636.356839237" watchObservedRunningTime="2026-09-15 21:21:08.496152012 +0000 UTC m=+636.358085818"
Sep 15 21:21:16.438302 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:16.438265    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:16.438796 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:16.438323    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:16.451014 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:16.450984    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:16.526720 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:16.526691    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz"
Sep 15 21:21:29.509104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.509070    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"]
Sep 15 21:21:29.512584 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.512564    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.514852 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.514829    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 15 21:21:29.523741 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.523698    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"]
Sep 15 21:21:29.579004 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.578970    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f91de23a-0f65-4377-8f30-1039e25e8448-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.579185 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.579015    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7p4c\" (UniqueName: \"kubernetes.io/projected/f91de23a-0f65-4377-8f30-1039e25e8448-kube-api-access-s7p4c\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.579185 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.579039    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f91de23a-0f65-4377-8f30-1039e25e8448-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.579185 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.579141    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f91de23a-0f65-4377-8f30-1039e25e8448-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.579314 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.579193    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f91de23a-0f65-4377-8f30-1039e25e8448-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.579314 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.579218    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f91de23a-0f65-4377-8f30-1039e25e8448-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.680496 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.680456    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f91de23a-0f65-4377-8f30-1039e25e8448-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.680704 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.680514    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7p4c\" (UniqueName: \"kubernetes.io/projected/f91de23a-0f65-4377-8f30-1039e25e8448-kube-api-access-s7p4c\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.680704 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.680539    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f91de23a-0f65-4377-8f30-1039e25e8448-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.680704 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.680582    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f91de23a-0f65-4377-8f30-1039e25e8448-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.680704 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.680649    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f91de23a-0f65-4377-8f30-1039e25e8448-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.680704 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.680677    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f91de23a-0f65-4377-8f30-1039e25e8448-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.681017 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.680995    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f91de23a-0f65-4377-8f30-1039e25e8448-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.681072 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.681020    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f91de23a-0f65-4377-8f30-1039e25e8448-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.681176 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.681146    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f91de23a-0f65-4377-8f30-1039e25e8448-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.682882 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.682866    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f91de23a-0f65-4377-8f30-1039e25e8448-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.683212 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.683188    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f91de23a-0f65-4377-8f30-1039e25e8448-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.689890 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.689872    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7p4c\" (UniqueName: \"kubernetes.io/projected/f91de23a-0f65-4377-8f30-1039e25e8448-kube-api-access-s7p4c\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j\" (UID: \"f91de23a-0f65-4377-8f30-1039e25e8448\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:29.824373 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:29.824334    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:30.098607 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:30.098572    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"]
Sep 15 21:21:30.100898 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:21:30.100870    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91de23a_0f65_4377_8f30_1039e25e8448.slice/crio-860b47b12865c9d69ff2b40053021f7c6ecf56dc2ed7f45fe4437aaada97417c WatchSource:0}: Error finding container 860b47b12865c9d69ff2b40053021f7c6ecf56dc2ed7f45fe4437aaada97417c: Status 404 returned error can't find the container with id 860b47b12865c9d69ff2b40053021f7c6ecf56dc2ed7f45fe4437aaada97417c
Sep 15 21:21:30.102165 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:30.102149    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:21:30.589510 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:30.589475    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j" event={"ID":"f91de23a-0f65-4377-8f30-1039e25e8448","Type":"ContainerStarted","Data":"e0f441c1523dd4f56cf5ead409db8282f84d08123fb839ad6eb3cacae79935e4"}
Sep 15 21:21:30.589934 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:30.589532    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j" event={"ID":"f91de23a-0f65-4377-8f30-1039e25e8448","Type":"ContainerStarted","Data":"860b47b12865c9d69ff2b40053021f7c6ecf56dc2ed7f45fe4437aaada97417c"}
Sep 15 21:21:30.609097 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:30.608994    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j" podStartSLOduration=1.412767765 podStartE2EDuration="1.608976718s" totalImagesPullingTime="196.208953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:21:30.608116395 +0000 UTC m=+658.470050202" watchObservedRunningTime="2026-09-15 21:21:30.608976718 +0000 UTC m=+658.470910527"
Sep 15 21:21:39.824709 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:39.824658    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:39.825167 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:39.824740    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:39.837210 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:39.837178    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:40.648277 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:40.648247    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j"
Sep 15 21:21:58.617831 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.617796    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"]
Sep 15 21:21:58.621302 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.621286    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.624042 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.624024    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 15 21:21:58.630210 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.630186    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"]
Sep 15 21:21:58.751950 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.751902    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.752148 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.751976    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.752148 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.752054    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.752148 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.752124    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb2nm\" (UniqueName: \"kubernetes.io/projected/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-kube-api-access-zb2nm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.752330 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.752149    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.752330 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.752175    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.852791 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.852748    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.852996 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.852801    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.852996 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.852866    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.852996 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.852911    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.852996 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.852978    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb2nm\" (UniqueName: \"kubernetes.io/projected/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-kube-api-access-zb2nm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.853213 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.853073    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.853308 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.853230    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.853308 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.853267    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.853457 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.853428    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.855237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.855210    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.855393 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.855365    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.863465 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.863422    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb2nm\" (UniqueName: \"kubernetes.io/projected/421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf-kube-api-access-zb2nm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vm45z\" (UID: \"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:58.933491 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:58.933388    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:21:59.208265 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:59.208185    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"]
Sep 15 21:21:59.211654 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:21:59.211625    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421c7bc4_ebd3_4f5b_aa03_0a3d112ca2cf.slice/crio-d9c6b2fbfdd64e40a39392c4db9391647536d97b15f028b44cab5104c3167935 WatchSource:0}: Error finding container d9c6b2fbfdd64e40a39392c4db9391647536d97b15f028b44cab5104c3167935: Status 404 returned error can't find the container with id d9c6b2fbfdd64e40a39392c4db9391647536d97b15f028b44cab5104c3167935
Sep 15 21:21:59.731765 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:59.731727    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z" event={"ID":"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf","Type":"ContainerStarted","Data":"3cd93b358c0495a14092c1ed87aee10b5dbf04170a1dd21a6e6b239e6897d381"}
Sep 15 21:21:59.732163 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:59.731800    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z" event={"ID":"421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf","Type":"ContainerStarted","Data":"d9c6b2fbfdd64e40a39392c4db9391647536d97b15f028b44cab5104c3167935"}
Sep 15 21:21:59.751266 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:21:59.750839    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z" podStartSLOduration=1.562979913 podStartE2EDuration="1.750782691s" totalImagesPullingTime="187.802778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:21:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:21:59.750313327 +0000 UTC m=+687.612247155" watchObservedRunningTime="2026-09-15 21:21:59.750782691 +0000 UTC m=+687.612716497"
Sep 15 21:22:08.934450 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:22:08.934409    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:22:08.934847 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:22:08.934471    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:22:08.946983 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:22:08.946958    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:22:09.793629 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:22:09.793599    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vm45z"
Sep 15 21:23:34.680690 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:34.680078    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-757c596f4c-9lkjt"]
Sep 15 21:23:34.680690 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:34.680554    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-757c596f4c-9lkjt" podUID="0d7b0e0d-2538-48f1-9354-ef8c430c8e38" containerName="authorino" containerID="cri-o://2b4f25ee7c2364c0d86ab410656f1be0395fe73a905d820ce44088bfa9cf3966" gracePeriod=30
Sep 15 21:23:34.933342 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:34.933280    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-757c596f4c-9lkjt"
Sep 15 21:23:34.991479 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:34.991445    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnzxr\" (UniqueName: \"kubernetes.io/projected/0d7b0e0d-2538-48f1-9354-ef8c430c8e38-kube-api-access-mnzxr\") pod \"0d7b0e0d-2538-48f1-9354-ef8c430c8e38\" (UID: \"0d7b0e0d-2538-48f1-9354-ef8c430c8e38\") "
Sep 15 21:23:34.991671 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:34.991540    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d7b0e0d-2538-48f1-9354-ef8c430c8e38-tls-cert\") pod \"0d7b0e0d-2538-48f1-9354-ef8c430c8e38\" (UID: \"0d7b0e0d-2538-48f1-9354-ef8c430c8e38\") "
Sep 15 21:23:34.993633 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:34.993596    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7b0e0d-2538-48f1-9354-ef8c430c8e38-kube-api-access-mnzxr" (OuterVolumeSpecName: "kube-api-access-mnzxr") pod "0d7b0e0d-2538-48f1-9354-ef8c430c8e38" (UID: "0d7b0e0d-2538-48f1-9354-ef8c430c8e38"). InnerVolumeSpecName "kube-api-access-mnzxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:23:35.002160 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:35.002124    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7b0e0d-2538-48f1-9354-ef8c430c8e38-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "0d7b0e0d-2538-48f1-9354-ef8c430c8e38" (UID: "0d7b0e0d-2538-48f1-9354-ef8c430c8e38"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:23:35.092941 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:35.092907    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnzxr\" (UniqueName: \"kubernetes.io/projected/0d7b0e0d-2538-48f1-9354-ef8c430c8e38-kube-api-access-mnzxr\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:23:35.092941 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:35.092936    2584 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d7b0e0d-2538-48f1-9354-ef8c430c8e38-tls-cert\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:23:35.210969 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:35.210888    2584 generic.go:356] "Generic (PLEG): container finished" podID="0d7b0e0d-2538-48f1-9354-ef8c430c8e38" containerID="2b4f25ee7c2364c0d86ab410656f1be0395fe73a905d820ce44088bfa9cf3966" exitCode=0
Sep 15 21:23:35.210969 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:35.210931    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-757c596f4c-9lkjt"
Sep 15 21:23:35.210969 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:35.210942    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-757c596f4c-9lkjt" event={"ID":"0d7b0e0d-2538-48f1-9354-ef8c430c8e38","Type":"ContainerDied","Data":"2b4f25ee7c2364c0d86ab410656f1be0395fe73a905d820ce44088bfa9cf3966"}
Sep 15 21:23:35.211243 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:35.210974    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-757c596f4c-9lkjt" event={"ID":"0d7b0e0d-2538-48f1-9354-ef8c430c8e38","Type":"ContainerDied","Data":"fc04cda3666785161ff3f362c96273d43a24ebf71d71f42699bdb5a741caeb33"}
Sep 15 21:23:35.211243 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:35.210991    2584 scope.go:117] "RemoveContainer" containerID="2b4f25ee7c2364c0d86ab410656f1be0395fe73a905d820ce44088bfa9cf3966"
Sep 15 21:23:35.220849 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:35.220830    2584 scope.go:117] "RemoveContainer" containerID="2b4f25ee7c2364c0d86ab410656f1be0395fe73a905d820ce44088bfa9cf3966"
Sep 15 21:23:35.221184 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:23:35.221158    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4f25ee7c2364c0d86ab410656f1be0395fe73a905d820ce44088bfa9cf3966\": container with ID starting with 2b4f25ee7c2364c0d86ab410656f1be0395fe73a905d820ce44088bfa9cf3966 not found: ID does not exist" containerID="2b4f25ee7c2364c0d86ab410656f1be0395fe73a905d820ce44088bfa9cf3966"
Sep 15 21:23:35.221277 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:35.221191    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4f25ee7c2364c0d86ab410656f1be0395fe73a905d820ce44088bfa9cf3966"} err="failed to get container status \"2b4f25ee7c2364c0d86ab410656f1be0395fe73a905d820ce44088bfa9cf3966\": rpc error: code = NotFound desc = could not find container \"2b4f25ee7c2364c0d86ab410656f1be0395fe73a905d820ce44088bfa9cf3966\": container with ID starting with 2b4f25ee7c2364c0d86ab410656f1be0395fe73a905d820ce44088bfa9cf3966 not found: ID does not exist"
Sep 15 21:23:35.240022 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:35.239996    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-757c596f4c-9lkjt"]
Sep 15 21:23:35.246078 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:35.246042    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-757c596f4c-9lkjt"]
Sep 15 21:23:36.672651 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:36.672621    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d7b0e0d-2538-48f1-9354-ef8c430c8e38" path="/var/lib/kubelet/pods/0d7b0e0d-2538-48f1-9354-ef8c430c8e38/volumes"
Sep 15 21:23:52.457284 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.457252    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"]
Sep 15 21:23:52.457821 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.457744    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d7b0e0d-2538-48f1-9354-ef8c430c8e38" containerName="authorino"
Sep 15 21:23:52.457821 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.457762    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d7b0e0d-2538-48f1-9354-ef8c430c8e38" containerName="authorino"
Sep 15 21:23:52.457904 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.457836    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d7b0e0d-2538-48f1-9354-ef8c430c8e38" containerName="authorino"
Sep 15 21:23:52.462298 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.462271    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.464919 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.464900    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-cfc1f4-openshift-default-dockercfg-j8hzp\""
Sep 15 21:23:52.472326 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.472295    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"]
Sep 15 21:23:52.546957 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.546926    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"]
Sep 15 21:23:52.550803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.550768    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.555312 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.555291    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6f66f7-openshift-default-dockercfg-4zrgt\""
Sep 15 21:23:52.556841 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.556815    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-envoy\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.556967 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.556855    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-workload-socket\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.556967 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.556888    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab13af8-a313-43a0-8447-3b16b24ee632-istiod-ca-cert\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.556967 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.556911    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phxcw\" (UniqueName: \"kubernetes.io/projected/0ab13af8-a313-43a0-8447-3b16b24ee632-kube-api-access-phxcw\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.556967 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.556947    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-data\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.557161 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.557033    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-podinfo\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.557161 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.557074    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-credential-socket\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.557278 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.557170    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-token\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.557278 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.557200    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-workload-certs\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.569630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.569599    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"]
Sep 15 21:23:52.628933 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.628900    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"]
Sep 15 21:23:52.633540 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.633517    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.637258 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.637232    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-318c22-openshift-default-dockercfg-6dzl2\""
Sep 15 21:23:52.650367 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.650345    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"]
Sep 15 21:23:52.657808 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.657778    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-workload-certs\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.657964 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.657828    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-envoy\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.657964 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.657860    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-workload-socket\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.657964 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.657910    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab13af8-a313-43a0-8447-3b16b24ee632-istiod-ca-cert\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.657964 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.657936    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phxcw\" (UniqueName: \"kubernetes.io/projected/0ab13af8-a313-43a0-8447-3b16b24ee632-kube-api-access-phxcw\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.658198 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.657969    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istiod-ca-cert\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.658198 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.658004    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-data\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.658198 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.658033    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-credential-socket\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.658198 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.658070    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-envoy\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.658198 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.658104    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-podinfo\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.658198 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.658135    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-credential-socket\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.658198 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.658187    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-podinfo\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.658604 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.658221    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-data\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.658604 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.658269    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-workload-socket\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.658604 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.658318    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-token\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.658604 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.658357    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-token\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.658604 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.658403    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-workload-certs\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.658604 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.658451    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2b89\" (UniqueName: \"kubernetes.io/projected/efc6a2e8-4101-46b5-ab18-125c552cd3a1-kube-api-access-d2b89\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.658604 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.658541    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-data\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.658997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.658809    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-workload-socket\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.659452 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.659430    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab13af8-a313-43a0-8447-3b16b24ee632-istiod-ca-cert\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.659694 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.659670    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-credential-socket\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.659817 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.659793    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-workload-certs\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.661762 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.661690    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-envoy\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.663122 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.663091    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-podinfo\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.675724 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.668630    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-token\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.675724 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.672064    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phxcw\" (UniqueName: \"kubernetes.io/projected/0ab13af8-a313-43a0-8447-3b16b24ee632-kube-api-access-phxcw\") pod \"e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") " pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.760038 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.759936    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-credential-socket\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.760038 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.759994    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-workload-socket\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.760267 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.760040    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-podinfo\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.760267 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.760078    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-data\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.760267 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.760125    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-workload-socket\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.760267 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.760175    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-token\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.760267 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.760208    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-data\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.760267 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.760233    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw4xt\" (UniqueName: \"kubernetes.io/projected/0425206a-e3e3-4644-9066-69c1fa315c2f-kube-api-access-cw4xt\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.760623 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.760289    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2b89\" (UniqueName: \"kubernetes.io/projected/efc6a2e8-4101-46b5-ab18-125c552cd3a1-kube-api-access-d2b89\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.760623 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.760329    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-workload-certs\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.760623 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.760397    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-workload-certs\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.760623 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.760435    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istiod-ca-cert\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.760623 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.760462    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-token\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.760623 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.760486    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0425206a-e3e3-4644-9066-69c1fa315c2f-istiod-ca-cert\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.760623 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.760521    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-envoy\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.760623 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.760550    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-credential-socket\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.760623 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.760582    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-podinfo\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.761080 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.760628    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-envoy\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.761149 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.761097    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-credential-socket\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.761656 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.761322    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-data\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.761656 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.761436    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-workload-socket\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.761656 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.761511    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istiod-ca-cert\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.761901 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.761701    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-workload-certs\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.763549 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.763463    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-podinfo\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.764073 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.764043    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-envoy\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.770486 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.770461    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-token\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.771148 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.771128    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2b89\" (UniqueName: \"kubernetes.io/projected/efc6a2e8-4101-46b5-ab18-125c552cd3a1-kube-api-access-d2b89\") pod \"e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") " pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.774131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.774033    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:52.862924 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.862516    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-data\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.862924 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.862559    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw4xt\" (UniqueName: \"kubernetes.io/projected/0425206a-e3e3-4644-9066-69c1fa315c2f-kube-api-access-cw4xt\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.862924 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.862635    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-workload-certs\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.862924 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.862668    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-token\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.862924 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.862691    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0425206a-e3e3-4644-9066-69c1fa315c2f-istiod-ca-cert\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.862924 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.862721    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-envoy\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.862924 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.862755    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-podinfo\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.862924 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.862795    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-credential-socket\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.862924 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.862825    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-workload-socket\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.863539 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.863193    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:52.863539 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.863229    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-workload-socket\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.863539 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.863500    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-data\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.866424 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.864045    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-workload-certs\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.866424 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.865232    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0425206a-e3e3-4644-9066-69c1fa315c2f-istiod-ca-cert\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.866424 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.865564    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-credential-socket\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.868497 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.868474    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-envoy\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.871406 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.870493    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-podinfo\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.877006 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.876980    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-token\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.878579 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.878557    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw4xt\" (UniqueName: \"kubernetes.io/projected/0425206a-e3e3-4644-9066-69c1fa315c2f-kube-api-access-cw4xt\") pod \"e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") " pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:52.948012 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:52.947817    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:53.065498 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.064261    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"]
Sep 15 21:23:53.067419 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:23:53.067050    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab13af8_a313_43a0_8447_3b16b24ee632.slice/crio-db3190b3832574bebf1812e247d2258fd209f582466b8c9f5fe807bf35ef729f WatchSource:0}: Error finding container db3190b3832574bebf1812e247d2258fd209f582466b8c9f5fe807bf35ef729f: Status 404 returned error can't find the container with id db3190b3832574bebf1812e247d2258fd209f582466b8c9f5fe807bf35ef729f
Sep 15 21:23:53.069473 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.069334    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:23:53.069473 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.069418    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:23:53.069473 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.069458    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:23:53.156281 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.156255    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"]
Sep 15 21:23:53.156648 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:23:53.156622    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc6a2e8_4101_46b5_ab18_125c552cd3a1.slice/crio-c6cb7ead924de1ec448d98437efef13015da318152cf72001f04df0b42854a5a WatchSource:0}: Error finding container c6cb7ead924de1ec448d98437efef13015da318152cf72001f04df0b42854a5a: Status 404 returned error can't find the container with id c6cb7ead924de1ec448d98437efef13015da318152cf72001f04df0b42854a5a
Sep 15 21:23:53.160046 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.160007    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:23:53.160147 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.160124    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:23:53.160204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.160168    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:23:53.255850 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.255763    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"]
Sep 15 21:23:53.261278 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:23:53.261251    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0425206a_e3e3_4644_9066_69c1fa315c2f.slice/crio-f831ed38ec5f31deeeab32326281be5afc111aa18ac562075894cacd511815a1 WatchSource:0}: Error finding container f831ed38ec5f31deeeab32326281be5afc111aa18ac562075894cacd511815a1: Status 404 returned error can't find the container with id f831ed38ec5f31deeeab32326281be5afc111aa18ac562075894cacd511815a1
Sep 15 21:23:53.264729 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.264691    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:23:53.264827 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.264781    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:23:53.264827 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.264824    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:23:53.318264 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.317403    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj" event={"ID":"0ab13af8-a313-43a0-8447-3b16b24ee632","Type":"ContainerStarted","Data":"a49805afa90a74c3c8debd5f3e6e865128bf37232fbaa40587055bc0d8d6a74a"}
Sep 15 21:23:53.318264 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.317464    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj" event={"ID":"0ab13af8-a313-43a0-8447-3b16b24ee632","Type":"ContainerStarted","Data":"db3190b3832574bebf1812e247d2258fd209f582466b8c9f5fe807bf35ef729f"}
Sep 15 21:23:53.319718 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.319655    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs" event={"ID":"0425206a-e3e3-4644-9066-69c1fa315c2f","Type":"ContainerStarted","Data":"f831ed38ec5f31deeeab32326281be5afc111aa18ac562075894cacd511815a1"}
Sep 15 21:23:53.322846 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.321369    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt" event={"ID":"efc6a2e8-4101-46b5-ab18-125c552cd3a1","Type":"ContainerStarted","Data":"19e481f6870af45588fae621c66bf5cdd14149fdd6916b56e9c770fee5da86d2"}
Sep 15 21:23:53.322846 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.321431    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt" event={"ID":"efc6a2e8-4101-46b5-ab18-125c552cd3a1","Type":"ContainerStarted","Data":"c6cb7ead924de1ec448d98437efef13015da318152cf72001f04df0b42854a5a"}
Sep 15 21:23:53.377426 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.377342    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj" podStartSLOduration=1.377320921 podStartE2EDuration="1.377320921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:23:53.355751577 +0000 UTC m=+801.217685384" watchObservedRunningTime="2026-09-15 21:23:53.377320921 +0000 UTC m=+801.239254718"
Sep 15 21:23:53.378126 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.378084    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt" podStartSLOduration=1.378057873 podStartE2EDuration="1.378057873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:23:53.377319133 +0000 UTC m=+801.239252942" watchObservedRunningTime="2026-09-15 21:23:53.378057873 +0000 UTC m=+801.239991681"
Sep 15 21:23:53.774308 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.774280    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:53.865458 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:53.865426    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:54.328358 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:54.328315    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs" event={"ID":"0425206a-e3e3-4644-9066-69c1fa315c2f","Type":"ContainerStarted","Data":"bdd22441729cc8e014465ba0a490f61ef090a1311e987a351872750031f68ddd"}
Sep 15 21:23:54.350389 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:54.350331    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs" podStartSLOduration=2.350314418 podStartE2EDuration="2.350314418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:23:54.347838684 +0000 UTC m=+802.209772488" watchObservedRunningTime="2026-09-15 21:23:54.350314418 +0000 UTC m=+802.212248223"
Sep 15 21:23:54.779538 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:54.779507    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:54.870356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:54.870326    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:54.949101 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:54.949067    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:54.954242 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:54.954215    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:55.334366 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:55.334334    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:55.334549 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:55.334413    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:55.334549 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:55.334443    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:55.334841 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:55.334821    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:23:55.334948 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:55.334882    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:23:55.335029 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:55.335017    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:23:57.677800 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:57.677760    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"]
Sep 15 21:23:57.682412 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:57.682391    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"
Sep 15 21:23:57.684783 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:57.684757    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-6f66f7\""
Sep 15 21:23:57.684989 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:57.684757    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-6f66f7-dockercfg-vtsv5\""
Sep 15 21:23:57.694326 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:57.694301    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"]
Sep 15 21:23:57.814407 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:57.814359    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1794ad-3eb4-450e-aaa9-424d0fe894eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w\" (UID: \"3d1794ad-3eb4-450e-aaa9-424d0fe894eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"
Sep 15 21:23:57.814575 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:57.814413    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtb4d\" (UniqueName: \"kubernetes.io/projected/3d1794ad-3eb4-450e-aaa9-424d0fe894eb-kube-api-access-xtb4d\") pod \"payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w\" (UID: \"3d1794ad-3eb4-450e-aaa9-424d0fe894eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"
Sep 15 21:23:57.915974 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:57.915939    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1794ad-3eb4-450e-aaa9-424d0fe894eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w\" (UID: \"3d1794ad-3eb4-450e-aaa9-424d0fe894eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"
Sep 15 21:23:57.915974 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:57.915974    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtb4d\" (UniqueName: \"kubernetes.io/projected/3d1794ad-3eb4-450e-aaa9-424d0fe894eb-kube-api-access-xtb4d\") pod \"payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w\" (UID: \"3d1794ad-3eb4-450e-aaa9-424d0fe894eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"
Sep 15 21:23:57.916662 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:57.916643    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1794ad-3eb4-450e-aaa9-424d0fe894eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w\" (UID: \"3d1794ad-3eb4-450e-aaa9-424d0fe894eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"
Sep 15 21:23:57.926457 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:57.926431    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtb4d\" (UniqueName: \"kubernetes.io/projected/3d1794ad-3eb4-450e-aaa9-424d0fe894eb-kube-api-access-xtb4d\") pod \"payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w\" (UID: \"3d1794ad-3eb4-450e-aaa9-424d0fe894eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"
Sep 15 21:23:57.998865 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:57.998796    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"
Sep 15 21:23:58.275302 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:58.275172    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"]
Sep 15 21:23:58.277696 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:23:58.277671    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1794ad_3eb4_450e_aaa9_424d0fe894eb.slice/crio-39007c822b8d9ccbd0d92c5dee460f695fe14f09a32a40bdb8ef358b1c0ba8bf WatchSource:0}: Error finding container 39007c822b8d9ccbd0d92c5dee460f695fe14f09a32a40bdb8ef358b1c0ba8bf: Status 404 returned error can't find the container with id 39007c822b8d9ccbd0d92c5dee460f695fe14f09a32a40bdb8ef358b1c0ba8bf
Sep 15 21:23:58.358032 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:58.357985    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w" event={"ID":"3d1794ad-3eb4-450e-aaa9-424d0fe894eb","Type":"ContainerStarted","Data":"913c2bd312451fc05cce53e109b8ab6a1ae0a55fd995ba5d36b0ce49f79c8f02"}
Sep 15 21:23:58.358192 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:58.358046    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w" event={"ID":"3d1794ad-3eb4-450e-aaa9-424d0fe894eb","Type":"ContainerStarted","Data":"39007c822b8d9ccbd0d92c5dee460f695fe14f09a32a40bdb8ef358b1c0ba8bf"}
Sep 15 21:23:58.358261 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:58.358192    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"
Sep 15 21:23:58.376078 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:58.376029    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w" podStartSLOduration=1.376013581 podStartE2EDuration="1.376013581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:23:58.374064724 +0000 UTC m=+806.235998530" watchObservedRunningTime="2026-09-15 21:23:58.376013581 +0000 UTC m=+806.237947386"
Sep 15 21:23:59.476914 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.476883    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"]
Sep 15 21:23:59.480483 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.480467    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"
Sep 15 21:23:59.483351 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.483333    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-cfc1f4\""
Sep 15 21:23:59.495780 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.495757    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"]
Sep 15 21:23:59.511424 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.511392    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"]
Sep 15 21:23:59.514814 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.514798    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"
Sep 15 21:23:59.517621 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.517597    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-cfc1f4-dockercfg-c49fk\""
Sep 15 21:23:59.533867 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.533843    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"]
Sep 15 21:23:59.535044 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.535020    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c802975-75f6-4098-8469-7fff9d2b77db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl\" (UID: \"9c802975-75f6-4098-8469-7fff9d2b77db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"
Sep 15 21:23:59.535213 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.535195    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cnrk\" (UniqueName: \"kubernetes.io/projected/9c802975-75f6-4098-8469-7fff9d2b77db-kube-api-access-8cnrk\") pod \"payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl\" (UID: \"9c802975-75f6-4098-8469-7fff9d2b77db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"
Sep 15 21:23:59.636332 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.636291    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnkpt\" (UniqueName: \"kubernetes.io/projected/6e1773c5-716b-42bb-95c9-78428703021e-kube-api-access-jnkpt\") pod \"payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp\" (UID: \"6e1773c5-716b-42bb-95c9-78428703021e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"
Sep 15 21:23:59.636527 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.636476    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1773c5-716b-42bb-95c9-78428703021e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp\" (UID: \"6e1773c5-716b-42bb-95c9-78428703021e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"
Sep 15 21:23:59.636582 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.636532    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c802975-75f6-4098-8469-7fff9d2b77db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl\" (UID: \"9c802975-75f6-4098-8469-7fff9d2b77db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"
Sep 15 21:23:59.636638 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.636587    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cnrk\" (UniqueName: \"kubernetes.io/projected/9c802975-75f6-4098-8469-7fff9d2b77db-kube-api-access-8cnrk\") pod \"payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl\" (UID: \"9c802975-75f6-4098-8469-7fff9d2b77db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"
Sep 15 21:23:59.637217 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.637198    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c802975-75f6-4098-8469-7fff9d2b77db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl\" (UID: \"9c802975-75f6-4098-8469-7fff9d2b77db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"
Sep 15 21:23:59.646971 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.646939    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cnrk\" (UniqueName: \"kubernetes.io/projected/9c802975-75f6-4098-8469-7fff9d2b77db-kube-api-access-8cnrk\") pod \"payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl\" (UID: \"9c802975-75f6-4098-8469-7fff9d2b77db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"
Sep 15 21:23:59.738236 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.738155    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnkpt\" (UniqueName: \"kubernetes.io/projected/6e1773c5-716b-42bb-95c9-78428703021e-kube-api-access-jnkpt\") pod \"payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp\" (UID: \"6e1773c5-716b-42bb-95c9-78428703021e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"
Sep 15 21:23:59.738374 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.738315    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1773c5-716b-42bb-95c9-78428703021e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp\" (UID: \"6e1773c5-716b-42bb-95c9-78428703021e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"
Sep 15 21:23:59.739110 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.739083    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1773c5-716b-42bb-95c9-78428703021e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp\" (UID: \"6e1773c5-716b-42bb-95c9-78428703021e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"
Sep 15 21:23:59.747805 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.747783    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnkpt\" (UniqueName: \"kubernetes.io/projected/6e1773c5-716b-42bb-95c9-78428703021e-kube-api-access-jnkpt\") pod \"payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp\" (UID: \"6e1773c5-716b-42bb-95c9-78428703021e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"
Sep 15 21:23:59.791134 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.791105    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"
Sep 15 21:23:59.826980 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:23:59.826703    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"
Sep 15 21:24:00.090259 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.090231    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"]
Sep 15 21:24:00.092633 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:24:00.092597    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c802975_75f6_4098_8469_7fff9d2b77db.slice/crio-90a21f3c6e31cd624e9d1aadacc258746f0b00e016b3f73a9892cc591126303d WatchSource:0}: Error finding container 90a21f3c6e31cd624e9d1aadacc258746f0b00e016b3f73a9892cc591126303d: Status 404 returned error can't find the container with id 90a21f3c6e31cd624e9d1aadacc258746f0b00e016b3f73a9892cc591126303d
Sep 15 21:24:00.137130 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.137101    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"]
Sep 15 21:24:00.139928 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:24:00.139901    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1773c5_716b_42bb_95c9_78428703021e.slice/crio-a7ae7ee0174541b71226ce84ddc3bd417ff1147e802705aa4e8461667d2ed8c9 WatchSource:0}: Error finding container a7ae7ee0174541b71226ce84ddc3bd417ff1147e802705aa4e8461667d2ed8c9: Status 404 returned error can't find the container with id a7ae7ee0174541b71226ce84ddc3bd417ff1147e802705aa4e8461667d2ed8c9
Sep 15 21:24:00.370639 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.370545    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp" event={"ID":"6e1773c5-716b-42bb-95c9-78428703021e","Type":"ContainerStarted","Data":"674473612b0ac214a6d1622407d894b290c6eb2a3438f207740bd76ec7e394fc"}
Sep 15 21:24:00.370639 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.370616    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp" event={"ID":"6e1773c5-716b-42bb-95c9-78428703021e","Type":"ContainerStarted","Data":"a7ae7ee0174541b71226ce84ddc3bd417ff1147e802705aa4e8461667d2ed8c9"}
Sep 15 21:24:00.370853 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.370670    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"
Sep 15 21:24:00.372756 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.372730    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl" event={"ID":"9c802975-75f6-4098-8469-7fff9d2b77db","Type":"ContainerStarted","Data":"31e6b12ec9979db33144465d44dbbbbcb6cdcd3839a34e9c030e34e8fa099add"}
Sep 15 21:24:00.372892 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.372773    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl" event={"ID":"9c802975-75f6-4098-8469-7fff9d2b77db","Type":"ContainerStarted","Data":"90a21f3c6e31cd624e9d1aadacc258746f0b00e016b3f73a9892cc591126303d"}
Sep 15 21:24:00.372892 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.372829    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"
Sep 15 21:24:00.391539 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.391486    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp" podStartSLOduration=1.391469592 podStartE2EDuration="1.391469592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:00.389240095 +0000 UTC m=+808.251173901" watchObservedRunningTime="2026-09-15 21:24:00.391469592 +0000 UTC m=+808.253403398"
Sep 15 21:24:00.410535 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.410492    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl" podStartSLOduration=1.41047432 podStartE2EDuration="1.41047432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:00.409458923 +0000 UTC m=+808.271392728" watchObservedRunningTime="2026-09-15 21:24:00.41047432 +0000 UTC m=+808.272408124"
Sep 15 21:24:00.502263 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.502230    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"]
Sep 15 21:24:00.506347 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.506324    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"
Sep 15 21:24:00.508487 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.508458    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-318c22\""
Sep 15 21:24:00.508750 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.508730    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-318c22-dockercfg-hkqkf\""
Sep 15 21:24:00.516250 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.516225    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"]
Sep 15 21:24:00.651424 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.651290    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ns9x\" (UniqueName: \"kubernetes.io/projected/70dd81f7-e79e-4f09-aba4-b92d32d6a31b-kube-api-access-8ns9x\") pod \"payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h\" (UID: \"70dd81f7-e79e-4f09-aba4-b92d32d6a31b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"
Sep 15 21:24:00.651424 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.651402    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70dd81f7-e79e-4f09-aba4-b92d32d6a31b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h\" (UID: \"70dd81f7-e79e-4f09-aba4-b92d32d6a31b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"
Sep 15 21:24:00.752486 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.752445    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ns9x\" (UniqueName: \"kubernetes.io/projected/70dd81f7-e79e-4f09-aba4-b92d32d6a31b-kube-api-access-8ns9x\") pod \"payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h\" (UID: \"70dd81f7-e79e-4f09-aba4-b92d32d6a31b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"
Sep 15 21:24:00.752655 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.752503    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70dd81f7-e79e-4f09-aba4-b92d32d6a31b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h\" (UID: \"70dd81f7-e79e-4f09-aba4-b92d32d6a31b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"
Sep 15 21:24:00.753182 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.753163    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70dd81f7-e79e-4f09-aba4-b92d32d6a31b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h\" (UID: \"70dd81f7-e79e-4f09-aba4-b92d32d6a31b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"
Sep 15 21:24:00.761170 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.761141    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ns9x\" (UniqueName: \"kubernetes.io/projected/70dd81f7-e79e-4f09-aba4-b92d32d6a31b-kube-api-access-8ns9x\") pod \"payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h\" (UID: \"70dd81f7-e79e-4f09-aba4-b92d32d6a31b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"
Sep 15 21:24:00.817498 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:00.817465    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"
Sep 15 21:24:01.095410 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:01.095371    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"]
Sep 15 21:24:01.096876 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:24:01.096850    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70dd81f7_e79e_4f09_aba4_b92d32d6a31b.slice/crio-64aabe209623c988c2d10ef97ff135d5e690b8a15354841c05208283e2b48827 WatchSource:0}: Error finding container 64aabe209623c988c2d10ef97ff135d5e690b8a15354841c05208283e2b48827: Status 404 returned error can't find the container with id 64aabe209623c988c2d10ef97ff135d5e690b8a15354841c05208283e2b48827
Sep 15 21:24:01.380965 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:01.380868    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h" event={"ID":"70dd81f7-e79e-4f09-aba4-b92d32d6a31b","Type":"ContainerStarted","Data":"b17940b5c1b3963cefed2f5db165bcc39dbf3e1583c892455a4f683215fc7eed"}
Sep 15 21:24:01.380965 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:01.380923    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"
Sep 15 21:24:01.380965 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:01.380941    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h" event={"ID":"70dd81f7-e79e-4f09-aba4-b92d32d6a31b","Type":"ContainerStarted","Data":"64aabe209623c988c2d10ef97ff135d5e690b8a15354841c05208283e2b48827"}
Sep 15 21:24:02.411729 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.411652    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h" podStartSLOduration=2.411634045 podStartE2EDuration="2.411634045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:01.405333374 +0000 UTC m=+809.267267180" watchObservedRunningTime="2026-09-15 21:24:02.411634045 +0000 UTC m=+810.273568013"
Sep 15 21:24:02.412834 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.412808    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"]
Sep 15 21:24:02.416618 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.416594    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.418648 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.418630    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d0431a5e-openshift-default-dockercfg-m5p86\""
Sep 15 21:24:02.428884 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.428859    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"]
Sep 15 21:24:02.573315 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.573283    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-workload-certs\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.573526 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.573331    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-credential-socket\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.573526 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.573423    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istiod-ca-cert\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.573526 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.573490    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-workload-socket\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.573697 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.573546    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-envoy\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.573697 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.573631    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-token\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.573697 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.573664    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-podinfo\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.573697 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.573691    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shtfx\" (UniqueName: \"kubernetes.io/projected/c175e8fa-868d-49dc-bdce-31e5bcfcf112-kube-api-access-shtfx\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.573892 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.573795    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-data\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.678302 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.678212    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-credential-socket\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.678302 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.678278    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istiod-ca-cert\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.678563 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.678319    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-workload-socket\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.678563 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.678374    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-envoy\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.678563 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.678458    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-token\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.678563 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.678498    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-podinfo\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.678563 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.678540    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtfx\" (UniqueName: \"kubernetes.io/projected/c175e8fa-868d-49dc-bdce-31e5bcfcf112-kube-api-access-shtfx\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.678868 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.678606    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-data\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.678868 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.678645    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-workload-certs\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.679119 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.679097    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-workload-certs\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.679449 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.679427    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-credential-socket\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.679624 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.679582    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-workload-socket\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.680151 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.680082    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istiod-ca-cert\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.680243 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.680159    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-data\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.682003 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.681982    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-envoy\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.682524 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.682503    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-podinfo\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.688445 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.688425    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-token\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.688527 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.688425    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shtfx\" (UniqueName: \"kubernetes.io/projected/c175e8fa-868d-49dc-bdce-31e5bcfcf112-kube-api-access-shtfx\") pod \"e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") " pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:02.728865 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:02.728824    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:03.017165 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:03.017142    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"]
Sep 15 21:24:03.018466 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:24:03.018437    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc175e8fa_868d_49dc_bdce_31e5bcfcf112.slice/crio-d1cf5092220ef5161df9f8610e0e5e0f85a5b75fefcbe52d28e671df9aae34c5 WatchSource:0}: Error finding container d1cf5092220ef5161df9f8610e0e5e0f85a5b75fefcbe52d28e671df9aae34c5: Status 404 returned error can't find the container with id d1cf5092220ef5161df9f8610e0e5e0f85a5b75fefcbe52d28e671df9aae34c5
Sep 15 21:24:03.020812 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:03.020778    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:24:03.020901 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:03.020844    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:24:03.020901 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:03.020873    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:24:03.396327 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:03.396289    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg" event={"ID":"c175e8fa-868d-49dc-bdce-31e5bcfcf112","Type":"ContainerStarted","Data":"f28f355fa30efa20ef3501e6e4de58e5265a43258fe6e44b6dcff43dfdd48937"}
Sep 15 21:24:03.396548 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:03.396352    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg" event={"ID":"c175e8fa-868d-49dc-bdce-31e5bcfcf112","Type":"ContainerStarted","Data":"d1cf5092220ef5161df9f8610e0e5e0f85a5b75fefcbe52d28e671df9aae34c5"}
Sep 15 21:24:03.415011 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:03.414968    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg" podStartSLOduration=1.414947631 podStartE2EDuration="1.414947631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:03.414186154 +0000 UTC m=+811.276119959" watchObservedRunningTime="2026-09-15 21:24:03.414947631 +0000 UTC m=+811.276881436"
Sep 15 21:24:03.729733 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:03.729663    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:04.734444 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:04.734416    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:05.407307 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:05.407279    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:05.408506 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:05.408482    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:09.372594 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:09.372560    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"
Sep 15 21:24:10.326712 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:10.323435    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"]
Sep 15 21:24:10.331116 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:10.329559    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"]
Sep 15 21:24:10.331116 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:10.329681    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"
Sep 15 21:24:10.336364 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:10.336338    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-04c28d\""
Sep 15 21:24:10.338712 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:10.338011    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-04c28d-dockercfg-fwrtn\""
Sep 15 21:24:10.429963 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:10.429925    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b89d2245-e565-4f55-b63a-93d50db3843b-plugins-config-volume\") pod \"payload-processing-e2e-autores-04c28d-585f648b55-kdh8d\" (UID: \"b89d2245-e565-4f55-b63a-93d50db3843b\") " pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"
Sep 15 21:24:10.430627 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:10.430599    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p46k\" (UniqueName: \"kubernetes.io/projected/b89d2245-e565-4f55-b63a-93d50db3843b-kube-api-access-7p46k\") pod \"payload-processing-e2e-autores-04c28d-585f648b55-kdh8d\" (UID: \"b89d2245-e565-4f55-b63a-93d50db3843b\") " pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"
Sep 15 21:24:10.531997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:10.531950    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p46k\" (UniqueName: \"kubernetes.io/projected/b89d2245-e565-4f55-b63a-93d50db3843b-kube-api-access-7p46k\") pod \"payload-processing-e2e-autores-04c28d-585f648b55-kdh8d\" (UID: \"b89d2245-e565-4f55-b63a-93d50db3843b\") " pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"
Sep 15 21:24:10.532355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:10.532330    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b89d2245-e565-4f55-b63a-93d50db3843b-plugins-config-volume\") pod \"payload-processing-e2e-autores-04c28d-585f648b55-kdh8d\" (UID: \"b89d2245-e565-4f55-b63a-93d50db3843b\") " pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"
Sep 15 21:24:10.533510 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:10.533489    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b89d2245-e565-4f55-b63a-93d50db3843b-plugins-config-volume\") pod \"payload-processing-e2e-autores-04c28d-585f648b55-kdh8d\" (UID: \"b89d2245-e565-4f55-b63a-93d50db3843b\") " pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"
Sep 15 21:24:10.543099 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:10.543078    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p46k\" (UniqueName: \"kubernetes.io/projected/b89d2245-e565-4f55-b63a-93d50db3843b-kube-api-access-7p46k\") pod \"payload-processing-e2e-autores-04c28d-585f648b55-kdh8d\" (UID: \"b89d2245-e565-4f55-b63a-93d50db3843b\") " pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"
Sep 15 21:24:10.726241 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:10.726160    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"
Sep 15 21:24:11.227225 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:11.227201    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"]
Sep 15 21:24:11.230179 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:24:11.230153    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89d2245_e565_4f55_b63a_93d50db3843b.slice/crio-76b369cd4d95a056963a7788b1b3f2255061d6f2c6b4692c94b04bf8b96f8d9a WatchSource:0}: Error finding container 76b369cd4d95a056963a7788b1b3f2255061d6f2c6b4692c94b04bf8b96f8d9a: Status 404 returned error can't find the container with id 76b369cd4d95a056963a7788b1b3f2255061d6f2c6b4692c94b04bf8b96f8d9a
Sep 15 21:24:11.383006 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:11.382972    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"
Sep 15 21:24:11.383205 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:11.383066    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"
Sep 15 21:24:11.496934 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:11.495932    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d" event={"ID":"b89d2245-e565-4f55-b63a-93d50db3843b","Type":"ContainerStarted","Data":"28ec9c29b1edb3e68f93978a99a4b7c66d16e302cb122260e0d42ce62cd5d3e9"}
Sep 15 21:24:11.496934 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:11.496004    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d" event={"ID":"b89d2245-e565-4f55-b63a-93d50db3843b","Type":"ContainerStarted","Data":"76b369cd4d95a056963a7788b1b3f2255061d6f2c6b4692c94b04bf8b96f8d9a"}
Sep 15 21:24:11.496934 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:11.496821    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"
Sep 15 21:24:11.517702 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:11.517649    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d" podStartSLOduration=1.517624891 podStartE2EDuration="1.517624891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:11.515286239 +0000 UTC m=+819.377220042" watchObservedRunningTime="2026-09-15 21:24:11.517624891 +0000 UTC m=+819.379558696"
Sep 15 21:24:12.389129 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:12.389097    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"
Sep 15 21:24:15.525639 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:15.525602    2584 generic.go:356] "Generic (PLEG): container finished" podID="0ab13af8-a313-43a0-8447-3b16b24ee632" containerID="a49805afa90a74c3c8debd5f3e6e865128bf37232fbaa40587055bc0d8d6a74a" exitCode=0
Sep 15 21:24:15.526020 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:15.525673    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj" event={"ID":"0ab13af8-a313-43a0-8447-3b16b24ee632","Type":"ContainerDied","Data":"a49805afa90a74c3c8debd5f3e6e865128bf37232fbaa40587055bc0d8d6a74a"}
Sep 15 21:24:15.526088 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:15.526074    2584 scope.go:117] "RemoveContainer" containerID="a49805afa90a74c3c8debd5f3e6e865128bf37232fbaa40587055bc0d8d6a74a"
Sep 15 21:24:15.527174 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:15.527147    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:24:15.527272 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:15.527211    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:24:15.527272 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:15.527242    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:24:16.535505 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:16.535475    2584 generic.go:356] "Generic (PLEG): container finished" podID="0425206a-e3e3-4644-9066-69c1fa315c2f" containerID="bdd22441729cc8e014465ba0a490f61ef090a1311e987a351872750031f68ddd" exitCode=0
Sep 15 21:24:16.535961 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:16.535550    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs" event={"ID":"0425206a-e3e3-4644-9066-69c1fa315c2f","Type":"ContainerDied","Data":"bdd22441729cc8e014465ba0a490f61ef090a1311e987a351872750031f68ddd"}
Sep 15 21:24:16.536019 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:16.535989    2584 scope.go:117] "RemoveContainer" containerID="bdd22441729cc8e014465ba0a490f61ef090a1311e987a351872750031f68ddd"
Sep 15 21:24:16.537291 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:16.537268    2584 generic.go:356] "Generic (PLEG): container finished" podID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerID="19e481f6870af45588fae621c66bf5cdd14149fdd6916b56e9c770fee5da86d2" exitCode=0
Sep 15 21:24:16.537428 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:16.537302    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:24:16.537428 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:16.537337    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt" event={"ID":"efc6a2e8-4101-46b5-ab18-125c552cd3a1","Type":"ContainerDied","Data":"19e481f6870af45588fae621c66bf5cdd14149fdd6916b56e9c770fee5da86d2"}
Sep 15 21:24:16.537428 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:16.537363    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:24:16.537579 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:16.537428    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:24:16.537810 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:16.537790    2584 scope.go:117] "RemoveContainer" containerID="19e481f6870af45588fae621c66bf5cdd14149fdd6916b56e9c770fee5da86d2"
Sep 15 21:24:16.538952 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:16.538913    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:24:16.539065 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:16.538996    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:24:16.539065 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:16.539048    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:24:16.539755 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:16.539736    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj" event={"ID":"0ab13af8-a313-43a0-8447-3b16b24ee632","Type":"ContainerStarted","Data":"d05515952f5f9c2c5c0cf9a56460f43d70dfae79bf75de6a22585ff16eb0477b"}
Sep 15 21:24:16.774813 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:16.774777    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:24:16.774997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:16.774903    2584 patch_prober.go:28] interesting pod/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 15 21:24:16.774997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:16.774945    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj" podUID="0ab13af8-a313-43a0-8447-3b16b24ee632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 15 21:24:17.547979 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:17.547940    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs" event={"ID":"0425206a-e3e3-4644-9066-69c1fa315c2f","Type":"ContainerStarted","Data":"31f9d7254b9c2ff9c74d38cf115e55aaacd5f88d333a21dab36f98435a3ce95f"}
Sep 15 21:24:17.554660 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:17.554625    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt" event={"ID":"efc6a2e8-4101-46b5-ab18-125c552cd3a1","Type":"ContainerStarted","Data":"2e46d419f70b5c2e5584e4843690921117c4d4d40fc45f233da08ba868a56c22"}
Sep 15 21:24:17.776009 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:17.775965    2584 patch_prober.go:28] interesting pod/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 15 21:24:17.776287 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:17.776260    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj" podUID="0ab13af8-a313-43a0-8447-3b16b24ee632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 15 21:24:17.865410 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:17.865296    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:24:17.865716 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:17.865352    2584 patch_prober.go:28] interesting pod/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 15 21:24:17.865808 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:17.865763    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt" podUID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 15 21:24:17.948311 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:17.948277    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:24:17.948716 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:17.948687    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 15 21:24:17.948838 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:17.948757    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs" podUID="0425206a-e3e3-4644-9066-69c1fa315c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 15 21:24:18.775064 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:18.775020    2584 patch_prober.go:28] interesting pod/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 15 21:24:18.775617 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:18.775131    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj" podUID="0ab13af8-a313-43a0-8447-3b16b24ee632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 15 21:24:18.865882 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:18.865397    2584 patch_prober.go:28] interesting pod/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 15 21:24:18.865882 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:18.865460    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt" podUID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 15 21:24:18.949426 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:18.949392    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 15 21:24:18.949720 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:18.949681    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs" podUID="0425206a-e3e3-4644-9066-69c1fa315c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 15 21:24:19.776404 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:19.774757    2584 patch_prober.go:28] interesting pod/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 15 21:24:19.776404 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:19.774820    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj" podUID="0ab13af8-a313-43a0-8447-3b16b24ee632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 15 21:24:19.870472 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:19.870439    2584 patch_prober.go:28] interesting pod/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 15 21:24:19.870737 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:19.870712    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt" podUID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 15 21:24:19.951863 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:19.951830    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 15 21:24:19.952158 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:19.952130    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs" podUID="0425206a-e3e3-4644-9066-69c1fa315c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 15 21:24:20.774959 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:20.774920    2584 patch_prober.go:28] interesting pod/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 15 21:24:20.775320 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:20.775289    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj" podUID="0ab13af8-a313-43a0-8447-3b16b24ee632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 15 21:24:20.865254 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:20.865219    2584 patch_prober.go:28] interesting pod/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 15 21:24:20.865862 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:20.865813    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt" podUID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 15 21:24:20.948992 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:20.948952    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 15 21:24:20.949304 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:20.949269    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs" podUID="0425206a-e3e3-4644-9066-69c1fa315c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 15 21:24:21.774717 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:21.774670    2584 patch_prober.go:28] interesting pod/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 15 21:24:21.774931 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:21.774741    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj" podUID="0ab13af8-a313-43a0-8447-3b16b24ee632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 15 21:24:21.865243 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:21.865200    2584 patch_prober.go:28] interesting pod/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 15 21:24:21.865622 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:21.865586    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt" podUID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 15 21:24:21.959541 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:21.959502    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 15 21:24:21.959825 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:21.959792    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs" podUID="0425206a-e3e3-4644-9066-69c1fa315c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 15 21:24:22.775400 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:22.774812    2584 patch_prober.go:28] interesting pod/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 15 21:24:22.775400 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:22.774875    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj" podUID="0ab13af8-a313-43a0-8447-3b16b24ee632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 15 21:24:22.864937 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:22.864902    2584 patch_prober.go:28] interesting pod/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 15 21:24:22.865409 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:22.865192    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt" podUID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 15 21:24:23.001403 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:22.998022    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 15 21:24:23.024532 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:23.024491    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs" podUID="0425206a-e3e3-4644-9066-69c1fa315c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 15 21:24:23.513485 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:23.513447    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"
Sep 15 21:24:23.803602 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:23.803569    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:24:23.805058 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:23.805039    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:24:23.813034 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:23.813011    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:24:23.865208 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:23.865172    2584 patch_prober.go:28] interesting pod/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 15 21:24:23.865415 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:23.865260    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt" podUID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 15 21:24:23.953010 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:23.952977    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 15 21:24:23.953739 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:23.953366    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs" podUID="0425206a-e3e3-4644-9066-69c1fa315c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 15 21:24:24.874657 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:24.874190    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:24:24.875192 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:24.875133    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:24:24.876121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:24.876102    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:24:24.965046 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:24.963656    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:24:24.965046 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:24.964910    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:24:24.968836 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:24.967886    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:24:25.267115 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:25.267030    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s"]
Sep 15 21:24:25.274855 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:25.274827    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s"
Sep 15 21:24:25.275735 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:25.275708    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s"]
Sep 15 21:24:25.277782 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:25.277758    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:24:25.277947 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:25.277930    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:24:25.326079 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:25.326052    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eefc969-5692-47f8-b07a-b7534fb270da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s\" (UID: \"8eefc969-5692-47f8-b07a-b7534fb270da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s"
Sep 15 21:24:25.427189 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:25.427156    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eefc969-5692-47f8-b07a-b7534fb270da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s\" (UID: \"8eefc969-5692-47f8-b07a-b7534fb270da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s"
Sep 15 21:24:25.427837 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:25.427814    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eefc969-5692-47f8-b07a-b7534fb270da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s\" (UID: \"8eefc969-5692-47f8-b07a-b7534fb270da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s"
Sep 15 21:24:25.590980 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:25.590945    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s"
Sep 15 21:24:25.863264 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:25.863166    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s"]
Sep 15 21:24:26.642407 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:26.642356    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s" event={"ID":"8eefc969-5692-47f8-b07a-b7534fb270da","Type":"ContainerStarted","Data":"71dacf046d792c03a6483943b9feaab0274da773c5162ae6f5593938b86dc451"}
Sep 15 21:24:29.664059 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:29.664023    2584 generic.go:356] "Generic (PLEG): container finished" podID="8eefc969-5692-47f8-b07a-b7534fb270da" containerID="db6f6f2691b0189eb4663995b96acf45fe8c70514dc5556fb76a9410be405323" exitCode=0
Sep 15 21:24:29.664456 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:29.664097    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s" event={"ID":"8eefc969-5692-47f8-b07a-b7534fb270da","Type":"ContainerDied","Data":"db6f6f2691b0189eb4663995b96acf45fe8c70514dc5556fb76a9410be405323"}
Sep 15 21:24:30.672410 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:30.672363    2584 generic.go:356] "Generic (PLEG): container finished" podID="c175e8fa-868d-49dc-bdce-31e5bcfcf112" containerID="f28f355fa30efa20ef3501e6e4de58e5265a43258fe6e44b6dcff43dfdd48937" exitCode=0
Sep 15 21:24:30.673869 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:30.673846    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg" event={"ID":"c175e8fa-868d-49dc-bdce-31e5bcfcf112","Type":"ContainerDied","Data":"f28f355fa30efa20ef3501e6e4de58e5265a43258fe6e44b6dcff43dfdd48937"}
Sep 15 21:24:30.674165 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:30.674150    2584 scope.go:117] "RemoveContainer" containerID="f28f355fa30efa20ef3501e6e4de58e5265a43258fe6e44b6dcff43dfdd48937"
Sep 15 21:24:30.675391 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:30.675354    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:24:30.675462 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:30.675442    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:24:30.675498 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:30.675490    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:24:30.854720 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:30.854486    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s"
Sep 15 21:24:30.894532 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:30.894498    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eefc969-5692-47f8-b07a-b7534fb270da-maas-api-service-ca\") pod \"8eefc969-5692-47f8-b07a-b7534fb270da\" (UID: \"8eefc969-5692-47f8-b07a-b7534fb270da\") "
Sep 15 21:24:30.894950 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:30.894923    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eefc969-5692-47f8-b07a-b7534fb270da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8eefc969-5692-47f8-b07a-b7534fb270da" (UID: "8eefc969-5692-47f8-b07a-b7534fb270da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:24:30.996196 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:30.996117    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eefc969-5692-47f8-b07a-b7534fb270da-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:24:31.683008 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:31.682978    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s"
Sep 15 21:24:31.683497 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:31.682998    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s" event={"ID":"8eefc969-5692-47f8-b07a-b7534fb270da","Type":"ContainerDied","Data":"71dacf046d792c03a6483943b9feaab0274da773c5162ae6f5593938b86dc451"}
Sep 15 21:24:31.683497 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:31.683238    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71dacf046d792c03a6483943b9feaab0274da773c5162ae6f5593938b86dc451"
Sep 15 21:24:31.685251 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:31.685226    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg" event={"ID":"c175e8fa-868d-49dc-bdce-31e5bcfcf112","Type":"ContainerStarted","Data":"78e856ac60000df2a5432e01fba9dd6b27c87b236448e67c236390c46be068bd"}
Sep 15 21:24:31.729472 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:31.729451    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:31.730509 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:31.730488    2584 patch_prober.go:28] interesting pod/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 15 21:24:31.730583 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:31.730531    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg" podUID="c175e8fa-868d-49dc-bdce-31e5bcfcf112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 15 21:24:32.729873 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:32.729846    2584 patch_prober.go:28] interesting pod/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 15 21:24:32.730194 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:32.729892    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg" podUID="c175e8fa-868d-49dc-bdce-31e5bcfcf112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 15 21:24:33.729347 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:33.729314    2584 patch_prober.go:28] interesting pod/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 15 21:24:33.729540 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:33.729373    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg" podUID="c175e8fa-868d-49dc-bdce-31e5bcfcf112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 15 21:24:34.729609 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:34.729581    2584 patch_prober.go:28] interesting pod/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 15 21:24:34.729986 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:34.729638    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg" podUID="c175e8fa-868d-49dc-bdce-31e5bcfcf112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 15 21:24:35.408131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:35.408092    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:35.729883 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:35.729679    2584 patch_prober.go:28] interesting pod/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 15 21:24:35.729883 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:35.729730    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg" podUID="c175e8fa-868d-49dc-bdce-31e5bcfcf112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 15 21:24:36.711971 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:36.711932    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s"]
Sep 15 21:24:36.715054 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:36.715027    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0431a5e-rxx7s"]
Sep 15 21:24:36.729787 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:36.729756    2584 patch_prober.go:28] interesting pod/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 15 21:24:36.729896 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:36.729808    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg" podUID="c175e8fa-868d-49dc-bdce-31e5bcfcf112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 15 21:24:36.834580 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:36.834546    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"]
Sep 15 21:24:36.834792 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:36.834767    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg" podUID="c175e8fa-868d-49dc-bdce-31e5bcfcf112" containerName="istio-proxy" containerID="cri-o://78e856ac60000df2a5432e01fba9dd6b27c87b236448e67c236390c46be068bd" gracePeriod=30
Sep 15 21:24:38.673408 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:38.673362    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eefc969-5692-47f8-b07a-b7534fb270da" path="/var/lib/kubelet/pods/8eefc969-5692-47f8-b07a-b7534fb270da/volumes"
Sep 15 21:24:42.086977 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.086954    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:42.229279 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.229191    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-envoy\") pod \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") "
Sep 15 21:24:42.229279 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.229246    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shtfx\" (UniqueName: \"kubernetes.io/projected/c175e8fa-868d-49dc-bdce-31e5bcfcf112-kube-api-access-shtfx\") pod \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") "
Sep 15 21:24:42.229279 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.229276    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-data\") pod \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") "
Sep 15 21:24:42.229602 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.229320    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-credential-socket\") pod \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") "
Sep 15 21:24:42.229602 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.229349    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istiod-ca-cert\") pod \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") "
Sep 15 21:24:42.229602 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.229402    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-workload-certs\") pod \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") "
Sep 15 21:24:42.229602 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.229438    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-podinfo\") pod \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") "
Sep 15 21:24:42.229602 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.229531    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-token\") pod \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") "
Sep 15 21:24:42.229602 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.229587    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-workload-socket\") pod \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\" (UID: \"c175e8fa-868d-49dc-bdce-31e5bcfcf112\") "
Sep 15 21:24:42.229602 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.229597    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-data" (OuterVolumeSpecName: "istio-data") pod "c175e8fa-868d-49dc-bdce-31e5bcfcf112" (UID: "c175e8fa-868d-49dc-bdce-31e5bcfcf112"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:24:42.229977 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.229683    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c175e8fa-868d-49dc-bdce-31e5bcfcf112" (UID: "c175e8fa-868d-49dc-bdce-31e5bcfcf112"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:24:42.229977 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.229788    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c175e8fa-868d-49dc-bdce-31e5bcfcf112" (UID: "c175e8fa-868d-49dc-bdce-31e5bcfcf112"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:24:42.229977 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.229918    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c175e8fa-868d-49dc-bdce-31e5bcfcf112" (UID: "c175e8fa-868d-49dc-bdce-31e5bcfcf112"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:24:42.229977 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.229941    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c175e8fa-868d-49dc-bdce-31e5bcfcf112" (UID: "c175e8fa-868d-49dc-bdce-31e5bcfcf112"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:24:42.230189 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.230069    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istiod-ca-cert\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:24:42.230189 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.230089    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-workload-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:24:42.230189 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.230105    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-workload-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:24:42.230189 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.230118    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-data\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:24:42.230189 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.230130    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-credential-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:24:42.231657 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.231633    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c175e8fa-868d-49dc-bdce-31e5bcfcf112-kube-api-access-shtfx" (OuterVolumeSpecName: "kube-api-access-shtfx") pod "c175e8fa-868d-49dc-bdce-31e5bcfcf112" (UID: "c175e8fa-868d-49dc-bdce-31e5bcfcf112"). InnerVolumeSpecName "kube-api-access-shtfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:24:42.231657 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.231642    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-token" (OuterVolumeSpecName: "istio-token") pod "c175e8fa-868d-49dc-bdce-31e5bcfcf112" (UID: "c175e8fa-868d-49dc-bdce-31e5bcfcf112"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:24:42.231886 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.231863    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c175e8fa-868d-49dc-bdce-31e5bcfcf112" (UID: "c175e8fa-868d-49dc-bdce-31e5bcfcf112"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:24:42.232074 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.232050    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c175e8fa-868d-49dc-bdce-31e5bcfcf112" (UID: "c175e8fa-868d-49dc-bdce-31e5bcfcf112"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:24:42.331099 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.331067    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-envoy\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:24:42.331099 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.331094    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shtfx\" (UniqueName: \"kubernetes.io/projected/c175e8fa-868d-49dc-bdce-31e5bcfcf112-kube-api-access-shtfx\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:24:42.331099 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.331106    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-podinfo\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:24:42.331322 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.331115    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c175e8fa-868d-49dc-bdce-31e5bcfcf112-istio-token\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:24:42.753655 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.753624    2584 generic.go:356] "Generic (PLEG): container finished" podID="c175e8fa-868d-49dc-bdce-31e5bcfcf112" containerID="78e856ac60000df2a5432e01fba9dd6b27c87b236448e67c236390c46be068bd" exitCode=0
Sep 15 21:24:42.753839 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.753682    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"
Sep 15 21:24:42.753839 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.753709    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg" event={"ID":"c175e8fa-868d-49dc-bdce-31e5bcfcf112","Type":"ContainerDied","Data":"78e856ac60000df2a5432e01fba9dd6b27c87b236448e67c236390c46be068bd"}
Sep 15 21:24:42.753839 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.753776    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg" event={"ID":"c175e8fa-868d-49dc-bdce-31e5bcfcf112","Type":"ContainerDied","Data":"d1cf5092220ef5161df9f8610e0e5e0f85a5b75fefcbe52d28e671df9aae34c5"}
Sep 15 21:24:42.753839 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.753808    2584 scope.go:117] "RemoveContainer" containerID="78e856ac60000df2a5432e01fba9dd6b27c87b236448e67c236390c46be068bd"
Sep 15 21:24:42.765923 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.765903    2584 scope.go:117] "RemoveContainer" containerID="f28f355fa30efa20ef3501e6e4de58e5265a43258fe6e44b6dcff43dfdd48937"
Sep 15 21:24:42.772810 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.772788    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"]
Sep 15 21:24:42.774480 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.774463    2584 scope.go:117] "RemoveContainer" containerID="78e856ac60000df2a5432e01fba9dd6b27c87b236448e67c236390c46be068bd"
Sep 15 21:24:42.774730 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:24:42.774714    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e856ac60000df2a5432e01fba9dd6b27c87b236448e67c236390c46be068bd\": container with ID starting with 78e856ac60000df2a5432e01fba9dd6b27c87b236448e67c236390c46be068bd not found: ID does not exist" containerID="78e856ac60000df2a5432e01fba9dd6b27c87b236448e67c236390c46be068bd"
Sep 15 21:24:42.774785 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.774739    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e856ac60000df2a5432e01fba9dd6b27c87b236448e67c236390c46be068bd"} err="failed to get container status \"78e856ac60000df2a5432e01fba9dd6b27c87b236448e67c236390c46be068bd\": rpc error: code = NotFound desc = could not find container \"78e856ac60000df2a5432e01fba9dd6b27c87b236448e67c236390c46be068bd\": container with ID starting with 78e856ac60000df2a5432e01fba9dd6b27c87b236448e67c236390c46be068bd not found: ID does not exist"
Sep 15 21:24:42.774785 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.774756    2584 scope.go:117] "RemoveContainer" containerID="f28f355fa30efa20ef3501e6e4de58e5265a43258fe6e44b6dcff43dfdd48937"
Sep 15 21:24:42.775059 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:24:42.775037    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28f355fa30efa20ef3501e6e4de58e5265a43258fe6e44b6dcff43dfdd48937\": container with ID starting with f28f355fa30efa20ef3501e6e4de58e5265a43258fe6e44b6dcff43dfdd48937 not found: ID does not exist" containerID="f28f355fa30efa20ef3501e6e4de58e5265a43258fe6e44b6dcff43dfdd48937"
Sep 15 21:24:42.775229 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.775067    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28f355fa30efa20ef3501e6e4de58e5265a43258fe6e44b6dcff43dfdd48937"} err="failed to get container status \"f28f355fa30efa20ef3501e6e4de58e5265a43258fe6e44b6dcff43dfdd48937\": rpc error: code = NotFound desc = could not find container \"f28f355fa30efa20ef3501e6e4de58e5265a43258fe6e44b6dcff43dfdd48937\": container with ID starting with f28f355fa30efa20ef3501e6e4de58e5265a43258fe6e44b6dcff43dfdd48937 not found: ID does not exist"
Sep 15 21:24:42.778579 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:42.778560    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d0431a5e-openshift-default-68575bc4c4-lz7dg"]
Sep 15 21:24:43.092200 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.092164    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"]
Sep 15 21:24:43.092723 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.092700    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c175e8fa-868d-49dc-bdce-31e5bcfcf112" containerName="istio-proxy"
Sep 15 21:24:43.092723 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.092723    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c175e8fa-868d-49dc-bdce-31e5bcfcf112" containerName="istio-proxy"
Sep 15 21:24:43.092890 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.092797    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eefc969-5692-47f8-b07a-b7534fb270da" containerName="revoke-keys"
Sep 15 21:24:43.092890 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.092809    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eefc969-5692-47f8-b07a-b7534fb270da" containerName="revoke-keys"
Sep 15 21:24:43.093008 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.092946    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c175e8fa-868d-49dc-bdce-31e5bcfcf112" containerName="istio-proxy"
Sep 15 21:24:43.093008 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.092964    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eefc969-5692-47f8-b07a-b7534fb270da" containerName="revoke-keys"
Sep 15 21:24:43.093008 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.092972    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c175e8fa-868d-49dc-bdce-31e5bcfcf112" containerName="istio-proxy"
Sep 15 21:24:43.097823 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.097801    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"
Sep 15 21:24:43.100303 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.100280    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-edbfd1c5\""
Sep 15 21:24:43.104045 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.104014    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"]
Sep 15 21:24:43.114555 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.114531    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"]
Sep 15 21:24:43.115206 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.115189    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c175e8fa-868d-49dc-bdce-31e5bcfcf112" containerName="istio-proxy"
Sep 15 21:24:43.115258 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.115212    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c175e8fa-868d-49dc-bdce-31e5bcfcf112" containerName="istio-proxy"
Sep 15 21:24:43.118737 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.118718    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"
Sep 15 21:24:43.120913 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.120893    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-edbfd1c5-dockercfg-br2zb\""
Sep 15 21:24:43.132028 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.132002    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"]
Sep 15 21:24:43.241544 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.241513    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwt87\" (UniqueName: \"kubernetes.io/projected/b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d-kube-api-access-lwt87\") pod \"payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv\" (UID: \"b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d\") " pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"
Sep 15 21:24:43.241708 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.241572    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213832e7-9059-4e83-9d8e-e47791aea0de-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg\" (UID: \"213832e7-9059-4e83-9d8e-e47791aea0de\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"
Sep 15 21:24:43.241708 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.241641    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv\" (UID: \"b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d\") " pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"
Sep 15 21:24:43.241708 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.241680    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwwz7\" (UniqueName: \"kubernetes.io/projected/213832e7-9059-4e83-9d8e-e47791aea0de-kube-api-access-xwwz7\") pod \"payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg\" (UID: \"213832e7-9059-4e83-9d8e-e47791aea0de\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"
Sep 15 21:24:43.343285 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.343202    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwt87\" (UniqueName: \"kubernetes.io/projected/b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d-kube-api-access-lwt87\") pod \"payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv\" (UID: \"b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d\") " pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"
Sep 15 21:24:43.343285 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.343257    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213832e7-9059-4e83-9d8e-e47791aea0de-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg\" (UID: \"213832e7-9059-4e83-9d8e-e47791aea0de\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"
Sep 15 21:24:43.343572 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.343311    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv\" (UID: \"b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d\") " pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"
Sep 15 21:24:43.343572 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.343349    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwwz7\" (UniqueName: \"kubernetes.io/projected/213832e7-9059-4e83-9d8e-e47791aea0de-kube-api-access-xwwz7\") pod \"payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg\" (UID: \"213832e7-9059-4e83-9d8e-e47791aea0de\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"
Sep 15 21:24:43.344019 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.343992    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213832e7-9059-4e83-9d8e-e47791aea0de-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg\" (UID: \"213832e7-9059-4e83-9d8e-e47791aea0de\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"
Sep 15 21:24:43.344104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.344030    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv\" (UID: \"b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d\") " pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"
Sep 15 21:24:43.353040 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.353008    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwwz7\" (UniqueName: \"kubernetes.io/projected/213832e7-9059-4e83-9d8e-e47791aea0de-kube-api-access-xwwz7\") pod \"payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg\" (UID: \"213832e7-9059-4e83-9d8e-e47791aea0de\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"
Sep 15 21:24:43.353141 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.353015    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwt87\" (UniqueName: \"kubernetes.io/projected/b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d-kube-api-access-lwt87\") pod \"payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv\" (UID: \"b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d\") " pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"
Sep 15 21:24:43.410071 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.410039    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"
Sep 15 21:24:43.428890 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.428862    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"
Sep 15 21:24:43.688411 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.688298    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"]
Sep 15 21:24:43.691234 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:24:43.691210    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213832e7_9059_4e83_9d8e_e47791aea0de.slice/crio-3a80d5febe43b60b07b5d819f975735df2fce3f177a8c1194d7a8b9c7d07297e WatchSource:0}: Error finding container 3a80d5febe43b60b07b5d819f975735df2fce3f177a8c1194d7a8b9c7d07297e: Status 404 returned error can't find the container with id 3a80d5febe43b60b07b5d819f975735df2fce3f177a8c1194d7a8b9c7d07297e
Sep 15 21:24:43.717037 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.717015    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"]
Sep 15 21:24:43.724362 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:24:43.724335    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f6a34b_3ef1_49d2_9fdf_6c2a7e3ee55d.slice/crio-54d8f8a15d970bf8ab361a6a78c873b1ec745a0e282bfb435274845e2fe39516 WatchSource:0}: Error finding container 54d8f8a15d970bf8ab361a6a78c873b1ec745a0e282bfb435274845e2fe39516: Status 404 returned error can't find the container with id 54d8f8a15d970bf8ab361a6a78c873b1ec745a0e282bfb435274845e2fe39516
Sep 15 21:24:43.771663 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.771631    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv" event={"ID":"b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d","Type":"ContainerStarted","Data":"54d8f8a15d970bf8ab361a6a78c873b1ec745a0e282bfb435274845e2fe39516"}
Sep 15 21:24:43.773516 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.773492    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg" event={"ID":"213832e7-9059-4e83-9d8e-e47791aea0de","Type":"ContainerStarted","Data":"8b5dd45ce0a9ff06ac3a553216f3e0219a52a6a984babf8e3381e995ab4b9b55"}
Sep 15 21:24:43.773675 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.773542    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"
Sep 15 21:24:43.773675 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.773561    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg" event={"ID":"213832e7-9059-4e83-9d8e-e47791aea0de","Type":"ContainerStarted","Data":"3a80d5febe43b60b07b5d819f975735df2fce3f177a8c1194d7a8b9c7d07297e"}
Sep 15 21:24:43.791649 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:43.791603    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg" podStartSLOduration=0.791587289 podStartE2EDuration="791.587289ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:43.789924003 +0000 UTC m=+851.651857809" watchObservedRunningTime="2026-09-15 21:24:43.791587289 +0000 UTC m=+851.653521137"
Sep 15 21:24:44.673648 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:44.673612    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c175e8fa-868d-49dc-bdce-31e5bcfcf112" path="/var/lib/kubelet/pods/c175e8fa-868d-49dc-bdce-31e5bcfcf112/volumes"
Sep 15 21:24:44.782641 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:44.782605    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv" event={"ID":"b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d","Type":"ContainerStarted","Data":"53e70c7dfeb2a516eb8d008cb6d67b2994891796f657918273f239f2b2228ce9"}
Sep 15 21:24:44.782785 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:44.782667    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"
Sep 15 21:24:44.800226 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:44.800178    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv" podStartSLOduration=1.800162429 podStartE2EDuration="1.800162429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:44.798233453 +0000 UTC m=+852.660167261" watchObservedRunningTime="2026-09-15 21:24:44.800162429 +0000 UTC m=+852.662096234"
Sep 15 21:24:54.784899 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:54.784871    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"
Sep 15 21:24:55.789693 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:24:55.789659    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"
Sep 15 21:25:05.672324 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:05.672287    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn"]
Sep 15 21:25:05.676204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:05.676185    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn"
Sep 15 21:25:05.678485 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:05.678458    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:25:05.678596 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:05.678458    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:25:05.682748 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:05.682723    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn"]
Sep 15 21:25:05.772467 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:05.772437    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99e65bad-bffc-4380-a18a-786df2319207-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn\" (UID: \"99e65bad-bffc-4380-a18a-786df2319207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn"
Sep 15 21:25:05.873264 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:05.873231    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99e65bad-bffc-4380-a18a-786df2319207-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn\" (UID: \"99e65bad-bffc-4380-a18a-786df2319207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn"
Sep 15 21:25:05.874009 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:05.873987    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99e65bad-bffc-4380-a18a-786df2319207-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn\" (UID: \"99e65bad-bffc-4380-a18a-786df2319207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn"
Sep 15 21:25:05.989730 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:05.989647    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn"
Sep 15 21:25:06.263878 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:06.263848    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn"]
Sep 15 21:25:06.265646 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:25:06.265614    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e65bad_bffc_4380_a18a_786df2319207.slice/crio-11d444968c5684325cca43e1562d10ccd6ab556778be910ff5fc4a8cf1d4ee30 WatchSource:0}: Error finding container 11d444968c5684325cca43e1562d10ccd6ab556778be910ff5fc4a8cf1d4ee30: Status 404 returned error can't find the container with id 11d444968c5684325cca43e1562d10ccd6ab556778be910ff5fc4a8cf1d4ee30
Sep 15 21:25:06.925057 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:06.925024    2584 generic.go:356] "Generic (PLEG): container finished" podID="99e65bad-bffc-4380-a18a-786df2319207" containerID="4c6cdf78a5cd27450936093e777f45cd4768ecfdc206104b9d58122ac86118d7" exitCode=0
Sep 15 21:25:06.925474 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:06.925074    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn" event={"ID":"99e65bad-bffc-4380-a18a-786df2319207","Type":"ContainerDied","Data":"4c6cdf78a5cd27450936093e777f45cd4768ecfdc206104b9d58122ac86118d7"}
Sep 15 21:25:06.925474 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:06.925128    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn" event={"ID":"99e65bad-bffc-4380-a18a-786df2319207","Type":"ContainerStarted","Data":"11d444968c5684325cca43e1562d10ccd6ab556778be910ff5fc4a8cf1d4ee30"}
Sep 15 21:25:08.085172 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:08.085149    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn"
Sep 15 21:25:08.197734 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:08.197637    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99e65bad-bffc-4380-a18a-786df2319207-maas-api-service-ca\") pod \"99e65bad-bffc-4380-a18a-786df2319207\" (UID: \"99e65bad-bffc-4380-a18a-786df2319207\") "
Sep 15 21:25:08.198017 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:08.197993    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e65bad-bffc-4380-a18a-786df2319207-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99e65bad-bffc-4380-a18a-786df2319207" (UID: "99e65bad-bffc-4380-a18a-786df2319207"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:25:08.299251 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:08.299212    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99e65bad-bffc-4380-a18a-786df2319207-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:08.937802 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:08.937725    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn"
Sep 15 21:25:08.937802 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:08.937741    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn" event={"ID":"99e65bad-bffc-4380-a18a-786df2319207","Type":"ContainerDied","Data":"11d444968c5684325cca43e1562d10ccd6ab556778be910ff5fc4a8cf1d4ee30"}
Sep 15 21:25:08.937802 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:08.937783    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11d444968c5684325cca43e1562d10ccd6ab556778be910ff5fc4a8cf1d4ee30"
Sep 15 21:25:12.629613 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.629582    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"]
Sep 15 21:25:12.630068 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.629865    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv" podUID="b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d" containerName="payload-processing" containerID="cri-o://53e70c7dfeb2a516eb8d008cb6d67b2994891796f657918273f239f2b2228ce9" gracePeriod=30
Sep 15 21:25:12.637210 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.637180    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"]
Sep 15 21:25:12.637502 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.637454    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg" podUID="213832e7-9059-4e83-9d8e-e47791aea0de" containerName="payload-pre-processing" containerID="cri-o://8b5dd45ce0a9ff06ac3a553216f3e0219a52a6a984babf8e3381e995ab4b9b55" gracePeriod=30
Sep 15 21:25:12.811758 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.811676    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn"]
Sep 15 21:25:12.814931 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.814900    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-edbfd1c5-4s7kn"]
Sep 15 21:25:12.915317 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.915295    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"
Sep 15 21:25:12.944523 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.944500    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"
Sep 15 21:25:12.969843 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.969817    2584 generic.go:356] "Generic (PLEG): container finished" podID="213832e7-9059-4e83-9d8e-e47791aea0de" containerID="8b5dd45ce0a9ff06ac3a553216f3e0219a52a6a984babf8e3381e995ab4b9b55" exitCode=0
Sep 15 21:25:12.969997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.969859    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"
Sep 15 21:25:12.969997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.969903    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg" event={"ID":"213832e7-9059-4e83-9d8e-e47791aea0de","Type":"ContainerDied","Data":"8b5dd45ce0a9ff06ac3a553216f3e0219a52a6a984babf8e3381e995ab4b9b55"}
Sep 15 21:25:12.969997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.969950    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg" event={"ID":"213832e7-9059-4e83-9d8e-e47791aea0de","Type":"ContainerDied","Data":"3a80d5febe43b60b07b5d819f975735df2fce3f177a8c1194d7a8b9c7d07297e"}
Sep 15 21:25:12.969997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.969981    2584 scope.go:117] "RemoveContainer" containerID="8b5dd45ce0a9ff06ac3a553216f3e0219a52a6a984babf8e3381e995ab4b9b55"
Sep 15 21:25:12.971509 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.971483    2584 generic.go:356] "Generic (PLEG): container finished" podID="b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d" containerID="53e70c7dfeb2a516eb8d008cb6d67b2994891796f657918273f239f2b2228ce9" exitCode=0
Sep 15 21:25:12.971630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.971552    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv" event={"ID":"b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d","Type":"ContainerDied","Data":"53e70c7dfeb2a516eb8d008cb6d67b2994891796f657918273f239f2b2228ce9"}
Sep 15 21:25:12.971630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.971587    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv" event={"ID":"b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d","Type":"ContainerDied","Data":"54d8f8a15d970bf8ab361a6a78c873b1ec745a0e282bfb435274845e2fe39516"}
Sep 15 21:25:12.971630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.971555    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"
Sep 15 21:25:12.979391 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.979357    2584 scope.go:117] "RemoveContainer" containerID="8b5dd45ce0a9ff06ac3a553216f3e0219a52a6a984babf8e3381e995ab4b9b55"
Sep 15 21:25:12.979708 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:25:12.979681    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5dd45ce0a9ff06ac3a553216f3e0219a52a6a984babf8e3381e995ab4b9b55\": container with ID starting with 8b5dd45ce0a9ff06ac3a553216f3e0219a52a6a984babf8e3381e995ab4b9b55 not found: ID does not exist" containerID="8b5dd45ce0a9ff06ac3a553216f3e0219a52a6a984babf8e3381e995ab4b9b55"
Sep 15 21:25:12.979791 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.979718    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5dd45ce0a9ff06ac3a553216f3e0219a52a6a984babf8e3381e995ab4b9b55"} err="failed to get container status \"8b5dd45ce0a9ff06ac3a553216f3e0219a52a6a984babf8e3381e995ab4b9b55\": rpc error: code = NotFound desc = could not find container \"8b5dd45ce0a9ff06ac3a553216f3e0219a52a6a984babf8e3381e995ab4b9b55\": container with ID starting with 8b5dd45ce0a9ff06ac3a553216f3e0219a52a6a984babf8e3381e995ab4b9b55 not found: ID does not exist"
Sep 15 21:25:12.979791 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.979742    2584 scope.go:117] "RemoveContainer" containerID="53e70c7dfeb2a516eb8d008cb6d67b2994891796f657918273f239f2b2228ce9"
Sep 15 21:25:12.988316 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.988292    2584 scope.go:117] "RemoveContainer" containerID="53e70c7dfeb2a516eb8d008cb6d67b2994891796f657918273f239f2b2228ce9"
Sep 15 21:25:12.988607 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:25:12.988580    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e70c7dfeb2a516eb8d008cb6d67b2994891796f657918273f239f2b2228ce9\": container with ID starting with 53e70c7dfeb2a516eb8d008cb6d67b2994891796f657918273f239f2b2228ce9 not found: ID does not exist" containerID="53e70c7dfeb2a516eb8d008cb6d67b2994891796f657918273f239f2b2228ce9"
Sep 15 21:25:12.988668 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:12.988614    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e70c7dfeb2a516eb8d008cb6d67b2994891796f657918273f239f2b2228ce9"} err="failed to get container status \"53e70c7dfeb2a516eb8d008cb6d67b2994891796f657918273f239f2b2228ce9\": rpc error: code = NotFound desc = could not find container \"53e70c7dfeb2a516eb8d008cb6d67b2994891796f657918273f239f2b2228ce9\": container with ID starting with 53e70c7dfeb2a516eb8d008cb6d67b2994891796f657918273f239f2b2228ce9 not found: ID does not exist"
Sep 15 21:25:13.048327 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.048288    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwt87\" (UniqueName: \"kubernetes.io/projected/b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d-kube-api-access-lwt87\") pod \"b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d\" (UID: \"b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d\") "
Sep 15 21:25:13.048515 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.048405    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213832e7-9059-4e83-9d8e-e47791aea0de-plugins-config-volume\") pod \"213832e7-9059-4e83-9d8e-e47791aea0de\" (UID: \"213832e7-9059-4e83-9d8e-e47791aea0de\") "
Sep 15 21:25:13.048515 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.048477    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwwz7\" (UniqueName: \"kubernetes.io/projected/213832e7-9059-4e83-9d8e-e47791aea0de-kube-api-access-xwwz7\") pod \"213832e7-9059-4e83-9d8e-e47791aea0de\" (UID: \"213832e7-9059-4e83-9d8e-e47791aea0de\") "
Sep 15 21:25:13.048654 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.048546    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d-plugins-config-volume\") pod \"b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d\" (UID: \"b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d\") "
Sep 15 21:25:13.049084 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.049052    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213832e7-9059-4e83-9d8e-e47791aea0de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "213832e7-9059-4e83-9d8e-e47791aea0de" (UID: "213832e7-9059-4e83-9d8e-e47791aea0de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:25:13.050671 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.050644    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d" (UID: "b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:25:13.051619 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.051595    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d-kube-api-access-lwt87" (OuterVolumeSpecName: "kube-api-access-lwt87") pod "b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d" (UID: "b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d"). InnerVolumeSpecName "kube-api-access-lwt87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:25:13.053590 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.053563    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213832e7-9059-4e83-9d8e-e47791aea0de-kube-api-access-xwwz7" (OuterVolumeSpecName: "kube-api-access-xwwz7") pod "213832e7-9059-4e83-9d8e-e47791aea0de" (UID: "213832e7-9059-4e83-9d8e-e47791aea0de"). InnerVolumeSpecName "kube-api-access-xwwz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:25:13.150288 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.150208    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwt87\" (UniqueName: \"kubernetes.io/projected/b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d-kube-api-access-lwt87\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:13.150288 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.150236    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213832e7-9059-4e83-9d8e-e47791aea0de-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:13.150288 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.150245    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwwz7\" (UniqueName: \"kubernetes.io/projected/213832e7-9059-4e83-9d8e-e47791aea0de-kube-api-access-xwwz7\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:13.150288 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.150254    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:13.296357 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.296322    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"]
Sep 15 21:25:13.299836 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.299812    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-edbfd1c5-996cf8b98-qvjcv"]
Sep 15 21:25:13.311154 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.311133    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"]
Sep 15 21:25:13.317242 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.317221    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-edbfd1c5-7895cdc998-4vgdg"]
Sep 15 21:25:13.715883 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.715850    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"]
Sep 15 21:25:13.716585 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.716565    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="213832e7-9059-4e83-9d8e-e47791aea0de" containerName="payload-pre-processing"
Sep 15 21:25:13.716585 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.716584    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="213832e7-9059-4e83-9d8e-e47791aea0de" containerName="payload-pre-processing"
Sep 15 21:25:13.716774 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.716601    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e65bad-bffc-4380-a18a-786df2319207" containerName="revoke-keys"
Sep 15 21:25:13.716774 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.716607    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e65bad-bffc-4380-a18a-786df2319207" containerName="revoke-keys"
Sep 15 21:25:13.716774 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.716619    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d" containerName="payload-processing"
Sep 15 21:25:13.716774 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.716625    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d" containerName="payload-processing"
Sep 15 21:25:13.716774 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.716694    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e65bad-bffc-4380-a18a-786df2319207" containerName="revoke-keys"
Sep 15 21:25:13.716774 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.716706    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="213832e7-9059-4e83-9d8e-e47791aea0de" containerName="payload-pre-processing"
Sep 15 21:25:13.716774 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.716712    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d" containerName="payload-processing"
Sep 15 21:25:13.721227 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.721207    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.723582 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.723566    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c1fb6d5a-openshift-default-dockercfg-bvgns\""
Sep 15 21:25:13.730994 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.730970    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"]
Sep 15 21:25:13.858504 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.858455    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-workload-certs\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.858701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.858544    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-data\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.858701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.858573    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4hlc\" (UniqueName: \"kubernetes.io/projected/b378c6db-f474-486f-aaaf-97e452c9d6c7-kube-api-access-z4hlc\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.858701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.858606    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b378c6db-f474-486f-aaaf-97e452c9d6c7-istiod-ca-cert\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.858701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.858645    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-podinfo\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.858937 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.858702    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-credential-socket\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.858937 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.858746    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-envoy\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.858937 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.858788    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-workload-socket\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.858937 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.858843    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-token\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.959899 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.959814    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-token\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.959899 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.959867    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-workload-certs\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.960130 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.959924    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-data\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.960130 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.959949    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4hlc\" (UniqueName: \"kubernetes.io/projected/b378c6db-f474-486f-aaaf-97e452c9d6c7-kube-api-access-z4hlc\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.960130 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.959976    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b378c6db-f474-486f-aaaf-97e452c9d6c7-istiod-ca-cert\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.960130 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.960010    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-podinfo\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.960130 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.960070    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-credential-socket\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.960130 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.960110    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-envoy\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.960503 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.960466    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-data\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.960815 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.960787    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-workload-socket\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.960928 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.960822    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-credential-socket\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.960928 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.960884    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-workload-certs\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.961138 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.961118    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-workload-socket\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.961246 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.961132    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b378c6db-f474-486f-aaaf-97e452c9d6c7-istiod-ca-cert\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.962472 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.962446    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-envoy\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.962598 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.962583    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-podinfo\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.975276 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.975243    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4hlc\" (UniqueName: \"kubernetes.io/projected/b378c6db-f474-486f-aaaf-97e452c9d6c7-kube-api-access-z4hlc\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:13.975535 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:13.975517    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-token\") pod \"e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") " pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:14.032805 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:14.032767    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:14.318216 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:14.318191    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"]
Sep 15 21:25:14.319302 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:25:14.319282    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb378c6db_f474_486f_aaaf_97e452c9d6c7.slice/crio-bd11e229e69386957672525be391c3ebf0556fc9f702ffe7e393f7ea607e9313 WatchSource:0}: Error finding container bd11e229e69386957672525be391c3ebf0556fc9f702ffe7e393f7ea607e9313: Status 404 returned error can't find the container with id bd11e229e69386957672525be391c3ebf0556fc9f702ffe7e393f7ea607e9313
Sep 15 21:25:14.321925 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:14.321887    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:25:14.322030 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:14.321963    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:25:14.322030 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:14.322005    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:25:14.675213 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:14.675133    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="213832e7-9059-4e83-9d8e-e47791aea0de" path="/var/lib/kubelet/pods/213832e7-9059-4e83-9d8e-e47791aea0de/volumes"
Sep 15 21:25:14.675600 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:14.675579    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99e65bad-bffc-4380-a18a-786df2319207" path="/var/lib/kubelet/pods/99e65bad-bffc-4380-a18a-786df2319207/volumes"
Sep 15 21:25:14.675965 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:14.675948    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d" path="/var/lib/kubelet/pods/b1f6a34b-3ef1-49d2-9fdf-6c2a7e3ee55d/volumes"
Sep 15 21:25:14.987537 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:14.987447    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr" event={"ID":"b378c6db-f474-486f-aaaf-97e452c9d6c7","Type":"ContainerStarted","Data":"25d8f543c43d84cef6f3e079a238e6b153f44e6311ddca2096283b3bb64599e3"}
Sep 15 21:25:14.987537 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:14.987500    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr" event={"ID":"b378c6db-f474-486f-aaaf-97e452c9d6c7","Type":"ContainerStarted","Data":"bd11e229e69386957672525be391c3ebf0556fc9f702ffe7e393f7ea607e9313"}
Sep 15 21:25:15.012856 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:15.012798    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr" podStartSLOduration=2.012778739 podStartE2EDuration="2.012778739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:15.009284211 +0000 UTC m=+882.871218016" watchObservedRunningTime="2026-09-15 21:25:15.012778739 +0000 UTC m=+882.874712584"
Sep 15 21:25:15.033061 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:15.033023    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:15.038063 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:15.038038    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:15.993558 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:15.993526    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:15.994625 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:15.994605    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:19.043892 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:19.043863    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"]
Sep 15 21:25:19.048997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:19.048970    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"
Sep 15 21:25:19.051439 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:19.051414    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c1fb6d5a\""
Sep 15 21:25:19.052002 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:19.051978    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c1fb6d5a-dockercfg-k9qsq\""
Sep 15 21:25:19.057436 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:19.057412    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"]
Sep 15 21:25:19.114401 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:19.114018    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6e37338-d2f6-42dd-a77d-446f6ec59fa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz\" (UID: \"d6e37338-d2f6-42dd-a77d-446f6ec59fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"
Sep 15 21:25:19.114401 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:19.114074    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwq8f\" (UniqueName: \"kubernetes.io/projected/d6e37338-d2f6-42dd-a77d-446f6ec59fa8-kube-api-access-lwq8f\") pod \"payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz\" (UID: \"d6e37338-d2f6-42dd-a77d-446f6ec59fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"
Sep 15 21:25:19.215651 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:19.215617    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6e37338-d2f6-42dd-a77d-446f6ec59fa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz\" (UID: \"d6e37338-d2f6-42dd-a77d-446f6ec59fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"
Sep 15 21:25:19.215651 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:19.215655    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwq8f\" (UniqueName: \"kubernetes.io/projected/d6e37338-d2f6-42dd-a77d-446f6ec59fa8-kube-api-access-lwq8f\") pod \"payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz\" (UID: \"d6e37338-d2f6-42dd-a77d-446f6ec59fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"
Sep 15 21:25:19.216283 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:19.216265    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6e37338-d2f6-42dd-a77d-446f6ec59fa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz\" (UID: \"d6e37338-d2f6-42dd-a77d-446f6ec59fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"
Sep 15 21:25:19.224806 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:19.224778    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwq8f\" (UniqueName: \"kubernetes.io/projected/d6e37338-d2f6-42dd-a77d-446f6ec59fa8-kube-api-access-lwq8f\") pod \"payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz\" (UID: \"d6e37338-d2f6-42dd-a77d-446f6ec59fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"
Sep 15 21:25:19.403996 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:19.403926    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"
Sep 15 21:25:19.684977 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:19.684957    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"]
Sep 15 21:25:19.687371 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:25:19.687340    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e37338_d2f6_42dd_a77d_446f6ec59fa8.slice/crio-b53046cef7c8ee3a90650de1c26811edc4ac19bfcd54e370f19622b9689d504a WatchSource:0}: Error finding container b53046cef7c8ee3a90650de1c26811edc4ac19bfcd54e370f19622b9689d504a: Status 404 returned error can't find the container with id b53046cef7c8ee3a90650de1c26811edc4ac19bfcd54e370f19622b9689d504a
Sep 15 21:25:20.018150 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:20.018050    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz" event={"ID":"d6e37338-d2f6-42dd-a77d-446f6ec59fa8","Type":"ContainerStarted","Data":"eef92b13af00cf80b8ef5edf42663f5e309528703f38786e3b9e0b3eae2f4c17"}
Sep 15 21:25:20.018150 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:20.018114    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz" event={"ID":"d6e37338-d2f6-42dd-a77d-446f6ec59fa8","Type":"ContainerStarted","Data":"b53046cef7c8ee3a90650de1c26811edc4ac19bfcd54e370f19622b9689d504a"}
Sep 15 21:25:20.018349 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:20.018224    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"
Sep 15 21:25:20.035397 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:20.035333    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz" podStartSLOduration=1.035318254 podStartE2EDuration="1.035318254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:20.033582181 +0000 UTC m=+887.895515987" watchObservedRunningTime="2026-09-15 21:25:20.035318254 +0000 UTC m=+887.897252058"
Sep 15 21:25:31.028052 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:31.027976    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"
Sep 15 21:25:32.569891 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:32.566320    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:25:32.580255 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:32.580079    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:25:41.144902 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:41.144835    2584 generic.go:356] "Generic (PLEG): container finished" podID="b378c6db-f474-486f-aaaf-97e452c9d6c7" containerID="25d8f543c43d84cef6f3e079a238e6b153f44e6311ddca2096283b3bb64599e3" exitCode=0
Sep 15 21:25:41.145487 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:41.144907    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr" event={"ID":"b378c6db-f474-486f-aaaf-97e452c9d6c7","Type":"ContainerDied","Data":"25d8f543c43d84cef6f3e079a238e6b153f44e6311ddca2096283b3bb64599e3"}
Sep 15 21:25:41.145487 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:41.145301    2584 scope.go:117] "RemoveContainer" containerID="25d8f543c43d84cef6f3e079a238e6b153f44e6311ddca2096283b3bb64599e3"
Sep 15 21:25:41.146440 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:41.146408    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:25:41.146536 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:41.146476    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:25:41.146536 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:41.146510    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:25:42.154661 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:42.154618    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr" event={"ID":"b378c6db-f474-486f-aaaf-97e452c9d6c7","Type":"ContainerStarted","Data":"43113fdcee19f18e91fe7ae2e9d415d05a4a15be1f5c67c301fe2435e7828190"}
Sep 15 21:25:42.278669 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:42.278628    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv"]
Sep 15 21:25:42.282702 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:42.282682    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv"
Sep 15 21:25:42.285221 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:42.285193    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:25:42.285341 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:42.285222    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:25:42.287179 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:42.287160    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv"]
Sep 15 21:25:42.381622 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:42.381590    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d51e9d-0849-4994-aad9-9cdcfd979fbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv\" (UID: \"34d51e9d-0849-4994-aad9-9cdcfd979fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv"
Sep 15 21:25:42.483209 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:42.483121    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d51e9d-0849-4994-aad9-9cdcfd979fbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv\" (UID: \"34d51e9d-0849-4994-aad9-9cdcfd979fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv"
Sep 15 21:25:42.483795 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:42.483769    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d51e9d-0849-4994-aad9-9cdcfd979fbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv\" (UID: \"34d51e9d-0849-4994-aad9-9cdcfd979fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv"
Sep 15 21:25:42.594887 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:42.594857    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv"
Sep 15 21:25:42.881102 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:42.881078    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv"]
Sep 15 21:25:42.883579 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:25:42.883546    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d51e9d_0849_4994_aad9_9cdcfd979fbe.slice/crio-f50ff05a321f74258b672d6bf4fa9c5eefdb8bdf31eee0dece546384e987390b WatchSource:0}: Error finding container f50ff05a321f74258b672d6bf4fa9c5eefdb8bdf31eee0dece546384e987390b: Status 404 returned error can't find the container with id f50ff05a321f74258b672d6bf4fa9c5eefdb8bdf31eee0dece546384e987390b
Sep 15 21:25:43.033926 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:43.033898    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:43.035171 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:43.035144    2584 patch_prober.go:28] interesting pod/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:25:43.035296 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:43.035189    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr" podUID="b378c6db-f474-486f-aaaf-97e452c9d6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 15 21:25:43.167151 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:43.167113    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv" event={"ID":"34d51e9d-0849-4994-aad9-9cdcfd979fbe","Type":"ContainerStarted","Data":"f50ff05a321f74258b672d6bf4fa9c5eefdb8bdf31eee0dece546384e987390b"}
Sep 15 21:25:44.033927 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:44.033900    2584 patch_prober.go:28] interesting pod/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:25:44.034123 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:44.033951    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr" podUID="b378c6db-f474-486f-aaaf-97e452c9d6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 15 21:25:44.178325 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:44.178284    2584 generic.go:356] "Generic (PLEG): container finished" podID="34d51e9d-0849-4994-aad9-9cdcfd979fbe" containerID="44f654e00b419accdae3e9a161206d35dc1ff818519a3814618cea65efa34b33" exitCode=0
Sep 15 21:25:44.178804 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:44.178354    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv" event={"ID":"34d51e9d-0849-4994-aad9-9cdcfd979fbe","Type":"ContainerDied","Data":"44f654e00b419accdae3e9a161206d35dc1ff818519a3814618cea65efa34b33"}
Sep 15 21:25:45.033806 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:45.033766    2584 patch_prober.go:28] interesting pod/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:25:45.033985 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:45.033824    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr" podUID="b378c6db-f474-486f-aaaf-97e452c9d6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 15 21:25:45.341973 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:45.341952    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv"
Sep 15 21:25:45.423979 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:45.423945    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d51e9d-0849-4994-aad9-9cdcfd979fbe-maas-api-service-ca\") pod \"34d51e9d-0849-4994-aad9-9cdcfd979fbe\" (UID: \"34d51e9d-0849-4994-aad9-9cdcfd979fbe\") "
Sep 15 21:25:45.424292 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:45.424269    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d51e9d-0849-4994-aad9-9cdcfd979fbe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34d51e9d-0849-4994-aad9-9cdcfd979fbe" (UID: "34d51e9d-0849-4994-aad9-9cdcfd979fbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:25:45.424397 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:45.424369    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d51e9d-0849-4994-aad9-9cdcfd979fbe-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:45.993773 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:45.993741    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:46.033495 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:46.033461    2584 patch_prober.go:28] interesting pod/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:25:46.033653 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:46.033520    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr" podUID="b378c6db-f474-486f-aaaf-97e452c9d6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 15 21:25:46.190682 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:46.190652    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv"
Sep 15 21:25:46.190682 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:46.190672    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv" event={"ID":"34d51e9d-0849-4994-aad9-9cdcfd979fbe","Type":"ContainerDied","Data":"f50ff05a321f74258b672d6bf4fa9c5eefdb8bdf31eee0dece546384e987390b"}
Sep 15 21:25:46.190899 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:46.190721    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f50ff05a321f74258b672d6bf4fa9c5eefdb8bdf31eee0dece546384e987390b"
Sep 15 21:25:47.033886 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:47.033848    2584 patch_prober.go:28] interesting pod/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:25:47.034263 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:47.033908    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr" podUID="b378c6db-f474-486f-aaaf-97e452c9d6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 15 21:25:47.938016 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:47.937932    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"]
Sep 15 21:25:47.938197 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:47.938161    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz" podUID="d6e37338-d2f6-42dd-a77d-446f6ec59fa8" containerName="payload-pre-processing" containerID="cri-o://eef92b13af00cf80b8ef5edf42663f5e309528703f38786e3b9e0b3eae2f4c17" gracePeriod=30
Sep 15 21:25:48.037854 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:48.037822    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:48.038750 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:48.038732    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:48.135829 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:48.135798    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv"]
Sep 15 21:25:48.141611 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:48.141585    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1fb6d5a-pt8mv"]
Sep 15 21:25:48.215431 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:48.213897    2584 generic.go:356] "Generic (PLEG): container finished" podID="d6e37338-d2f6-42dd-a77d-446f6ec59fa8" containerID="eef92b13af00cf80b8ef5edf42663f5e309528703f38786e3b9e0b3eae2f4c17" exitCode=0
Sep 15 21:25:48.215431 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:48.214811    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz" event={"ID":"d6e37338-d2f6-42dd-a77d-446f6ec59fa8","Type":"ContainerDied","Data":"eef92b13af00cf80b8ef5edf42663f5e309528703f38786e3b9e0b3eae2f4c17"}
Sep 15 21:25:48.215431 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:48.214861    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz" event={"ID":"d6e37338-d2f6-42dd-a77d-446f6ec59fa8","Type":"ContainerDied","Data":"b53046cef7c8ee3a90650de1c26811edc4ac19bfcd54e370f19622b9689d504a"}
Sep 15 21:25:48.215431 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:48.214885    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b53046cef7c8ee3a90650de1c26811edc4ac19bfcd54e370f19622b9689d504a"
Sep 15 21:25:48.236021 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:48.235988    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"
Sep 15 21:25:48.356993 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:48.356952    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwq8f\" (UniqueName: \"kubernetes.io/projected/d6e37338-d2f6-42dd-a77d-446f6ec59fa8-kube-api-access-lwq8f\") pod \"d6e37338-d2f6-42dd-a77d-446f6ec59fa8\" (UID: \"d6e37338-d2f6-42dd-a77d-446f6ec59fa8\") "
Sep 15 21:25:48.357182 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:48.357110    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6e37338-d2f6-42dd-a77d-446f6ec59fa8-plugins-config-volume\") pod \"d6e37338-d2f6-42dd-a77d-446f6ec59fa8\" (UID: \"d6e37338-d2f6-42dd-a77d-446f6ec59fa8\") "
Sep 15 21:25:48.357488 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:48.357458    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6e37338-d2f6-42dd-a77d-446f6ec59fa8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d6e37338-d2f6-42dd-a77d-446f6ec59fa8" (UID: "d6e37338-d2f6-42dd-a77d-446f6ec59fa8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:25:48.357705 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:48.357686    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6e37338-d2f6-42dd-a77d-446f6ec59fa8-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:48.359447 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:48.359418    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e37338-d2f6-42dd-a77d-446f6ec59fa8-kube-api-access-lwq8f" (OuterVolumeSpecName: "kube-api-access-lwq8f") pod "d6e37338-d2f6-42dd-a77d-446f6ec59fa8" (UID: "d6e37338-d2f6-42dd-a77d-446f6ec59fa8"). InnerVolumeSpecName "kube-api-access-lwq8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:25:48.458520 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:48.458455    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwq8f\" (UniqueName: \"kubernetes.io/projected/d6e37338-d2f6-42dd-a77d-446f6ec59fa8-kube-api-access-lwq8f\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:48.672899 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:48.672863    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34d51e9d-0849-4994-aad9-9cdcfd979fbe" path="/var/lib/kubelet/pods/34d51e9d-0849-4994-aad9-9cdcfd979fbe/volumes"
Sep 15 21:25:49.219620 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:49.219580    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"
Sep 15 21:25:49.241472 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:49.241444    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"]
Sep 15 21:25:49.245480 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:49.245455    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1fb6d5a-59b5ff47f8-jsvlz"]
Sep 15 21:25:49.296460 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:49.296432    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"]
Sep 15 21:25:49.296685 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:49.296658    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr" podUID="b378c6db-f474-486f-aaaf-97e452c9d6c7" containerName="istio-proxy" containerID="cri-o://43113fdcee19f18e91fe7ae2e9d415d05a4a15be1f5c67c301fe2435e7828190" gracePeriod=30
Sep 15 21:25:50.672491 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:50.672459    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6e37338-d2f6-42dd-a77d-446f6ec59fa8" path="/var/lib/kubelet/pods/d6e37338-d2f6-42dd-a77d-446f6ec59fa8/volumes"
Sep 15 21:25:54.551773 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.551745    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:54.720531 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.720446    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4hlc\" (UniqueName: \"kubernetes.io/projected/b378c6db-f474-486f-aaaf-97e452c9d6c7-kube-api-access-z4hlc\") pod \"b378c6db-f474-486f-aaaf-97e452c9d6c7\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") "
Sep 15 21:25:54.720531 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.720477    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-podinfo\") pod \"b378c6db-f474-486f-aaaf-97e452c9d6c7\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") "
Sep 15 21:25:54.720755 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.720552    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-workload-socket\") pod \"b378c6db-f474-486f-aaaf-97e452c9d6c7\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") "
Sep 15 21:25:54.720755 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.720605    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-credential-socket\") pod \"b378c6db-f474-486f-aaaf-97e452c9d6c7\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") "
Sep 15 21:25:54.720755 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.720626    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-data\") pod \"b378c6db-f474-486f-aaaf-97e452c9d6c7\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") "
Sep 15 21:25:54.720755 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.720650    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-token\") pod \"b378c6db-f474-486f-aaaf-97e452c9d6c7\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") "
Sep 15 21:25:54.720755 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.720667    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-workload-certs\") pod \"b378c6db-f474-486f-aaaf-97e452c9d6c7\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") "
Sep 15 21:25:54.720755 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.720725    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-envoy\") pod \"b378c6db-f474-486f-aaaf-97e452c9d6c7\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") "
Sep 15 21:25:54.721057 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.720791    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b378c6db-f474-486f-aaaf-97e452c9d6c7-istiod-ca-cert\") pod \"b378c6db-f474-486f-aaaf-97e452c9d6c7\" (UID: \"b378c6db-f474-486f-aaaf-97e452c9d6c7\") "
Sep 15 21:25:54.721057 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.720956    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b378c6db-f474-486f-aaaf-97e452c9d6c7" (UID: "b378c6db-f474-486f-aaaf-97e452c9d6c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:25:54.721057 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.720968    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b378c6db-f474-486f-aaaf-97e452c9d6c7" (UID: "b378c6db-f474-486f-aaaf-97e452c9d6c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:25:54.721057 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.720995    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-data" (OuterVolumeSpecName: "istio-data") pod "b378c6db-f474-486f-aaaf-97e452c9d6c7" (UID: "b378c6db-f474-486f-aaaf-97e452c9d6c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:25:54.721263 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.721082    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b378c6db-f474-486f-aaaf-97e452c9d6c7" (UID: "b378c6db-f474-486f-aaaf-97e452c9d6c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:25:54.721442 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.721417    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b378c6db-f474-486f-aaaf-97e452c9d6c7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b378c6db-f474-486f-aaaf-97e452c9d6c7" (UID: "b378c6db-f474-486f-aaaf-97e452c9d6c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:25:54.721679 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.721662    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-credential-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:54.721748 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.721679    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-data\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:54.721748 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.721688    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-workload-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:54.721748 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.721696    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b378c6db-f474-486f-aaaf-97e452c9d6c7-istiod-ca-cert\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:54.721748 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.721703    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-workload-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:54.722828 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.722800    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-token" (OuterVolumeSpecName: "istio-token") pod "b378c6db-f474-486f-aaaf-97e452c9d6c7" (UID: "b378c6db-f474-486f-aaaf-97e452c9d6c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:25:54.722921 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.722826    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b378c6db-f474-486f-aaaf-97e452c9d6c7-kube-api-access-z4hlc" (OuterVolumeSpecName: "kube-api-access-z4hlc") pod "b378c6db-f474-486f-aaaf-97e452c9d6c7" (UID: "b378c6db-f474-486f-aaaf-97e452c9d6c7"). InnerVolumeSpecName "kube-api-access-z4hlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:25:54.723122 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.723101    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b378c6db-f474-486f-aaaf-97e452c9d6c7" (UID: "b378c6db-f474-486f-aaaf-97e452c9d6c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:25:54.723122 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.723115    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b378c6db-f474-486f-aaaf-97e452c9d6c7" (UID: "b378c6db-f474-486f-aaaf-97e452c9d6c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:25:54.822752 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.822719    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-token\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:54.822752 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.822747    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-envoy\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:54.822938 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.822760    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4hlc\" (UniqueName: \"kubernetes.io/projected/b378c6db-f474-486f-aaaf-97e452c9d6c7-kube-api-access-z4hlc\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:54.822938 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:54.822774    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b378c6db-f474-486f-aaaf-97e452c9d6c7-istio-podinfo\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:25:55.257781 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:55.257750    2584 generic.go:356] "Generic (PLEG): container finished" podID="b378c6db-f474-486f-aaaf-97e452c9d6c7" containerID="43113fdcee19f18e91fe7ae2e9d415d05a4a15be1f5c67c301fe2435e7828190" exitCode=0
Sep 15 21:25:55.257971 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:55.257828    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"
Sep 15 21:25:55.257971 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:55.257831    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr" event={"ID":"b378c6db-f474-486f-aaaf-97e452c9d6c7","Type":"ContainerDied","Data":"43113fdcee19f18e91fe7ae2e9d415d05a4a15be1f5c67c301fe2435e7828190"}
Sep 15 21:25:55.257971 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:55.257957    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr" event={"ID":"b378c6db-f474-486f-aaaf-97e452c9d6c7","Type":"ContainerDied","Data":"bd11e229e69386957672525be391c3ebf0556fc9f702ffe7e393f7ea607e9313"}
Sep 15 21:25:55.258131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:55.257987    2584 scope.go:117] "RemoveContainer" containerID="43113fdcee19f18e91fe7ae2e9d415d05a4a15be1f5c67c301fe2435e7828190"
Sep 15 21:25:55.269122 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:55.269102    2584 scope.go:117] "RemoveContainer" containerID="25d8f543c43d84cef6f3e079a238e6b153f44e6311ddca2096283b3bb64599e3"
Sep 15 21:25:55.280083 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:55.280059    2584 scope.go:117] "RemoveContainer" containerID="43113fdcee19f18e91fe7ae2e9d415d05a4a15be1f5c67c301fe2435e7828190"
Sep 15 21:25:55.280686 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:25:55.280663    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43113fdcee19f18e91fe7ae2e9d415d05a4a15be1f5c67c301fe2435e7828190\": container with ID starting with 43113fdcee19f18e91fe7ae2e9d415d05a4a15be1f5c67c301fe2435e7828190 not found: ID does not exist" containerID="43113fdcee19f18e91fe7ae2e9d415d05a4a15be1f5c67c301fe2435e7828190"
Sep 15 21:25:55.280791 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:55.280697    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43113fdcee19f18e91fe7ae2e9d415d05a4a15be1f5c67c301fe2435e7828190"} err="failed to get container status \"43113fdcee19f18e91fe7ae2e9d415d05a4a15be1f5c67c301fe2435e7828190\": rpc error: code = NotFound desc = could not find container \"43113fdcee19f18e91fe7ae2e9d415d05a4a15be1f5c67c301fe2435e7828190\": container with ID starting with 43113fdcee19f18e91fe7ae2e9d415d05a4a15be1f5c67c301fe2435e7828190 not found: ID does not exist"
Sep 15 21:25:55.280791 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:55.280722    2584 scope.go:117] "RemoveContainer" containerID="25d8f543c43d84cef6f3e079a238e6b153f44e6311ddca2096283b3bb64599e3"
Sep 15 21:25:55.281104 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:25:55.281081    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d8f543c43d84cef6f3e079a238e6b153f44e6311ddca2096283b3bb64599e3\": container with ID starting with 25d8f543c43d84cef6f3e079a238e6b153f44e6311ddca2096283b3bb64599e3 not found: ID does not exist" containerID="25d8f543c43d84cef6f3e079a238e6b153f44e6311ddca2096283b3bb64599e3"
Sep 15 21:25:55.281219 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:55.281112    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d8f543c43d84cef6f3e079a238e6b153f44e6311ddca2096283b3bb64599e3"} err="failed to get container status \"25d8f543c43d84cef6f3e079a238e6b153f44e6311ddca2096283b3bb64599e3\": rpc error: code = NotFound desc = could not find container \"25d8f543c43d84cef6f3e079a238e6b153f44e6311ddca2096283b3bb64599e3\": container with ID starting with 25d8f543c43d84cef6f3e079a238e6b153f44e6311ddca2096283b3bb64599e3 not found: ID does not exist"
Sep 15 21:25:55.283048 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:55.283028    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"]
Sep 15 21:25:55.286755 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:55.286737    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c1fb6d5a-openshift-default-bc68d6c6d-mjcwr"]
Sep 15 21:25:56.673295 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:56.673264    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b378c6db-f474-486f-aaaf-97e452c9d6c7" path="/var/lib/kubelet/pods/b378c6db-f474-486f-aaaf-97e452c9d6c7/volumes"
Sep 15 21:25:57.626692 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.626661    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"]
Sep 15 21:25:57.627115 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.627101    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b378c6db-f474-486f-aaaf-97e452c9d6c7" containerName="istio-proxy"
Sep 15 21:25:57.627115 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.627115    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b378c6db-f474-486f-aaaf-97e452c9d6c7" containerName="istio-proxy"
Sep 15 21:25:57.627244 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.627129    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6e37338-d2f6-42dd-a77d-446f6ec59fa8" containerName="payload-pre-processing"
Sep 15 21:25:57.627244 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.627135    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6e37338-d2f6-42dd-a77d-446f6ec59fa8" containerName="payload-pre-processing"
Sep 15 21:25:57.627244 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.627155    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d51e9d-0849-4994-aad9-9cdcfd979fbe" containerName="revoke-keys"
Sep 15 21:25:57.627244 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.627161    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d51e9d-0849-4994-aad9-9cdcfd979fbe" containerName="revoke-keys"
Sep 15 21:25:57.627244 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.627222    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b378c6db-f474-486f-aaaf-97e452c9d6c7" containerName="istio-proxy"
Sep 15 21:25:57.627244 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.627230    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="34d51e9d-0849-4994-aad9-9cdcfd979fbe" containerName="revoke-keys"
Sep 15 21:25:57.627244 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.627238    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6e37338-d2f6-42dd-a77d-446f6ec59fa8" containerName="payload-pre-processing"
Sep 15 21:25:57.631775 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.631758    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"
Sep 15 21:25:57.633803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.633777    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ce9161fd\""
Sep 15 21:25:57.637543 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.637520    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"]
Sep 15 21:25:57.669656 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.669626    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"]
Sep 15 21:25:57.670077 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.670064    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b378c6db-f474-486f-aaaf-97e452c9d6c7" containerName="istio-proxy"
Sep 15 21:25:57.670077 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.670077    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b378c6db-f474-486f-aaaf-97e452c9d6c7" containerName="istio-proxy"
Sep 15 21:25:57.670194 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.670148    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b378c6db-f474-486f-aaaf-97e452c9d6c7" containerName="istio-proxy"
Sep 15 21:25:57.673182 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.673167    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"
Sep 15 21:25:57.675738 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.675719    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ce9161fd-dockercfg-fs7xr\""
Sep 15 21:25:57.684530 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.684508    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"]
Sep 15 21:25:57.762585 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.762114    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqv8k\" (UniqueName: \"kubernetes.io/projected/52e7f9df-ec89-4ac2-b752-4b7965aee631-kube-api-access-bqv8k\") pod \"payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf\" (UID: \"52e7f9df-ec89-4ac2-b752-4b7965aee631\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"
Sep 15 21:25:57.762585 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.762214    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e7f9df-ec89-4ac2-b752-4b7965aee631-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf\" (UID: \"52e7f9df-ec89-4ac2-b752-4b7965aee631\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"
Sep 15 21:25:57.863728 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.863692    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqv8k\" (UniqueName: \"kubernetes.io/projected/52e7f9df-ec89-4ac2-b752-4b7965aee631-kube-api-access-bqv8k\") pod \"payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf\" (UID: \"52e7f9df-ec89-4ac2-b752-4b7965aee631\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"
Sep 15 21:25:57.863915 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.863817    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e7f9df-ec89-4ac2-b752-4b7965aee631-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf\" (UID: \"52e7f9df-ec89-4ac2-b752-4b7965aee631\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"
Sep 15 21:25:57.863915 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.863901    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pc9m\" (UniqueName: \"kubernetes.io/projected/95041038-4cf4-4923-a3f2-9bad5ad2011d-kube-api-access-5pc9m\") pod \"payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j\" (UID: \"95041038-4cf4-4923-a3f2-9bad5ad2011d\") " pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"
Sep 15 21:25:57.864029 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.863960    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95041038-4cf4-4923-a3f2-9bad5ad2011d-plugins-config-volume\") pod \"payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j\" (UID: \"95041038-4cf4-4923-a3f2-9bad5ad2011d\") " pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"
Sep 15 21:25:57.864606 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.864579    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e7f9df-ec89-4ac2-b752-4b7965aee631-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf\" (UID: \"52e7f9df-ec89-4ac2-b752-4b7965aee631\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"
Sep 15 21:25:57.872081 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.872064    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqv8k\" (UniqueName: \"kubernetes.io/projected/52e7f9df-ec89-4ac2-b752-4b7965aee631-kube-api-access-bqv8k\") pod \"payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf\" (UID: \"52e7f9df-ec89-4ac2-b752-4b7965aee631\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"
Sep 15 21:25:57.944234 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.944165    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"
Sep 15 21:25:57.964981 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.964950    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95041038-4cf4-4923-a3f2-9bad5ad2011d-plugins-config-volume\") pod \"payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j\" (UID: \"95041038-4cf4-4923-a3f2-9bad5ad2011d\") " pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"
Sep 15 21:25:57.965158 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.965140    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pc9m\" (UniqueName: \"kubernetes.io/projected/95041038-4cf4-4923-a3f2-9bad5ad2011d-kube-api-access-5pc9m\") pod \"payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j\" (UID: \"95041038-4cf4-4923-a3f2-9bad5ad2011d\") " pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"
Sep 15 21:25:57.965585 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.965567    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95041038-4cf4-4923-a3f2-9bad5ad2011d-plugins-config-volume\") pod \"payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j\" (UID: \"95041038-4cf4-4923-a3f2-9bad5ad2011d\") " pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"
Sep 15 21:25:57.973857 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.973835    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pc9m\" (UniqueName: \"kubernetes.io/projected/95041038-4cf4-4923-a3f2-9bad5ad2011d-kube-api-access-5pc9m\") pod \"payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j\" (UID: \"95041038-4cf4-4923-a3f2-9bad5ad2011d\") " pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"
Sep 15 21:25:57.983765 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:57.983741    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"
Sep 15 21:25:58.217866 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:58.217838    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"]
Sep 15 21:25:58.219728 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:25:58.219702    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e7f9df_ec89_4ac2_b752_4b7965aee631.slice/crio-2f6e0e2b9979aa8dd7bf730aa47f67f5c8b0b5b700f49e124bb3e0145f465363 WatchSource:0}: Error finding container 2f6e0e2b9979aa8dd7bf730aa47f67f5c8b0b5b700f49e124bb3e0145f465363: Status 404 returned error can't find the container with id 2f6e0e2b9979aa8dd7bf730aa47f67f5c8b0b5b700f49e124bb3e0145f465363
Sep 15 21:25:58.265468 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:58.265441    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"]
Sep 15 21:25:58.270495 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:25:58.270473    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95041038_4cf4_4923_a3f2_9bad5ad2011d.slice/crio-21647df989135f9646b457f1ca911bd615649e781e33aa1f77fef652f79e86e4 WatchSource:0}: Error finding container 21647df989135f9646b457f1ca911bd615649e781e33aa1f77fef652f79e86e4: Status 404 returned error can't find the container with id 21647df989135f9646b457f1ca911bd615649e781e33aa1f77fef652f79e86e4
Sep 15 21:25:58.284950 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:58.284917    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j" event={"ID":"95041038-4cf4-4923-a3f2-9bad5ad2011d","Type":"ContainerStarted","Data":"21647df989135f9646b457f1ca911bd615649e781e33aa1f77fef652f79e86e4"}
Sep 15 21:25:58.286297 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:58.286276    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf" event={"ID":"52e7f9df-ec89-4ac2-b752-4b7965aee631","Type":"ContainerStarted","Data":"2f6e0e2b9979aa8dd7bf730aa47f67f5c8b0b5b700f49e124bb3e0145f465363"}
Sep 15 21:25:59.294684 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:59.294642    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j" event={"ID":"95041038-4cf4-4923-a3f2-9bad5ad2011d","Type":"ContainerStarted","Data":"6c2535f93a681f2eaf58fbc88e0cd644644143846205e283937b509d3e7880c6"}
Sep 15 21:25:59.295142 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:59.294704    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"
Sep 15 21:25:59.296366 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:59.296339    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf" event={"ID":"52e7f9df-ec89-4ac2-b752-4b7965aee631","Type":"ContainerStarted","Data":"0be31d1c96fd54d30633ed129ec49da3280e3033da23a1391d06a137ed1c526e"}
Sep 15 21:25:59.296522 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:59.296441    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"
Sep 15 21:25:59.312521 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:59.312481    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j" podStartSLOduration=2.312467836 podStartE2EDuration="2.312467836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:59.311008738 +0000 UTC m=+927.172942542" watchObservedRunningTime="2026-09-15 21:25:59.312467836 +0000 UTC m=+927.174401640"
Sep 15 21:25:59.327003 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:25:59.326966    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf" podStartSLOduration=2.326953337 podStartE2EDuration="2.326953337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:59.324863445 +0000 UTC m=+927.186797248" watchObservedRunningTime="2026-09-15 21:25:59.326953337 +0000 UTC m=+927.188887142"
Sep 15 21:26:10.303989 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:10.303964    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"
Sep 15 21:26:10.304368 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:10.304026    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"
Sep 15 21:26:22.990190 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:22.990155    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn"]
Sep 15 21:26:22.994246 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:22.994227    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn"
Sep 15 21:26:22.996438 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:22.996419    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:26:22.996526 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:22.996441    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:26:23.001788 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:23.001770    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn"]
Sep 15 21:26:23.131204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:23.131172    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0f07a4-0633-4c2c-bbfc-87b73057efb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn\" (UID: \"af0f07a4-0633-4c2c-bbfc-87b73057efb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn"
Sep 15 21:26:23.232756 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:23.232721    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0f07a4-0633-4c2c-bbfc-87b73057efb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn\" (UID: \"af0f07a4-0633-4c2c-bbfc-87b73057efb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn"
Sep 15 21:26:23.233314 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:23.233294    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0f07a4-0633-4c2c-bbfc-87b73057efb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn\" (UID: \"af0f07a4-0633-4c2c-bbfc-87b73057efb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn"
Sep 15 21:26:23.307107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:23.307066    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn"
Sep 15 21:26:23.577757 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:23.577734    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn"]
Sep 15 21:26:23.579613 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:26:23.579584    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0f07a4_0633_4c2c_bbfc_87b73057efb8.slice/crio-04c740452b2afbee778d28ea640835767a326715b534874bd81a9ee0ebcec242 WatchSource:0}: Error finding container 04c740452b2afbee778d28ea640835767a326715b534874bd81a9ee0ebcec242: Status 404 returned error can't find the container with id 04c740452b2afbee778d28ea640835767a326715b534874bd81a9ee0ebcec242
Sep 15 21:26:24.443274 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:24.443246    2584 generic.go:356] "Generic (PLEG): container finished" podID="af0f07a4-0633-4c2c-bbfc-87b73057efb8" containerID="9395b527155194a6a062e70fce9ba9cb944de3b42330e0b17f387768b5056263" exitCode=0
Sep 15 21:26:24.443658 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:24.443331    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn" event={"ID":"af0f07a4-0633-4c2c-bbfc-87b73057efb8","Type":"ContainerDied","Data":"9395b527155194a6a062e70fce9ba9cb944de3b42330e0b17f387768b5056263"}
Sep 15 21:26:24.443658 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:24.443401    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn" event={"ID":"af0f07a4-0633-4c2c-bbfc-87b73057efb8","Type":"ContainerStarted","Data":"04c740452b2afbee778d28ea640835767a326715b534874bd81a9ee0ebcec242"}
Sep 15 21:26:25.574753 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:25.574732    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn"
Sep 15 21:26:25.755461 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:25.755348    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0f07a4-0633-4c2c-bbfc-87b73057efb8-maas-api-service-ca\") pod \"af0f07a4-0633-4c2c-bbfc-87b73057efb8\" (UID: \"af0f07a4-0633-4c2c-bbfc-87b73057efb8\") "
Sep 15 21:26:25.755729 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:25.755704    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0f07a4-0633-4c2c-bbfc-87b73057efb8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af0f07a4-0633-4c2c-bbfc-87b73057efb8" (UID: "af0f07a4-0633-4c2c-bbfc-87b73057efb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:26:25.856525 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:25.856494    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0f07a4-0633-4c2c-bbfc-87b73057efb8-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:26:25.952739 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:25.952705    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04c28d-qlnbj"]
Sep 15 21:26:25.953226 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:25.953209    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af0f07a4-0633-4c2c-bbfc-87b73057efb8" containerName="revoke-keys"
Sep 15 21:26:25.953226 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:25.953225    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="af0f07a4-0633-4c2c-bbfc-87b73057efb8" containerName="revoke-keys"
Sep 15 21:26:25.953429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:25.953315    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="af0f07a4-0633-4c2c-bbfc-87b73057efb8" containerName="revoke-keys"
Sep 15 21:26:25.956687 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:25.956666    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04c28d-qlnbj"
Sep 15 21:26:25.963542 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:25.963516    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04c28d-qlnbj"]
Sep 15 21:26:26.058109 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:26.058071    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db9da6e7-19c6-4f73-8927-b4ab92300a33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-04c28d-qlnbj\" (UID: \"db9da6e7-19c6-4f73-8927-b4ab92300a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04c28d-qlnbj"
Sep 15 21:26:26.159670 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:26.159637    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db9da6e7-19c6-4f73-8927-b4ab92300a33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-04c28d-qlnbj\" (UID: \"db9da6e7-19c6-4f73-8927-b4ab92300a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04c28d-qlnbj"
Sep 15 21:26:26.160220 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:26.160201    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db9da6e7-19c6-4f73-8927-b4ab92300a33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-04c28d-qlnbj\" (UID: \"db9da6e7-19c6-4f73-8927-b4ab92300a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04c28d-qlnbj"
Sep 15 21:26:26.268838 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:26.268799    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04c28d-qlnbj"
Sep 15 21:26:26.456017 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:26.455993    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn"
Sep 15 21:26:26.456180 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:26.456037    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn" event={"ID":"af0f07a4-0633-4c2c-bbfc-87b73057efb8","Type":"ContainerDied","Data":"04c740452b2afbee778d28ea640835767a326715b534874bd81a9ee0ebcec242"}
Sep 15 21:26:26.456180 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:26.456090    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04c740452b2afbee778d28ea640835767a326715b534874bd81a9ee0ebcec242"
Sep 15 21:26:26.546315 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:26.546219    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04c28d-qlnbj"]
Sep 15 21:26:26.548903 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:26:26.548874    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9da6e7_19c6_4f73_8927_b4ab92300a33.slice/crio-d02e958a84d717899b063295e8c938b6734d32fa8f5de2b400521d0c368512ee WatchSource:0}: Error finding container d02e958a84d717899b063295e8c938b6734d32fa8f5de2b400521d0c368512ee: Status 404 returned error can't find the container with id d02e958a84d717899b063295e8c938b6734d32fa8f5de2b400521d0c368512ee
Sep 15 21:26:27.462934 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:27.462899    2584 generic.go:356] "Generic (PLEG): container finished" podID="db9da6e7-19c6-4f73-8927-b4ab92300a33" containerID="e2b5548ff687698c22a348cfb947c13e13528ab299bbfda3491f7077fe0b4e83" exitCode=0
Sep 15 21:26:27.463279 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:27.462965    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04c28d-qlnbj" event={"ID":"db9da6e7-19c6-4f73-8927-b4ab92300a33","Type":"ContainerDied","Data":"e2b5548ff687698c22a348cfb947c13e13528ab299bbfda3491f7077fe0b4e83"}
Sep 15 21:26:27.463279 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:27.463018    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04c28d-qlnbj" event={"ID":"db9da6e7-19c6-4f73-8927-b4ab92300a33","Type":"ContainerStarted","Data":"d02e958a84d717899b063295e8c938b6734d32fa8f5de2b400521d0c368512ee"}
Sep 15 21:26:28.629209 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:28.629190    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04c28d-qlnbj"
Sep 15 21:26:28.783329 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:28.783304    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db9da6e7-19c6-4f73-8927-b4ab92300a33-maas-api-service-ca\") pod \"db9da6e7-19c6-4f73-8927-b4ab92300a33\" (UID: \"db9da6e7-19c6-4f73-8927-b4ab92300a33\") "
Sep 15 21:26:28.783695 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:28.783671    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9da6e7-19c6-4f73-8927-b4ab92300a33-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "db9da6e7-19c6-4f73-8927-b4ab92300a33" (UID: "db9da6e7-19c6-4f73-8927-b4ab92300a33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:26:28.884940 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:28.884915    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db9da6e7-19c6-4f73-8927-b4ab92300a33-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:26:29.475711 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:29.475676    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04c28d-qlnbj"
Sep 15 21:26:29.475893 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:29.475719    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04c28d-qlnbj" event={"ID":"db9da6e7-19c6-4f73-8927-b4ab92300a33","Type":"ContainerDied","Data":"d02e958a84d717899b063295e8c938b6734d32fa8f5de2b400521d0c368512ee"}
Sep 15 21:26:29.475893 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:29.475775    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d02e958a84d717899b063295e8c938b6734d32fa8f5de2b400521d0c368512ee"
Sep 15 21:26:35.048438 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.048402    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"]
Sep 15 21:26:35.049025 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.048651    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j" podUID="95041038-4cf4-4923-a3f2-9bad5ad2011d" containerName="payload-processing" containerID="cri-o://6c2535f93a681f2eaf58fbc88e0cd644644143846205e283937b509d3e7880c6" gracePeriod=30
Sep 15 21:26:35.056881 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.056854    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"]
Sep 15 21:26:35.058530 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.057266    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf" podUID="52e7f9df-ec89-4ac2-b752-4b7965aee631" containerName="payload-pre-processing" containerID="cri-o://0be31d1c96fd54d30633ed129ec49da3280e3033da23a1391d06a137ed1c526e" gracePeriod=30
Sep 15 21:26:35.295189 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.295162    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn"]
Sep 15 21:26:35.302012 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.301942    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ce9161fd-mgjfn"]
Sep 15 21:26:35.336804 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.336644    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"
Sep 15 21:26:35.352119 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.352097    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"
Sep 15 21:26:35.453923 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.453841    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95041038-4cf4-4923-a3f2-9bad5ad2011d-plugins-config-volume\") pod \"95041038-4cf4-4923-a3f2-9bad5ad2011d\" (UID: \"95041038-4cf4-4923-a3f2-9bad5ad2011d\") "
Sep 15 21:26:35.454113 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.453953    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pc9m\" (UniqueName: \"kubernetes.io/projected/95041038-4cf4-4923-a3f2-9bad5ad2011d-kube-api-access-5pc9m\") pod \"95041038-4cf4-4923-a3f2-9bad5ad2011d\" (UID: \"95041038-4cf4-4923-a3f2-9bad5ad2011d\") "
Sep 15 21:26:35.454113 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.453985    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e7f9df-ec89-4ac2-b752-4b7965aee631-plugins-config-volume\") pod \"52e7f9df-ec89-4ac2-b752-4b7965aee631\" (UID: \"52e7f9df-ec89-4ac2-b752-4b7965aee631\") "
Sep 15 21:26:35.454113 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.454017    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqv8k\" (UniqueName: \"kubernetes.io/projected/52e7f9df-ec89-4ac2-b752-4b7965aee631-kube-api-access-bqv8k\") pod \"52e7f9df-ec89-4ac2-b752-4b7965aee631\" (UID: \"52e7f9df-ec89-4ac2-b752-4b7965aee631\") "
Sep 15 21:26:35.455193 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.455112    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e7f9df-ec89-4ac2-b752-4b7965aee631-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "52e7f9df-ec89-4ac2-b752-4b7965aee631" (UID: "52e7f9df-ec89-4ac2-b752-4b7965aee631"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:26:35.455193 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.455154    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95041038-4cf4-4923-a3f2-9bad5ad2011d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95041038-4cf4-4923-a3f2-9bad5ad2011d" (UID: "95041038-4cf4-4923-a3f2-9bad5ad2011d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:26:35.456654 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.456633    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e7f9df-ec89-4ac2-b752-4b7965aee631-kube-api-access-bqv8k" (OuterVolumeSpecName: "kube-api-access-bqv8k") pod "52e7f9df-ec89-4ac2-b752-4b7965aee631" (UID: "52e7f9df-ec89-4ac2-b752-4b7965aee631"). InnerVolumeSpecName "kube-api-access-bqv8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:26:35.456926 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.456903    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95041038-4cf4-4923-a3f2-9bad5ad2011d-kube-api-access-5pc9m" (OuterVolumeSpecName: "kube-api-access-5pc9m") pod "95041038-4cf4-4923-a3f2-9bad5ad2011d" (UID: "95041038-4cf4-4923-a3f2-9bad5ad2011d"). InnerVolumeSpecName "kube-api-access-5pc9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:26:35.514677 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.514650    2584 generic.go:356] "Generic (PLEG): container finished" podID="95041038-4cf4-4923-a3f2-9bad5ad2011d" containerID="6c2535f93a681f2eaf58fbc88e0cd644644143846205e283937b509d3e7880c6" exitCode=0
Sep 15 21:26:35.514780 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.514695    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"
Sep 15 21:26:35.514780 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.514738    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j" event={"ID":"95041038-4cf4-4923-a3f2-9bad5ad2011d","Type":"ContainerDied","Data":"6c2535f93a681f2eaf58fbc88e0cd644644143846205e283937b509d3e7880c6"}
Sep 15 21:26:35.514875 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.514791    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j" event={"ID":"95041038-4cf4-4923-a3f2-9bad5ad2011d","Type":"ContainerDied","Data":"21647df989135f9646b457f1ca911bd615649e781e33aa1f77fef652f79e86e4"}
Sep 15 21:26:35.514875 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.514813    2584 scope.go:117] "RemoveContainer" containerID="6c2535f93a681f2eaf58fbc88e0cd644644143846205e283937b509d3e7880c6"
Sep 15 21:26:35.516105 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.516083    2584 generic.go:356] "Generic (PLEG): container finished" podID="52e7f9df-ec89-4ac2-b752-4b7965aee631" containerID="0be31d1c96fd54d30633ed129ec49da3280e3033da23a1391d06a137ed1c526e" exitCode=0
Sep 15 21:26:35.516167 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.516136    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"
Sep 15 21:26:35.516219 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.516173    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf" event={"ID":"52e7f9df-ec89-4ac2-b752-4b7965aee631","Type":"ContainerDied","Data":"0be31d1c96fd54d30633ed129ec49da3280e3033da23a1391d06a137ed1c526e"}
Sep 15 21:26:35.516262 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.516221    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf" event={"ID":"52e7f9df-ec89-4ac2-b752-4b7965aee631","Type":"ContainerDied","Data":"2f6e0e2b9979aa8dd7bf730aa47f67f5c8b0b5b700f49e124bb3e0145f465363"}
Sep 15 21:26:35.523936 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.523921    2584 scope.go:117] "RemoveContainer" containerID="6c2535f93a681f2eaf58fbc88e0cd644644143846205e283937b509d3e7880c6"
Sep 15 21:26:35.524164 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:26:35.524145    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2535f93a681f2eaf58fbc88e0cd644644143846205e283937b509d3e7880c6\": container with ID starting with 6c2535f93a681f2eaf58fbc88e0cd644644143846205e283937b509d3e7880c6 not found: ID does not exist" containerID="6c2535f93a681f2eaf58fbc88e0cd644644143846205e283937b509d3e7880c6"
Sep 15 21:26:35.524232 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.524175    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2535f93a681f2eaf58fbc88e0cd644644143846205e283937b509d3e7880c6"} err="failed to get container status \"6c2535f93a681f2eaf58fbc88e0cd644644143846205e283937b509d3e7880c6\": rpc error: code = NotFound desc = could not find container \"6c2535f93a681f2eaf58fbc88e0cd644644143846205e283937b509d3e7880c6\": container with ID starting with 6c2535f93a681f2eaf58fbc88e0cd644644143846205e283937b509d3e7880c6 not found: ID does not exist"
Sep 15 21:26:35.524232 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.524194    2584 scope.go:117] "RemoveContainer" containerID="0be31d1c96fd54d30633ed129ec49da3280e3033da23a1391d06a137ed1c526e"
Sep 15 21:26:35.538478 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.538448    2584 scope.go:117] "RemoveContainer" containerID="0be31d1c96fd54d30633ed129ec49da3280e3033da23a1391d06a137ed1c526e"
Sep 15 21:26:35.539148 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:26:35.539072    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be31d1c96fd54d30633ed129ec49da3280e3033da23a1391d06a137ed1c526e\": container with ID starting with 0be31d1c96fd54d30633ed129ec49da3280e3033da23a1391d06a137ed1c526e not found: ID does not exist" containerID="0be31d1c96fd54d30633ed129ec49da3280e3033da23a1391d06a137ed1c526e"
Sep 15 21:26:35.539148 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.539104    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be31d1c96fd54d30633ed129ec49da3280e3033da23a1391d06a137ed1c526e"} err="failed to get container status \"0be31d1c96fd54d30633ed129ec49da3280e3033da23a1391d06a137ed1c526e\": rpc error: code = NotFound desc = could not find container \"0be31d1c96fd54d30633ed129ec49da3280e3033da23a1391d06a137ed1c526e\": container with ID starting with 0be31d1c96fd54d30633ed129ec49da3280e3033da23a1391d06a137ed1c526e not found: ID does not exist"
Sep 15 21:26:35.555089 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.554957    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pc9m\" (UniqueName: \"kubernetes.io/projected/95041038-4cf4-4923-a3f2-9bad5ad2011d-kube-api-access-5pc9m\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:26:35.555089 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.554991    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e7f9df-ec89-4ac2-b752-4b7965aee631-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:26:35.555089 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.555005    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqv8k\" (UniqueName: \"kubernetes.io/projected/52e7f9df-ec89-4ac2-b752-4b7965aee631-kube-api-access-bqv8k\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:26:35.555089 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.555018    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95041038-4cf4-4923-a3f2-9bad5ad2011d-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:26:35.564262 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.564235    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"]
Sep 15 21:26:35.572438 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.572413    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ce9161fd-5ff5b4cf8b-vbflf"]
Sep 15 21:26:35.581841 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.581816    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"]
Sep 15 21:26:35.585531 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:35.585512    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ce9161fd-8478c685f7-4xh9j"]
Sep 15 21:26:36.672597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:36.672563    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52e7f9df-ec89-4ac2-b752-4b7965aee631" path="/var/lib/kubelet/pods/52e7f9df-ec89-4ac2-b752-4b7965aee631/volumes"
Sep 15 21:26:36.672955 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:36.672943    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95041038-4cf4-4923-a3f2-9bad5ad2011d" path="/var/lib/kubelet/pods/95041038-4cf4-4923-a3f2-9bad5ad2011d/volumes"
Sep 15 21:26:36.673224 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:36.673211    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af0f07a4-0633-4c2c-bbfc-87b73057efb8" path="/var/lib/kubelet/pods/af0f07a4-0633-4c2c-bbfc-87b73057efb8/volumes"
Sep 15 21:26:40.818819 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:40.818791    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"]
Sep 15 21:26:40.819288 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:40.819003    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d" podUID="b89d2245-e565-4f55-b63a-93d50db3843b" containerName="payload-processing" containerID="cri-o://28ec9c29b1edb3e68f93978a99a4b7c66d16e302cb122260e0d42ce62cd5d3e9" gracePeriod=30
Sep 15 21:26:41.069890 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.069752    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"
Sep 15 21:26:41.072519 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.072450    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04c28d-qlnbj"]
Sep 15 21:26:41.074696 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.074674    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-04c28d-qlnbj"]
Sep 15 21:26:41.112051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.112017    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p46k\" (UniqueName: \"kubernetes.io/projected/b89d2245-e565-4f55-b63a-93d50db3843b-kube-api-access-7p46k\") pod \"b89d2245-e565-4f55-b63a-93d50db3843b\" (UID: \"b89d2245-e565-4f55-b63a-93d50db3843b\") "
Sep 15 21:26:41.112212 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.112077    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b89d2245-e565-4f55-b63a-93d50db3843b-plugins-config-volume\") pod \"b89d2245-e565-4f55-b63a-93d50db3843b\" (UID: \"b89d2245-e565-4f55-b63a-93d50db3843b\") "
Sep 15 21:26:41.112527 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.112500    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b89d2245-e565-4f55-b63a-93d50db3843b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b89d2245-e565-4f55-b63a-93d50db3843b" (UID: "b89d2245-e565-4f55-b63a-93d50db3843b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:26:41.112679 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.112652    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b89d2245-e565-4f55-b63a-93d50db3843b-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:26:41.114494 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.114453    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89d2245-e565-4f55-b63a-93d50db3843b-kube-api-access-7p46k" (OuterVolumeSpecName: "kube-api-access-7p46k") pod "b89d2245-e565-4f55-b63a-93d50db3843b" (UID: "b89d2245-e565-4f55-b63a-93d50db3843b"). InnerVolumeSpecName "kube-api-access-7p46k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:26:41.214168 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.214130    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p46k\" (UniqueName: \"kubernetes.io/projected/b89d2245-e565-4f55-b63a-93d50db3843b-kube-api-access-7p46k\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:26:41.554217 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.554183    2584 generic.go:356] "Generic (PLEG): container finished" podID="b89d2245-e565-4f55-b63a-93d50db3843b" containerID="28ec9c29b1edb3e68f93978a99a4b7c66d16e302cb122260e0d42ce62cd5d3e9" exitCode=0
Sep 15 21:26:41.554427 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.554231    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"
Sep 15 21:26:41.554427 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.554263    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d" event={"ID":"b89d2245-e565-4f55-b63a-93d50db3843b","Type":"ContainerDied","Data":"28ec9c29b1edb3e68f93978a99a4b7c66d16e302cb122260e0d42ce62cd5d3e9"}
Sep 15 21:26:41.554427 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.554330    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d" event={"ID":"b89d2245-e565-4f55-b63a-93d50db3843b","Type":"ContainerDied","Data":"76b369cd4d95a056963a7788b1b3f2255061d6f2c6b4692c94b04bf8b96f8d9a"}
Sep 15 21:26:41.554427 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.554350    2584 scope.go:117] "RemoveContainer" containerID="28ec9c29b1edb3e68f93978a99a4b7c66d16e302cb122260e0d42ce62cd5d3e9"
Sep 15 21:26:41.564335 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.564316    2584 scope.go:117] "RemoveContainer" containerID="28ec9c29b1edb3e68f93978a99a4b7c66d16e302cb122260e0d42ce62cd5d3e9"
Sep 15 21:26:41.564622 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:26:41.564604    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ec9c29b1edb3e68f93978a99a4b7c66d16e302cb122260e0d42ce62cd5d3e9\": container with ID starting with 28ec9c29b1edb3e68f93978a99a4b7c66d16e302cb122260e0d42ce62cd5d3e9 not found: ID does not exist" containerID="28ec9c29b1edb3e68f93978a99a4b7c66d16e302cb122260e0d42ce62cd5d3e9"
Sep 15 21:26:41.564680 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.564631    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ec9c29b1edb3e68f93978a99a4b7c66d16e302cb122260e0d42ce62cd5d3e9"} err="failed to get container status \"28ec9c29b1edb3e68f93978a99a4b7c66d16e302cb122260e0d42ce62cd5d3e9\": rpc error: code = NotFound desc = could not find container \"28ec9c29b1edb3e68f93978a99a4b7c66d16e302cb122260e0d42ce62cd5d3e9\": container with ID starting with 28ec9c29b1edb3e68f93978a99a4b7c66d16e302cb122260e0d42ce62cd5d3e9 not found: ID does not exist"
Sep 15 21:26:41.577975 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.577955    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"]
Sep 15 21:26:41.582634 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:41.582613    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-04c28d-585f648b55-kdh8d"]
Sep 15 21:26:42.673211 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:42.673179    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b89d2245-e565-4f55-b63a-93d50db3843b" path="/var/lib/kubelet/pods/b89d2245-e565-4f55-b63a-93d50db3843b/volumes"
Sep 15 21:26:42.673597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:26:42.673535    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db9da6e7-19c6-4f73-8927-b4ab92300a33" path="/var/lib/kubelet/pods/db9da6e7-19c6-4f73-8927-b4ab92300a33/volumes"
Sep 15 21:27:24.531549 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.531508    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"]
Sep 15 21:27:24.532614 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.532218    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52e7f9df-ec89-4ac2-b752-4b7965aee631" containerName="payload-pre-processing"
Sep 15 21:27:24.532614 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.532241    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="52e7f9df-ec89-4ac2-b752-4b7965aee631" containerName="payload-pre-processing"
Sep 15 21:27:24.532614 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.532254    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95041038-4cf4-4923-a3f2-9bad5ad2011d" containerName="payload-processing"
Sep 15 21:27:24.532614 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.532263    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="95041038-4cf4-4923-a3f2-9bad5ad2011d" containerName="payload-processing"
Sep 15 21:27:24.532614 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.532295    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b89d2245-e565-4f55-b63a-93d50db3843b" containerName="payload-processing"
Sep 15 21:27:24.532614 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.532304    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b89d2245-e565-4f55-b63a-93d50db3843b" containerName="payload-processing"
Sep 15 21:27:24.532614 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.532326    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db9da6e7-19c6-4f73-8927-b4ab92300a33" containerName="revoke-keys"
Sep 15 21:27:24.532614 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.532334    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="db9da6e7-19c6-4f73-8927-b4ab92300a33" containerName="revoke-keys"
Sep 15 21:27:24.532614 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.532467    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="db9da6e7-19c6-4f73-8927-b4ab92300a33" containerName="revoke-keys"
Sep 15 21:27:24.532614 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.532493    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b89d2245-e565-4f55-b63a-93d50db3843b" containerName="payload-processing"
Sep 15 21:27:24.532614 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.532504    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="52e7f9df-ec89-4ac2-b752-4b7965aee631" containerName="payload-pre-processing"
Sep 15 21:27:24.532614 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.532515    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="95041038-4cf4-4923-a3f2-9bad5ad2011d" containerName="payload-processing"
Sep 15 21:27:24.536636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.536617    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.538888 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.538868    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-026d11-openshift-default-dockercfg-h4xsn\""
Sep 15 21:27:24.551406 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.551041    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"]
Sep 15 21:27:24.676801 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.676768    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-workload-socket\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.676985 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.676807    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-podinfo\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.676985 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.676844    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-envoy\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.676985 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.676884    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istiod-ca-cert\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.676985 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.676957    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd65c\" (UniqueName: \"kubernetes.io/projected/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-kube-api-access-rd65c\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.677215 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.676991    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-credential-socket\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.677215 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.677014    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-workload-certs\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.677215 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.677074    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-data\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.677215 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.677147    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-token\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.777930 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.777898    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-data\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.778095 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.778046    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-token\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.778095 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.778075    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-workload-socket\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.778218 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.778095    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-podinfo\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.778218 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.778126    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-envoy\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.778218 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.778161    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istiod-ca-cert\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.778218 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.778194    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd65c\" (UniqueName: \"kubernetes.io/projected/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-kube-api-access-rd65c\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.778444 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.778224    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-credential-socket\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.778444 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.778256    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-workload-certs\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.778444 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.778280    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-data\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.778617 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.778434    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-workload-socket\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.778672 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.778639    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-workload-certs\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.778734 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.778668    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-credential-socket\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.779060 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.779008    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istiod-ca-cert\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.780334 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.780311    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-envoy\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.780570 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.780552    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-podinfo\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.789613 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.789549    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-token\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.789981 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.789960    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd65c\" (UniqueName: \"kubernetes.io/projected/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-kube-api-access-rd65c\") pod \"e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") " pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:24.848456 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:24.848432    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:25.125816 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:25.125792    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"]
Sep 15 21:27:25.127337 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:27:25.127311    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8391d48a_ac9c_433a_8dd3_f4854d9c85fa.slice/crio-bda3b4e1f7c8956b133c10af3103eff2fbe7c6516cca1680bb3efbe3184d7970 WatchSource:0}: Error finding container bda3b4e1f7c8956b133c10af3103eff2fbe7c6516cca1680bb3efbe3184d7970: Status 404 returned error can't find the container with id bda3b4e1f7c8956b133c10af3103eff2fbe7c6516cca1680bb3efbe3184d7970
Sep 15 21:27:25.129610 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:25.129579    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:27:25.129718 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:25.129654    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:27:25.129718 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:25.129700    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:27:25.815422 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:25.815366    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6" event={"ID":"8391d48a-ac9c-433a-8dd3-f4854d9c85fa","Type":"ContainerStarted","Data":"0000346d01c83db767ef1eabaa94eeb17f5a27138ccb10e39be3911b284b359a"}
Sep 15 21:27:25.815791 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:25.815448    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6" event={"ID":"8391d48a-ac9c-433a-8dd3-f4854d9c85fa","Type":"ContainerStarted","Data":"bda3b4e1f7c8956b133c10af3103eff2fbe7c6516cca1680bb3efbe3184d7970"}
Sep 15 21:27:25.836218 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:25.836174    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6" podStartSLOduration=1.836160149 podStartE2EDuration="1.836160149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:25.834862435 +0000 UTC m=+1013.696796240" watchObservedRunningTime="2026-09-15 21:27:25.836160149 +0000 UTC m=+1013.698093954"
Sep 15 21:27:25.849393 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:25.849358    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:26.855106 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:26.855076    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:27.828284 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:27.828246    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:27.829149 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:27.829131    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:31.354260 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:31.354224    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"]
Sep 15 21:27:31.358454 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:31.358437    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"
Sep 15 21:27:31.360750 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:31.360730    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-026d11\""
Sep 15 21:27:31.360862 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:31.360800    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-026d11-dockercfg-5h7gf\""
Sep 15 21:27:31.368826 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:31.368805    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"]
Sep 15 21:27:31.547284 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:31.547254    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zjlr\" (UniqueName: \"kubernetes.io/projected/8ef7245c-7e26-49f3-9a0d-dee76cb676b4-kube-api-access-2zjlr\") pod \"payload-processing-e2e-explicit-026d11-584f886449-6fvtv\" (UID: \"8ef7245c-7e26-49f3-9a0d-dee76cb676b4\") " pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"
Sep 15 21:27:31.547494 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:31.547421    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ef7245c-7e26-49f3-9a0d-dee76cb676b4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-026d11-584f886449-6fvtv\" (UID: \"8ef7245c-7e26-49f3-9a0d-dee76cb676b4\") " pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"
Sep 15 21:27:31.648688 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:31.648571    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zjlr\" (UniqueName: \"kubernetes.io/projected/8ef7245c-7e26-49f3-9a0d-dee76cb676b4-kube-api-access-2zjlr\") pod \"payload-processing-e2e-explicit-026d11-584f886449-6fvtv\" (UID: \"8ef7245c-7e26-49f3-9a0d-dee76cb676b4\") " pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"
Sep 15 21:27:31.648851 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:31.648787    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ef7245c-7e26-49f3-9a0d-dee76cb676b4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-026d11-584f886449-6fvtv\" (UID: \"8ef7245c-7e26-49f3-9a0d-dee76cb676b4\") " pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"
Sep 15 21:27:31.649546 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:31.649520    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ef7245c-7e26-49f3-9a0d-dee76cb676b4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-026d11-584f886449-6fvtv\" (UID: \"8ef7245c-7e26-49f3-9a0d-dee76cb676b4\") " pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"
Sep 15 21:27:31.657628 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:31.657598    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zjlr\" (UniqueName: \"kubernetes.io/projected/8ef7245c-7e26-49f3-9a0d-dee76cb676b4-kube-api-access-2zjlr\") pod \"payload-processing-e2e-explicit-026d11-584f886449-6fvtv\" (UID: \"8ef7245c-7e26-49f3-9a0d-dee76cb676b4\") " pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"
Sep 15 21:27:31.671630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:31.671609    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"
Sep 15 21:27:31.947103 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:31.947065    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"]
Sep 15 21:27:31.948446 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:27:31.948416    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef7245c_7e26_49f3_9a0d_dee76cb676b4.slice/crio-cf2a450b845b604198ab5d963c1be8244e1fc21d93a36afea85e025bd362c3ca WatchSource:0}: Error finding container cf2a450b845b604198ab5d963c1be8244e1fc21d93a36afea85e025bd362c3ca: Status 404 returned error can't find the container with id cf2a450b845b604198ab5d963c1be8244e1fc21d93a36afea85e025bd362c3ca
Sep 15 21:27:32.865867 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:32.865836    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv" event={"ID":"8ef7245c-7e26-49f3-9a0d-dee76cb676b4","Type":"ContainerStarted","Data":"c65195c7ec449fbbf5a56182e0b368a0c7eb9a7601c1bfd649aca715c72b270d"}
Sep 15 21:27:32.866333 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:32.865891    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv" event={"ID":"8ef7245c-7e26-49f3-9a0d-dee76cb676b4","Type":"ContainerStarted","Data":"cf2a450b845b604198ab5d963c1be8244e1fc21d93a36afea85e025bd362c3ca"}
Sep 15 21:27:32.866333 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:32.865991    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"
Sep 15 21:27:32.884527 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:32.884440    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv" podStartSLOduration=1.884427091 podStartE2EDuration="1.884427091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:32.882443486 +0000 UTC m=+1020.744377292" watchObservedRunningTime="2026-09-15 21:27:32.884427091 +0000 UTC m=+1020.746360897"
Sep 15 21:27:43.879443 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:43.879408    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"
Sep 15 21:27:51.986546 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:51.986504    2584 generic.go:356] "Generic (PLEG): container finished" podID="8391d48a-ac9c-433a-8dd3-f4854d9c85fa" containerID="0000346d01c83db767ef1eabaa94eeb17f5a27138ccb10e39be3911b284b359a" exitCode=0
Sep 15 21:27:51.987080 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:51.986553    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6" event={"ID":"8391d48a-ac9c-433a-8dd3-f4854d9c85fa","Type":"ContainerDied","Data":"0000346d01c83db767ef1eabaa94eeb17f5a27138ccb10e39be3911b284b359a"}
Sep 15 21:27:51.987153 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:51.987118    2584 scope.go:117] "RemoveContainer" containerID="0000346d01c83db767ef1eabaa94eeb17f5a27138ccb10e39be3911b284b359a"
Sep 15 21:27:51.988659 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:51.988621    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:27:51.988777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:51.988714    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:27:51.988777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:51.988757    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:27:53.002291 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:53.002255    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6" event={"ID":"8391d48a-ac9c-433a-8dd3-f4854d9c85fa","Type":"ContainerStarted","Data":"9dee42adf5cf08cc5b9c5a640f0c5f6b67b8ff9e504c21cc2b325bba277e7318"}
Sep 15 21:27:53.849595 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:53.849559    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:53.850754 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:53.850719    2584 patch_prober.go:28] interesting pod/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 15 21:27:53.850894 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:53.850784    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6" podUID="8391d48a-ac9c-433a-8dd3-f4854d9c85fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 15 21:27:54.849340 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:54.849296    2584 patch_prober.go:28] interesting pod/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 15 21:27:54.849886 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:54.849362    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6" podUID="8391d48a-ac9c-433a-8dd3-f4854d9c85fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 15 21:27:55.849297 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:55.849250    2584 patch_prober.go:28] interesting pod/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 15 21:27:55.849623 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:55.849594    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6" podUID="8391d48a-ac9c-433a-8dd3-f4854d9c85fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 15 21:27:56.849323 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:56.849277    2584 patch_prober.go:28] interesting pod/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 15 21:27:56.849565 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:56.849344    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6" podUID="8391d48a-ac9c-433a-8dd3-f4854d9c85fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 15 21:27:57.828603 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:57.828564    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:57.849463 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:57.849435    2584 patch_prober.go:28] interesting pod/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 15 21:27:57.849683 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:57.849656    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6" podUID="8391d48a-ac9c-433a-8dd3-f4854d9c85fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 15 21:27:58.853701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:58.853673    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:27:58.854782 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:27:58.854764    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:28:13.905276 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:13.905242    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"]
Sep 15 21:28:13.909525 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:13.909504    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:13.911900 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:13.911877    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6f66f7\"/\"kube-root-ca.crt\""
Sep 15 21:28:13.912781 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:13.912763    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6f66f7\"/\"default-dockercfg-vlvng\""
Sep 15 21:28:13.912896 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:13.912795    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6f66f7\"/\"openshift-service-ca.crt\""
Sep 15 21:28:13.912976 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:13.912954    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6f66f7\"/\"premium-w4-6f66f7-kserve-self-signed-certs\""
Sep 15 21:28:13.919899 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:13.919880    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"]
Sep 15 21:28:13.999117 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:13.999071    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef0ed4d5-9eca-43a8-b817-18f79a35d213-tls-certs\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:13.999283 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:13.999126    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-home\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:13.999354 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:13.999274    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-dshm\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:13.999354 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:13.999318    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-model-cache\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:13.999354 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:13.999350    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-tmp-dir\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:13.999513 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:13.999422    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qmft\" (UniqueName: \"kubernetes.io/projected/ef0ed4d5-9eca-43a8-b817-18f79a35d213-kube-api-access-5qmft\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:14.100617 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:14.100574    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qmft\" (UniqueName: \"kubernetes.io/projected/ef0ed4d5-9eca-43a8-b817-18f79a35d213-kube-api-access-5qmft\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:14.100812 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:14.100772    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef0ed4d5-9eca-43a8-b817-18f79a35d213-tls-certs\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:14.100812 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:14.100809    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-home\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:14.100934 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:14.100869    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-dshm\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:14.100934 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:14.100896    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-model-cache\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:14.100934 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:14.100923    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-tmp-dir\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:14.101317 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:14.101291    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-model-cache\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:14.101445 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:14.101421    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-tmp-dir\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:14.101585 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:14.101564    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-home\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:14.103821 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:14.103792    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-dshm\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:14.103914 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:14.103843    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef0ed4d5-9eca-43a8-b817-18f79a35d213-tls-certs\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:14.114441 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:14.114415    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qmft\" (UniqueName: \"kubernetes.io/projected/ef0ed4d5-9eca-43a8-b817-18f79a35d213-kube-api-access-5qmft\") pod \"premium-w4-6f66f7-kserve-5bf58b849c-5pznt\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") " pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:14.220812 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:14.220730    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:14.493541 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:14.493482    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"]
Sep 15 21:28:14.496103 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:28:14.496069    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef0ed4d5_9eca_43a8_b817_18f79a35d213.slice/crio-1c36a8588644150aad0dbca611f76527215056404df7964cffa947f1e6b1d942 WatchSource:0}: Error finding container 1c36a8588644150aad0dbca611f76527215056404df7964cffa947f1e6b1d942: Status 404 returned error can't find the container with id 1c36a8588644150aad0dbca611f76527215056404df7964cffa947f1e6b1d942
Sep 15 21:28:15.135551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:15.135518    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt" event={"ID":"ef0ed4d5-9eca-43a8-b817-18f79a35d213","Type":"ContainerStarted","Data":"6e743941d0a1f495d5ffe2a03932adc03095fdf523a04eb00b3eebf51fb57781"}
Sep 15 21:28:15.135551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:15.135561    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt" event={"ID":"ef0ed4d5-9eca-43a8-b817-18f79a35d213","Type":"ContainerStarted","Data":"1c36a8588644150aad0dbca611f76527215056404df7964cffa947f1e6b1d942"}
Sep 15 21:28:15.154926 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:15.154878    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt" podStartSLOduration=2.154862506 podStartE2EDuration="2.154862506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:28:15.15287659 +0000 UTC m=+1063.014810398" watchObservedRunningTime="2026-09-15 21:28:15.154862506 +0000 UTC m=+1063.016796312"
Sep 15 21:28:23.062896 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:23.062865    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"]
Sep 15 21:28:23.066773 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:23.066752    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"
Sep 15 21:28:23.069104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:23.069078    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-af7b8de1\""
Sep 15 21:28:23.069104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:23.069091    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-af7b8de1-dockercfg-wzhnl\""
Sep 15 21:28:23.076261 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:23.076241    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"]
Sep 15 21:28:23.205451 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:23.205412    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrqpf\" (UniqueName: \"kubernetes.io/projected/806aab9e-cc58-4780-9bb2-d7382e5d298a-kube-api-access-wrqpf\") pod \"payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb\" (UID: \"806aab9e-cc58-4780-9bb2-d7382e5d298a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"
Sep 15 21:28:23.205761 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:23.205735    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/806aab9e-cc58-4780-9bb2-d7382e5d298a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb\" (UID: \"806aab9e-cc58-4780-9bb2-d7382e5d298a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"
Sep 15 21:28:23.307219 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:23.307179    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrqpf\" (UniqueName: \"kubernetes.io/projected/806aab9e-cc58-4780-9bb2-d7382e5d298a-kube-api-access-wrqpf\") pod \"payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb\" (UID: \"806aab9e-cc58-4780-9bb2-d7382e5d298a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"
Sep 15 21:28:23.307420 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:23.307246    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/806aab9e-cc58-4780-9bb2-d7382e5d298a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb\" (UID: \"806aab9e-cc58-4780-9bb2-d7382e5d298a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"
Sep 15 21:28:23.307918 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:23.307897    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/806aab9e-cc58-4780-9bb2-d7382e5d298a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb\" (UID: \"806aab9e-cc58-4780-9bb2-d7382e5d298a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"
Sep 15 21:28:23.315506 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:23.315449    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrqpf\" (UniqueName: \"kubernetes.io/projected/806aab9e-cc58-4780-9bb2-d7382e5d298a-kube-api-access-wrqpf\") pod \"payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb\" (UID: \"806aab9e-cc58-4780-9bb2-d7382e5d298a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"
Sep 15 21:28:23.378798 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:23.378759    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"
Sep 15 21:28:23.651789 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:23.651740    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"]
Sep 15 21:28:23.654796 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:28:23.654758    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806aab9e_cc58_4780_9bb2_d7382e5d298a.slice/crio-249d8dd3717951a418c461c5c226ad1261657f92996819db87e9045c1efa2faf WatchSource:0}: Error finding container 249d8dd3717951a418c461c5c226ad1261657f92996819db87e9045c1efa2faf: Status 404 returned error can't find the container with id 249d8dd3717951a418c461c5c226ad1261657f92996819db87e9045c1efa2faf
Sep 15 21:28:24.200973 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:24.200902    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb" event={"ID":"806aab9e-cc58-4780-9bb2-d7382e5d298a","Type":"ContainerStarted","Data":"31ae3950ca8b526261e63cb054ca4421894924b5b55a73d530a05163b35102b1"}
Sep 15 21:28:24.200973 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:24.200975    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb" event={"ID":"806aab9e-cc58-4780-9bb2-d7382e5d298a","Type":"ContainerStarted","Data":"249d8dd3717951a418c461c5c226ad1261657f92996819db87e9045c1efa2faf"}
Sep 15 21:28:24.201480 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:24.201101    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"
Sep 15 21:28:24.219253 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:24.219209    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb" podStartSLOduration=1.219194311 podStartE2EDuration="1.219194311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:28:24.216914394 +0000 UTC m=+1072.078848199" watchObservedRunningTime="2026-09-15 21:28:24.219194311 +0000 UTC m=+1072.081128116"
Sep 15 21:28:24.221742 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:24.221724    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:24.221805 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:24.221776    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:24.234570 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:24.234548    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:25.218545 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:25.218467    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:28:35.209912 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:35.209879    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"
Sep 15 21:28:45.305759 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.305726    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"]
Sep 15 21:28:45.309819 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.309801    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.312307 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.312283    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-026d11\"/\"kube-root-ca.crt\""
Sep 15 21:28:45.313043 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.313025    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-026d11\"/\"openshift-service-ca.crt\""
Sep 15 21:28:45.313164 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.313061    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-026d11\"/\"default-dockercfg-rwmxg\""
Sep 15 21:28:45.313164 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.313095    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-026d11\"/\"explicit-model-026d11-kserve-self-signed-certs\""
Sep 15 21:28:45.316950 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.316929    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"]
Sep 15 21:28:45.433330 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.433298    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gtrv\" (UniqueName: \"kubernetes.io/projected/45be76d9-38a9-4ff0-accd-135b47eb9e57-kube-api-access-4gtrv\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.433504 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.433361    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-home\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.433504 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.433404    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45be76d9-38a9-4ff0-accd-135b47eb9e57-tls-certs\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.433504 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.433442    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-tmp-dir\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.433616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.433517    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-model-cache\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.433616 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.433551    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-dshm\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.534539 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.534506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-tmp-dir\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.534714 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.534562    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-model-cache\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.534714 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.534586    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-dshm\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.534714 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.534633    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtrv\" (UniqueName: \"kubernetes.io/projected/45be76d9-38a9-4ff0-accd-135b47eb9e57-kube-api-access-4gtrv\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.534714 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.534693    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-home\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.534714 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.534713    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45be76d9-38a9-4ff0-accd-135b47eb9e57-tls-certs\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.534997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.534877    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-tmp-dir\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.534997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.534944    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-model-cache\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.535109 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.535067    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-home\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.536874 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.536853    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-dshm\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.537103 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.537085    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45be76d9-38a9-4ff0-accd-135b47eb9e57-tls-certs\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.545132 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.545110    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtrv\" (UniqueName: \"kubernetes.io/projected/45be76d9-38a9-4ff0-accd-135b47eb9e57-kube-api-access-4gtrv\") pod \"explicit-model-026d11-kserve-55bc5695d9-vnkzv\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") " pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.621298 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.621211    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:45.895427 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:45.895398    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"]
Sep 15 21:28:45.897564 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:28:45.897538    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45be76d9_38a9_4ff0_accd_135b47eb9e57.slice/crio-fe1db5a6e09cfcc25967a9f8e4f80acbcee78cbddbc4778f16e2f99930fca70c WatchSource:0}: Error finding container fe1db5a6e09cfcc25967a9f8e4f80acbcee78cbddbc4778f16e2f99930fca70c: Status 404 returned error can't find the container with id fe1db5a6e09cfcc25967a9f8e4f80acbcee78cbddbc4778f16e2f99930fca70c
Sep 15 21:28:46.334126 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:46.334093    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv" event={"ID":"45be76d9-38a9-4ff0-accd-135b47eb9e57","Type":"ContainerStarted","Data":"f1fbed81566a8ffb0be45ddd8c99ad30f4c8915c20316a0de45310642667706b"}
Sep 15 21:28:46.334126 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:46.334131    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv" event={"ID":"45be76d9-38a9-4ff0-accd-135b47eb9e57","Type":"ContainerStarted","Data":"fe1db5a6e09cfcc25967a9f8e4f80acbcee78cbddbc4778f16e2f99930fca70c"}
Sep 15 21:28:46.351040 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:46.350994    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv" podStartSLOduration=1.3509788409999999 podStartE2EDuration="1.350978841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:28:46.349571296 +0000 UTC m=+1094.211505102" watchObservedRunningTime="2026-09-15 21:28:46.350978841 +0000 UTC m=+1094.212912645"
Sep 15 21:28:55.622082 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:55.622045    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:55.622535 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:55.622106    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:55.634333 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:55.634302    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:55.996306 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:55.996231    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh"]
Sep 15 21:28:56.000429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:56.000411    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh"
Sep 15 21:28:56.002526 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:56.002505    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:28:56.002786 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:56.002764    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:28:56.006489 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:56.006463    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh"]
Sep 15 21:28:56.148040 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:56.148005    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f86b2cd-fa6e-4500-bf59-8250f7624604-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh\" (UID: \"5f86b2cd-fa6e-4500-bf59-8250f7624604\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh"
Sep 15 21:28:56.249048 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:56.248945    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f86b2cd-fa6e-4500-bf59-8250f7624604-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh\" (UID: \"5f86b2cd-fa6e-4500-bf59-8250f7624604\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh"
Sep 15 21:28:56.249796 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:56.249763    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f86b2cd-fa6e-4500-bf59-8250f7624604-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh\" (UID: \"5f86b2cd-fa6e-4500-bf59-8250f7624604\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh"
Sep 15 21:28:56.313639 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:56.313611    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh"
Sep 15 21:28:56.408292 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:56.408264    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:28:56.588504 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:56.588320    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh"]
Sep 15 21:28:56.591287 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:28:56.591259    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f86b2cd_fa6e_4500_bf59_8250f7624604.slice/crio-358eef59db36aa5fc7954aeeb4cfc548f6f05e6eda0d5b4c992bf84b946a16a1 WatchSource:0}: Error finding container 358eef59db36aa5fc7954aeeb4cfc548f6f05e6eda0d5b4c992bf84b946a16a1: Status 404 returned error can't find the container with id 358eef59db36aa5fc7954aeeb4cfc548f6f05e6eda0d5b4c992bf84b946a16a1
Sep 15 21:28:57.403146 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:57.403115    2584 generic.go:356] "Generic (PLEG): container finished" podID="5f86b2cd-fa6e-4500-bf59-8250f7624604" containerID="1301e734ada6e555a45b39ed39933560d78800ecb23e0526a9308cb017df85b0" exitCode=0
Sep 15 21:28:57.403502 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:57.403210    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh" event={"ID":"5f86b2cd-fa6e-4500-bf59-8250f7624604","Type":"ContainerDied","Data":"1301e734ada6e555a45b39ed39933560d78800ecb23e0526a9308cb017df85b0"}
Sep 15 21:28:57.403502 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:57.403263    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh" event={"ID":"5f86b2cd-fa6e-4500-bf59-8250f7624604","Type":"ContainerStarted","Data":"358eef59db36aa5fc7954aeeb4cfc548f6f05e6eda0d5b4c992bf84b946a16a1"}
Sep 15 21:28:58.569407 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:58.569351    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh"
Sep 15 21:28:58.673218 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:58.673127    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f86b2cd-fa6e-4500-bf59-8250f7624604-maas-api-service-ca\") pod \"5f86b2cd-fa6e-4500-bf59-8250f7624604\" (UID: \"5f86b2cd-fa6e-4500-bf59-8250f7624604\") "
Sep 15 21:28:58.673519 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:58.673495    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f86b2cd-fa6e-4500-bf59-8250f7624604-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f86b2cd-fa6e-4500-bf59-8250f7624604" (UID: "5f86b2cd-fa6e-4500-bf59-8250f7624604"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:28:58.774557 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:58.774519    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f86b2cd-fa6e-4500-bf59-8250f7624604-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:28:59.416311 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:59.416278    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh"
Sep 15 21:28:59.416541 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:59.416323    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh" event={"ID":"5f86b2cd-fa6e-4500-bf59-8250f7624604","Type":"ContainerDied","Data":"358eef59db36aa5fc7954aeeb4cfc548f6f05e6eda0d5b4c992bf84b946a16a1"}
Sep 15 21:28:59.416541 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:28:59.416418    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="358eef59db36aa5fc7954aeeb4cfc548f6f05e6eda0d5b4c992bf84b946a16a1"
Sep 15 21:29:06.509257 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:06.509222    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"]
Sep 15 21:29:06.509756 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:06.509504    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb" podUID="806aab9e-cc58-4780-9bb2-d7382e5d298a" containerName="payload-pre-processing" containerID="cri-o://31ae3950ca8b526261e63cb054ca4421894924b5b55a73d530a05163b35102b1" gracePeriod=30
Sep 15 21:29:06.686045 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:06.686010    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh"]
Sep 15 21:29:06.688540 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:06.688519    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af7b8de1-sdnrh"]
Sep 15 21:29:06.814300 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:06.814275    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"
Sep 15 21:29:06.962607 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:06.962577    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/806aab9e-cc58-4780-9bb2-d7382e5d298a-plugins-config-volume\") pod \"806aab9e-cc58-4780-9bb2-d7382e5d298a\" (UID: \"806aab9e-cc58-4780-9bb2-d7382e5d298a\") "
Sep 15 21:29:06.962798 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:06.962715    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrqpf\" (UniqueName: \"kubernetes.io/projected/806aab9e-cc58-4780-9bb2-d7382e5d298a-kube-api-access-wrqpf\") pod \"806aab9e-cc58-4780-9bb2-d7382e5d298a\" (UID: \"806aab9e-cc58-4780-9bb2-d7382e5d298a\") "
Sep 15 21:29:06.962992 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:06.962967    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806aab9e-cc58-4780-9bb2-d7382e5d298a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "806aab9e-cc58-4780-9bb2-d7382e5d298a" (UID: "806aab9e-cc58-4780-9bb2-d7382e5d298a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:29:06.964734 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:06.964712    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806aab9e-cc58-4780-9bb2-d7382e5d298a-kube-api-access-wrqpf" (OuterVolumeSpecName: "kube-api-access-wrqpf") pod "806aab9e-cc58-4780-9bb2-d7382e5d298a" (UID: "806aab9e-cc58-4780-9bb2-d7382e5d298a"). InnerVolumeSpecName "kube-api-access-wrqpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:29:07.063915 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:07.063827    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/806aab9e-cc58-4780-9bb2-d7382e5d298a-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:07.063915 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:07.063860    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrqpf\" (UniqueName: \"kubernetes.io/projected/806aab9e-cc58-4780-9bb2-d7382e5d298a-kube-api-access-wrqpf\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:07.470519 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:07.470435    2584 generic.go:356] "Generic (PLEG): container finished" podID="806aab9e-cc58-4780-9bb2-d7382e5d298a" containerID="31ae3950ca8b526261e63cb054ca4421894924b5b55a73d530a05163b35102b1" exitCode=0
Sep 15 21:29:07.470519 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:07.470478    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"
Sep 15 21:29:07.470735 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:07.470478    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb" event={"ID":"806aab9e-cc58-4780-9bb2-d7382e5d298a","Type":"ContainerDied","Data":"31ae3950ca8b526261e63cb054ca4421894924b5b55a73d530a05163b35102b1"}
Sep 15 21:29:07.470735 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:07.470604    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb" event={"ID":"806aab9e-cc58-4780-9bb2-d7382e5d298a","Type":"ContainerDied","Data":"249d8dd3717951a418c461c5c226ad1261657f92996819db87e9045c1efa2faf"}
Sep 15 21:29:07.470735 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:07.470628    2584 scope.go:117] "RemoveContainer" containerID="31ae3950ca8b526261e63cb054ca4421894924b5b55a73d530a05163b35102b1"
Sep 15 21:29:07.480899 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:07.480883    2584 scope.go:117] "RemoveContainer" containerID="31ae3950ca8b526261e63cb054ca4421894924b5b55a73d530a05163b35102b1"
Sep 15 21:29:07.481171 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:29:07.481154    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ae3950ca8b526261e63cb054ca4421894924b5b55a73d530a05163b35102b1\": container with ID starting with 31ae3950ca8b526261e63cb054ca4421894924b5b55a73d530a05163b35102b1 not found: ID does not exist" containerID="31ae3950ca8b526261e63cb054ca4421894924b5b55a73d530a05163b35102b1"
Sep 15 21:29:07.481208 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:07.481180    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ae3950ca8b526261e63cb054ca4421894924b5b55a73d530a05163b35102b1"} err="failed to get container status \"31ae3950ca8b526261e63cb054ca4421894924b5b55a73d530a05163b35102b1\": rpc error: code = NotFound desc = could not find container \"31ae3950ca8b526261e63cb054ca4421894924b5b55a73d530a05163b35102b1\": container with ID starting with 31ae3950ca8b526261e63cb054ca4421894924b5b55a73d530a05163b35102b1 not found: ID does not exist"
Sep 15 21:29:07.495500 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:07.495477    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"]
Sep 15 21:29:07.501595 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:07.501572    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-af7b8de1-6866c9b9f8-9n5gb"]
Sep 15 21:29:08.673216 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:08.673141    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f86b2cd-fa6e-4500-bf59-8250f7624604" path="/var/lib/kubelet/pods/5f86b2cd-fa6e-4500-bf59-8250f7624604/volumes"
Sep 15 21:29:08.673567 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:08.673540    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="806aab9e-cc58-4780-9bb2-d7382e5d298a" path="/var/lib/kubelet/pods/806aab9e-cc58-4780-9bb2-d7382e5d298a/volumes"
Sep 15 21:29:12.626225 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:12.626186    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"]
Sep 15 21:29:12.757586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:12.757546    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-026d11-r6hjt"]
Sep 15 21:29:12.758110 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:12.758090    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f86b2cd-fa6e-4500-bf59-8250f7624604" containerName="revoke-keys"
Sep 15 21:29:12.758222 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:12.758112    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f86b2cd-fa6e-4500-bf59-8250f7624604" containerName="revoke-keys"
Sep 15 21:29:12.758222 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:12.758127    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="806aab9e-cc58-4780-9bb2-d7382e5d298a" containerName="payload-pre-processing"
Sep 15 21:29:12.758222 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:12.758136    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="806aab9e-cc58-4780-9bb2-d7382e5d298a" containerName="payload-pre-processing"
Sep 15 21:29:12.758393 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:12.758275    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f86b2cd-fa6e-4500-bf59-8250f7624604" containerName="revoke-keys"
Sep 15 21:29:12.758393 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:12.758289    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="806aab9e-cc58-4780-9bb2-d7382e5d298a" containerName="payload-pre-processing"
Sep 15 21:29:12.763144 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:12.763124    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-026d11-r6hjt"
Sep 15 21:29:12.765668 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:12.765643    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:29:12.765942 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:12.765710    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:29:12.769117 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:12.769099    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-026d11-r6hjt"]
Sep 15 21:29:12.834217 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:12.834180    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca833fe-ff03-41a8-ba12-a3bc19298ec4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-026d11-r6hjt\" (UID: \"1ca833fe-ff03-41a8-ba12-a3bc19298ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-026d11-r6hjt"
Sep 15 21:29:12.935365 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:12.935279    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca833fe-ff03-41a8-ba12-a3bc19298ec4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-026d11-r6hjt\" (UID: \"1ca833fe-ff03-41a8-ba12-a3bc19298ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-026d11-r6hjt"
Sep 15 21:29:12.935933 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:12.935916    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca833fe-ff03-41a8-ba12-a3bc19298ec4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-026d11-r6hjt\" (UID: \"1ca833fe-ff03-41a8-ba12-a3bc19298ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-026d11-r6hjt"
Sep 15 21:29:13.077106 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:13.076861    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-026d11-r6hjt"
Sep 15 21:29:13.350858 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:13.350740    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-026d11-r6hjt"]
Sep 15 21:29:13.517515 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:13.517483    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-026d11-r6hjt" event={"ID":"1ca833fe-ff03-41a8-ba12-a3bc19298ec4","Type":"ContainerStarted","Data":"51b3846748a05bdc86180c808e595f2907a98c259e2a3e48f3ec184032c850ee"}
Sep 15 21:29:14.525870 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:14.525839    2584 generic.go:356] "Generic (PLEG): container finished" podID="1ca833fe-ff03-41a8-ba12-a3bc19298ec4" containerID="de5bfd393205cc5a8f009c3e2144a9eaea79096ca1a9514e38d16e82219f0321" exitCode=0
Sep 15 21:29:14.526242 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:14.525924    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-026d11-r6hjt" event={"ID":"1ca833fe-ff03-41a8-ba12-a3bc19298ec4","Type":"ContainerDied","Data":"de5bfd393205cc5a8f009c3e2144a9eaea79096ca1a9514e38d16e82219f0321"}
Sep 15 21:29:15.683393 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:15.683360    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-026d11-r6hjt"
Sep 15 21:29:15.767777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:15.767751    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca833fe-ff03-41a8-ba12-a3bc19298ec4-maas-api-service-ca\") pod \"1ca833fe-ff03-41a8-ba12-a3bc19298ec4\" (UID: \"1ca833fe-ff03-41a8-ba12-a3bc19298ec4\") "
Sep 15 21:29:15.768108 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:15.768067    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca833fe-ff03-41a8-ba12-a3bc19298ec4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1ca833fe-ff03-41a8-ba12-a3bc19298ec4" (UID: "1ca833fe-ff03-41a8-ba12-a3bc19298ec4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:29:15.768870 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:15.768848    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca833fe-ff03-41a8-ba12-a3bc19298ec4-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:16.544323 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:16.544289    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-026d11-r6hjt" event={"ID":"1ca833fe-ff03-41a8-ba12-a3bc19298ec4","Type":"ContainerDied","Data":"51b3846748a05bdc86180c808e595f2907a98c259e2a3e48f3ec184032c850ee"}
Sep 15 21:29:16.544607 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:16.544345    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51b3846748a05bdc86180c808e595f2907a98c259e2a3e48f3ec184032c850ee"
Sep 15 21:29:16.544607 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:16.544296    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-026d11-r6hjt"
Sep 15 21:29:24.382137 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:24.382103    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"]
Sep 15 21:29:24.382637 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:24.382337    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv" podUID="8ef7245c-7e26-49f3-9a0d-dee76cb676b4" containerName="payload-processing" containerID="cri-o://c65195c7ec449fbbf5a56182e0b368a0c7eb9a7601c1bfd649aca715c72b270d" gracePeriod=30
Sep 15 21:29:24.585259 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:24.585224    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-026d11-r6hjt"]
Sep 15 21:29:24.587547 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:24.587518    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-026d11-r6hjt"]
Sep 15 21:29:24.596741 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:24.596713    2584 generic.go:356] "Generic (PLEG): container finished" podID="8ef7245c-7e26-49f3-9a0d-dee76cb676b4" containerID="c65195c7ec449fbbf5a56182e0b368a0c7eb9a7601c1bfd649aca715c72b270d" exitCode=0
Sep 15 21:29:24.596869 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:24.596780    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv" event={"ID":"8ef7245c-7e26-49f3-9a0d-dee76cb676b4","Type":"ContainerDied","Data":"c65195c7ec449fbbf5a56182e0b368a0c7eb9a7601c1bfd649aca715c72b270d"}
Sep 15 21:29:24.661431 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:24.661406    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"
Sep 15 21:29:24.674077 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:24.674047    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ca833fe-ff03-41a8-ba12-a3bc19298ec4" path="/var/lib/kubelet/pods/1ca833fe-ff03-41a8-ba12-a3bc19298ec4/volumes"
Sep 15 21:29:24.766871 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:24.766825    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ef7245c-7e26-49f3-9a0d-dee76cb676b4-plugins-config-volume\") pod \"8ef7245c-7e26-49f3-9a0d-dee76cb676b4\" (UID: \"8ef7245c-7e26-49f3-9a0d-dee76cb676b4\") "
Sep 15 21:29:24.767077 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:24.766936    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zjlr\" (UniqueName: \"kubernetes.io/projected/8ef7245c-7e26-49f3-9a0d-dee76cb676b4-kube-api-access-2zjlr\") pod \"8ef7245c-7e26-49f3-9a0d-dee76cb676b4\" (UID: \"8ef7245c-7e26-49f3-9a0d-dee76cb676b4\") "
Sep 15 21:29:24.767937 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:24.767897    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef7245c-7e26-49f3-9a0d-dee76cb676b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ef7245c-7e26-49f3-9a0d-dee76cb676b4" (UID: "8ef7245c-7e26-49f3-9a0d-dee76cb676b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:29:24.770219 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:24.770113    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef7245c-7e26-49f3-9a0d-dee76cb676b4-kube-api-access-2zjlr" (OuterVolumeSpecName: "kube-api-access-2zjlr") pod "8ef7245c-7e26-49f3-9a0d-dee76cb676b4" (UID: "8ef7245c-7e26-49f3-9a0d-dee76cb676b4"). InnerVolumeSpecName "kube-api-access-2zjlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:29:24.868486 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:24.868456    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ef7245c-7e26-49f3-9a0d-dee76cb676b4-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:24.868486 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:24.868482    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zjlr\" (UniqueName: \"kubernetes.io/projected/8ef7245c-7e26-49f3-9a0d-dee76cb676b4-kube-api-access-2zjlr\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:25.603958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:25.603928    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"
Sep 15 21:29:25.604428 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:25.603980    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv" event={"ID":"8ef7245c-7e26-49f3-9a0d-dee76cb676b4","Type":"ContainerDied","Data":"cf2a450b845b604198ab5d963c1be8244e1fc21d93a36afea85e025bd362c3ca"}
Sep 15 21:29:25.604428 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:25.604039    2584 scope.go:117] "RemoveContainer" containerID="c65195c7ec449fbbf5a56182e0b368a0c7eb9a7601c1bfd649aca715c72b270d"
Sep 15 21:29:25.629391 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:25.629315    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"]
Sep 15 21:29:25.633538 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:25.633514    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-026d11-584f886449-6fvtv"]
Sep 15 21:29:26.674482 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:26.674455    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ef7245c-7e26-49f3-9a0d-dee76cb676b4" path="/var/lib/kubelet/pods/8ef7245c-7e26-49f3-9a0d-dee76cb676b4/volumes"
Sep 15 21:29:27.646071 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:27.646026    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv" podUID="45be76d9-38a9-4ff0-accd-135b47eb9e57" containerName="main" containerID="cri-o://f1fbed81566a8ffb0be45ddd8c99ad30f4c8915c20316a0de45310642667706b" gracePeriod=45
Sep 15 21:29:27.890204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:27.890178    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:29:28.002352 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.002270    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-model-cache\") pod \"45be76d9-38a9-4ff0-accd-135b47eb9e57\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") "
Sep 15 21:29:28.002352 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.002306    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-tmp-dir\") pod \"45be76d9-38a9-4ff0-accd-135b47eb9e57\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") "
Sep 15 21:29:28.002633 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.002452    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-dshm\") pod \"45be76d9-38a9-4ff0-accd-135b47eb9e57\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") "
Sep 15 21:29:28.002633 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.002484    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-home\") pod \"45be76d9-38a9-4ff0-accd-135b47eb9e57\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") "
Sep 15 21:29:28.002633 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.002522    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45be76d9-38a9-4ff0-accd-135b47eb9e57-tls-certs\") pod \"45be76d9-38a9-4ff0-accd-135b47eb9e57\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") "
Sep 15 21:29:28.002633 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.002560    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-model-cache" (OuterVolumeSpecName: "model-cache") pod "45be76d9-38a9-4ff0-accd-135b47eb9e57" (UID: "45be76d9-38a9-4ff0-accd-135b47eb9e57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:28.002633 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.002609    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gtrv\" (UniqueName: \"kubernetes.io/projected/45be76d9-38a9-4ff0-accd-135b47eb9e57-kube-api-access-4gtrv\") pod \"45be76d9-38a9-4ff0-accd-135b47eb9e57\" (UID: \"45be76d9-38a9-4ff0-accd-135b47eb9e57\") "
Sep 15 21:29:28.002863 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.002651    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45be76d9-38a9-4ff0-accd-135b47eb9e57" (UID: "45be76d9-38a9-4ff0-accd-135b47eb9e57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:28.002863 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.002754    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-home" (OuterVolumeSpecName: "home") pod "45be76d9-38a9-4ff0-accd-135b47eb9e57" (UID: "45be76d9-38a9-4ff0-accd-135b47eb9e57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:28.002972 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.002951    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-model-cache\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:28.002972 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.002970    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-tmp-dir\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:28.002972 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.002979    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-home\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:28.004714 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.004687    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-dshm" (OuterVolumeSpecName: "dshm") pod "45be76d9-38a9-4ff0-accd-135b47eb9e57" (UID: "45be76d9-38a9-4ff0-accd-135b47eb9e57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:28.004807 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.004742    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45be76d9-38a9-4ff0-accd-135b47eb9e57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45be76d9-38a9-4ff0-accd-135b47eb9e57" (UID: "45be76d9-38a9-4ff0-accd-135b47eb9e57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:29:28.004807 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.004787    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45be76d9-38a9-4ff0-accd-135b47eb9e57-kube-api-access-4gtrv" (OuterVolumeSpecName: "kube-api-access-4gtrv") pod "45be76d9-38a9-4ff0-accd-135b47eb9e57" (UID: "45be76d9-38a9-4ff0-accd-135b47eb9e57"). InnerVolumeSpecName "kube-api-access-4gtrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:29:28.103803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.103769    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45be76d9-38a9-4ff0-accd-135b47eb9e57-dshm\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:28.103803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.103796    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45be76d9-38a9-4ff0-accd-135b47eb9e57-tls-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:28.103803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.103808    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gtrv\" (UniqueName: \"kubernetes.io/projected/45be76d9-38a9-4ff0-accd-135b47eb9e57-kube-api-access-4gtrv\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:28.624510 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.624473    2584 generic.go:356] "Generic (PLEG): container finished" podID="45be76d9-38a9-4ff0-accd-135b47eb9e57" containerID="f1fbed81566a8ffb0be45ddd8c99ad30f4c8915c20316a0de45310642667706b" exitCode=0
Sep 15 21:29:28.624719 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.624523    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"
Sep 15 21:29:28.624719 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.624551    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv" event={"ID":"45be76d9-38a9-4ff0-accd-135b47eb9e57","Type":"ContainerDied","Data":"f1fbed81566a8ffb0be45ddd8c99ad30f4c8915c20316a0de45310642667706b"}
Sep 15 21:29:28.624719 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.624581    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv" event={"ID":"45be76d9-38a9-4ff0-accd-135b47eb9e57","Type":"ContainerDied","Data":"fe1db5a6e09cfcc25967a9f8e4f80acbcee78cbddbc4778f16e2f99930fca70c"}
Sep 15 21:29:28.624719 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.624605    2584 scope.go:117] "RemoveContainer" containerID="f1fbed81566a8ffb0be45ddd8c99ad30f4c8915c20316a0de45310642667706b"
Sep 15 21:29:28.634954 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.634934    2584 scope.go:117] "RemoveContainer" containerID="f1fbed81566a8ffb0be45ddd8c99ad30f4c8915c20316a0de45310642667706b"
Sep 15 21:29:28.635223 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:29:28.635204    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1fbed81566a8ffb0be45ddd8c99ad30f4c8915c20316a0de45310642667706b\": container with ID starting with f1fbed81566a8ffb0be45ddd8c99ad30f4c8915c20316a0de45310642667706b not found: ID does not exist" containerID="f1fbed81566a8ffb0be45ddd8c99ad30f4c8915c20316a0de45310642667706b"
Sep 15 21:29:28.635295 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.635231    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fbed81566a8ffb0be45ddd8c99ad30f4c8915c20316a0de45310642667706b"} err="failed to get container status \"f1fbed81566a8ffb0be45ddd8c99ad30f4c8915c20316a0de45310642667706b\": rpc error: code = NotFound desc = could not find container \"f1fbed81566a8ffb0be45ddd8c99ad30f4c8915c20316a0de45310642667706b\": container with ID starting with f1fbed81566a8ffb0be45ddd8c99ad30f4c8915c20316a0de45310642667706b not found: ID does not exist"
Sep 15 21:29:28.646118 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.646096    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"]
Sep 15 21:29:28.652278 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.652254    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-026d11/explicit-model-026d11-kserve-55bc5695d9-vnkzv"]
Sep 15 21:29:28.675765 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:28.675741    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45be76d9-38a9-4ff0-accd-135b47eb9e57" path="/var/lib/kubelet/pods/45be76d9-38a9-4ff0-accd-135b47eb9e57/volumes"
Sep 15 21:29:31.666321 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:31.666286    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"]
Sep 15 21:29:31.666806 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:31.666585    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6" podUID="8391d48a-ac9c-433a-8dd3-f4854d9c85fa" containerName="istio-proxy" containerID="cri-o://9dee42adf5cf08cc5b9c5a640f0c5f6b67b8ff9e504c21cc2b325bba277e7318" gracePeriod=30
Sep 15 21:29:35.672227 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.672189    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"]
Sep 15 21:29:35.672667 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.672651    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45be76d9-38a9-4ff0-accd-135b47eb9e57" containerName="main"
Sep 15 21:29:35.672667 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.672664    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="45be76d9-38a9-4ff0-accd-135b47eb9e57" containerName="main"
Sep 15 21:29:35.672753 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.672672    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca833fe-ff03-41a8-ba12-a3bc19298ec4" containerName="revoke-keys"
Sep 15 21:29:35.672753 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.672678    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca833fe-ff03-41a8-ba12-a3bc19298ec4" containerName="revoke-keys"
Sep 15 21:29:35.672753 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.672704    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ef7245c-7e26-49f3-9a0d-dee76cb676b4" containerName="payload-processing"
Sep 15 21:29:35.672753 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.672710    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ef7245c-7e26-49f3-9a0d-dee76cb676b4" containerName="payload-processing"
Sep 15 21:29:35.672898 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.672786    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ca833fe-ff03-41a8-ba12-a3bc19298ec4" containerName="revoke-keys"
Sep 15 21:29:35.672898 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.672795    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ef7245c-7e26-49f3-9a0d-dee76cb676b4" containerName="payload-processing"
Sep 15 21:29:35.672898 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.672803    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="45be76d9-38a9-4ff0-accd-135b47eb9e57" containerName="main"
Sep 15 21:29:35.677373 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.677355    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"
Sep 15 21:29:35.679801 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.679779    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-dbcc8f\""
Sep 15 21:29:35.679979 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.679874    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-dbcc8f-dockercfg-zjf9x\""
Sep 15 21:29:35.686681 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.686655    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"]
Sep 15 21:29:35.714227 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.714201    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"]
Sep 15 21:29:35.717787 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.717766    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"
Sep 15 21:29:35.726895 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.726875    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"]
Sep 15 21:29:35.786634 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.785942    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qlw\" (UniqueName: \"kubernetes.io/projected/1986b9b7-1c49-455a-b608-39889c14d670-kube-api-access-t2qlw\") pod \"payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627\" (UID: \"1986b9b7-1c49-455a-b608-39889c14d670\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"
Sep 15 21:29:35.786634 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.786155    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1986b9b7-1c49-455a-b608-39889c14d670-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627\" (UID: \"1986b9b7-1c49-455a-b608-39889c14d670\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"
Sep 15 21:29:35.887353 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.887321    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qlw\" (UniqueName: \"kubernetes.io/projected/1986b9b7-1c49-455a-b608-39889c14d670-kube-api-access-t2qlw\") pod \"payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627\" (UID: \"1986b9b7-1c49-455a-b608-39889c14d670\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"
Sep 15 21:29:35.887544 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.887413    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrkm4\" (UniqueName: \"kubernetes.io/projected/915666e3-94e8-4820-ba19-b25b136f686b-kube-api-access-xrkm4\") pod \"payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2\" (UID: \"915666e3-94e8-4820-ba19-b25b136f686b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"
Sep 15 21:29:35.887544 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.887453    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1986b9b7-1c49-455a-b608-39889c14d670-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627\" (UID: \"1986b9b7-1c49-455a-b608-39889c14d670\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"
Sep 15 21:29:35.887544 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.887517    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915666e3-94e8-4820-ba19-b25b136f686b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2\" (UID: \"915666e3-94e8-4820-ba19-b25b136f686b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"
Sep 15 21:29:35.888090 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.888072    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1986b9b7-1c49-455a-b608-39889c14d670-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627\" (UID: \"1986b9b7-1c49-455a-b608-39889c14d670\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"
Sep 15 21:29:35.895522 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.895498    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qlw\" (UniqueName: \"kubernetes.io/projected/1986b9b7-1c49-455a-b608-39889c14d670-kube-api-access-t2qlw\") pod \"payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627\" (UID: \"1986b9b7-1c49-455a-b608-39889c14d670\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"
Sep 15 21:29:35.988670 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.988573    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915666e3-94e8-4820-ba19-b25b136f686b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2\" (UID: \"915666e3-94e8-4820-ba19-b25b136f686b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"
Sep 15 21:29:35.988831 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.988681    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrkm4\" (UniqueName: \"kubernetes.io/projected/915666e3-94e8-4820-ba19-b25b136f686b-kube-api-access-xrkm4\") pod \"payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2\" (UID: \"915666e3-94e8-4820-ba19-b25b136f686b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"
Sep 15 21:29:35.988831 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.988688    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"
Sep 15 21:29:35.989314 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.989298    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915666e3-94e8-4820-ba19-b25b136f686b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2\" (UID: \"915666e3-94e8-4820-ba19-b25b136f686b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"
Sep 15 21:29:35.996797 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:35.996768    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrkm4\" (UniqueName: \"kubernetes.io/projected/915666e3-94e8-4820-ba19-b25b136f686b-kube-api-access-xrkm4\") pod \"payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2\" (UID: \"915666e3-94e8-4820-ba19-b25b136f686b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"
Sep 15 21:29:36.028689 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:36.028650    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"
Sep 15 21:29:36.263782 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:36.263753    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"]
Sep 15 21:29:36.266032 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:29:36.266008    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1986b9b7_1c49_455a_b608_39889c14d670.slice/crio-641c5eab1e56dd784a7189a2f28be8f2f95c1b1aa8f01c115ddd540993cef99d WatchSource:0}: Error finding container 641c5eab1e56dd784a7189a2f28be8f2f95c1b1aa8f01c115ddd540993cef99d: Status 404 returned error can't find the container with id 641c5eab1e56dd784a7189a2f28be8f2f95c1b1aa8f01c115ddd540993cef99d
Sep 15 21:29:36.313796 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:36.313773    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"]
Sep 15 21:29:36.316076 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:29:36.316051    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915666e3_94e8_4820_ba19_b25b136f686b.slice/crio-52678bcb68a606a11efb70d96a74e92037b161510d2295f2c1175302b836eca4 WatchSource:0}: Error finding container 52678bcb68a606a11efb70d96a74e92037b161510d2295f2c1175302b836eca4: Status 404 returned error can't find the container with id 52678bcb68a606a11efb70d96a74e92037b161510d2295f2c1175302b836eca4
Sep 15 21:29:36.710344 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:36.710310    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627" event={"ID":"1986b9b7-1c49-455a-b608-39889c14d670","Type":"ContainerStarted","Data":"a06ed5fc48ab806b83098681d30f644ce8c7f47b2dabc4beed531295822f4c22"}
Sep 15 21:29:36.711019 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:36.710389    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"
Sep 15 21:29:36.711019 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:36.710414    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627" event={"ID":"1986b9b7-1c49-455a-b608-39889c14d670","Type":"ContainerStarted","Data":"641c5eab1e56dd784a7189a2f28be8f2f95c1b1aa8f01c115ddd540993cef99d"}
Sep 15 21:29:36.712187 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:36.712159    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2" event={"ID":"915666e3-94e8-4820-ba19-b25b136f686b","Type":"ContainerStarted","Data":"d7ce632f8770208e0ed41170f1aedc66b08449090436d806e59c029e7bc44e9e"}
Sep 15 21:29:36.712289 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:36.712209    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2" event={"ID":"915666e3-94e8-4820-ba19-b25b136f686b","Type":"ContainerStarted","Data":"52678bcb68a606a11efb70d96a74e92037b161510d2295f2c1175302b836eca4"}
Sep 15 21:29:36.712289 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:36.712238    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"
Sep 15 21:29:36.728062 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:36.728013    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627" podStartSLOduration=1.727997807 podStartE2EDuration="1.727997807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:36.725829686 +0000 UTC m=+1144.587763495" watchObservedRunningTime="2026-09-15 21:29:36.727997807 +0000 UTC m=+1144.589931611"
Sep 15 21:29:36.742587 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:36.742547    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2" podStartSLOduration=1.742531567 podStartE2EDuration="1.742531567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:36.74120101 +0000 UTC m=+1144.603134817" watchObservedRunningTime="2026-09-15 21:29:36.742531567 +0000 UTC m=+1144.604465372"
Sep 15 21:29:36.939546 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:36.939490    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:29:37.103206 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.103169    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-credential-socket\") pod \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") "
Sep 15 21:29:37.103408 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.103233    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-workload-socket\") pod \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") "
Sep 15 21:29:37.103408 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.103290    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-token\") pod \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") "
Sep 15 21:29:37.103408 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.103327    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-data\") pod \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") "
Sep 15 21:29:37.103750 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.103364    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd65c\" (UniqueName: \"kubernetes.io/projected/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-kube-api-access-rd65c\") pod \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") "
Sep 15 21:29:37.103800 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.103784    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istiod-ca-cert\") pod \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") "
Sep 15 21:29:37.103846 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.103497    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8391d48a-ac9c-433a-8dd3-f4854d9c85fa" (UID: "8391d48a-ac9c-433a-8dd3-f4854d9c85fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:37.103893 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.103537    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8391d48a-ac9c-433a-8dd3-f4854d9c85fa" (UID: "8391d48a-ac9c-433a-8dd3-f4854d9c85fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:37.103893 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.103621    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "8391d48a-ac9c-433a-8dd3-f4854d9c85fa" (UID: "8391d48a-ac9c-433a-8dd3-f4854d9c85fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:37.103997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.103905    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-workload-certs\") pod \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") "
Sep 15 21:29:37.103997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.103940    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-podinfo\") pod \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") "
Sep 15 21:29:37.103997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.103966    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-envoy\") pod \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\" (UID: \"8391d48a-ac9c-433a-8dd3-f4854d9c85fa\") "
Sep 15 21:29:37.104160 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.104090    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8391d48a-ac9c-433a-8dd3-f4854d9c85fa" (UID: "8391d48a-ac9c-433a-8dd3-f4854d9c85fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:29:37.104160 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.104139    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8391d48a-ac9c-433a-8dd3-f4854d9c85fa" (UID: "8391d48a-ac9c-433a-8dd3-f4854d9c85fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:37.104418 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.104373    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-workload-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:37.104523 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.104424    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-credential-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:37.104523 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.104439    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-workload-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:37.104523 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.104453    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-data\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:37.104523 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.104466    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istiod-ca-cert\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:37.105694 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.105656    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "8391d48a-ac9c-433a-8dd3-f4854d9c85fa" (UID: "8391d48a-ac9c-433a-8dd3-f4854d9c85fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:29:37.105907 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.105876    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-kube-api-access-rd65c" (OuterVolumeSpecName: "kube-api-access-rd65c") pod "8391d48a-ac9c-433a-8dd3-f4854d9c85fa" (UID: "8391d48a-ac9c-433a-8dd3-f4854d9c85fa"). InnerVolumeSpecName "kube-api-access-rd65c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:29:37.106036 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.106021    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8391d48a-ac9c-433a-8dd3-f4854d9c85fa" (UID: "8391d48a-ac9c-433a-8dd3-f4854d9c85fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:29:37.106211 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.106195    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8391d48a-ac9c-433a-8dd3-f4854d9c85fa" (UID: "8391d48a-ac9c-433a-8dd3-f4854d9c85fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:37.205819 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.205735    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd65c\" (UniqueName: \"kubernetes.io/projected/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-kube-api-access-rd65c\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:37.205819 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.205764    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-podinfo\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:37.205819 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.205776    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-envoy\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:37.205819 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.205789    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8391d48a-ac9c-433a-8dd3-f4854d9c85fa-istio-token\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:29:37.720715 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.720683    2584 generic.go:356] "Generic (PLEG): container finished" podID="8391d48a-ac9c-433a-8dd3-f4854d9c85fa" containerID="9dee42adf5cf08cc5b9c5a640f0c5f6b67b8ff9e504c21cc2b325bba277e7318" exitCode=0
Sep 15 21:29:37.721184 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.720739    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"
Sep 15 21:29:37.721184 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.720770    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6" event={"ID":"8391d48a-ac9c-433a-8dd3-f4854d9c85fa","Type":"ContainerDied","Data":"9dee42adf5cf08cc5b9c5a640f0c5f6b67b8ff9e504c21cc2b325bba277e7318"}
Sep 15 21:29:37.721184 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.720836    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6" event={"ID":"8391d48a-ac9c-433a-8dd3-f4854d9c85fa","Type":"ContainerDied","Data":"bda3b4e1f7c8956b133c10af3103eff2fbe7c6516cca1680bb3efbe3184d7970"}
Sep 15 21:29:37.721184 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.720870    2584 scope.go:117] "RemoveContainer" containerID="9dee42adf5cf08cc5b9c5a640f0c5f6b67b8ff9e504c21cc2b325bba277e7318"
Sep 15 21:29:37.730854 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.730839    2584 scope.go:117] "RemoveContainer" containerID="0000346d01c83db767ef1eabaa94eeb17f5a27138ccb10e39be3911b284b359a"
Sep 15 21:29:37.739168 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.739152    2584 scope.go:117] "RemoveContainer" containerID="9dee42adf5cf08cc5b9c5a640f0c5f6b67b8ff9e504c21cc2b325bba277e7318"
Sep 15 21:29:37.739467 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:29:37.739447    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dee42adf5cf08cc5b9c5a640f0c5f6b67b8ff9e504c21cc2b325bba277e7318\": container with ID starting with 9dee42adf5cf08cc5b9c5a640f0c5f6b67b8ff9e504c21cc2b325bba277e7318 not found: ID does not exist" containerID="9dee42adf5cf08cc5b9c5a640f0c5f6b67b8ff9e504c21cc2b325bba277e7318"
Sep 15 21:29:37.739515 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.739476    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dee42adf5cf08cc5b9c5a640f0c5f6b67b8ff9e504c21cc2b325bba277e7318"} err="failed to get container status \"9dee42adf5cf08cc5b9c5a640f0c5f6b67b8ff9e504c21cc2b325bba277e7318\": rpc error: code = NotFound desc = could not find container \"9dee42adf5cf08cc5b9c5a640f0c5f6b67b8ff9e504c21cc2b325bba277e7318\": container with ID starting with 9dee42adf5cf08cc5b9c5a640f0c5f6b67b8ff9e504c21cc2b325bba277e7318 not found: ID does not exist"
Sep 15 21:29:37.739515 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.739491    2584 scope.go:117] "RemoveContainer" containerID="0000346d01c83db767ef1eabaa94eeb17f5a27138ccb10e39be3911b284b359a"
Sep 15 21:29:37.739732 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:29:37.739710    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0000346d01c83db767ef1eabaa94eeb17f5a27138ccb10e39be3911b284b359a\": container with ID starting with 0000346d01c83db767ef1eabaa94eeb17f5a27138ccb10e39be3911b284b359a not found: ID does not exist" containerID="0000346d01c83db767ef1eabaa94eeb17f5a27138ccb10e39be3911b284b359a"
Sep 15 21:29:37.739780 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.739741    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0000346d01c83db767ef1eabaa94eeb17f5a27138ccb10e39be3911b284b359a"} err="failed to get container status \"0000346d01c83db767ef1eabaa94eeb17f5a27138ccb10e39be3911b284b359a\": rpc error: code = NotFound desc = could not find container \"0000346d01c83db767ef1eabaa94eeb17f5a27138ccb10e39be3911b284b359a\": container with ID starting with 0000346d01c83db767ef1eabaa94eeb17f5a27138ccb10e39be3911b284b359a not found: ID does not exist"
Sep 15 21:29:37.745075 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.745051    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"]
Sep 15 21:29:37.748336 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:37.748314    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-026d11-openshift-default-bd59f8848-b7rl6"]
Sep 15 21:29:38.679672 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:38.679637    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8391d48a-ac9c-433a-8dd3-f4854d9c85fa" path="/var/lib/kubelet/pods/8391d48a-ac9c-433a-8dd3-f4854d9c85fa/volumes"
Sep 15 21:29:47.723583 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:47.723550    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"
Sep 15 21:29:47.724046 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:47.723639    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"
Sep 15 21:29:49.592474 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.592443    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"]
Sep 15 21:29:49.592916 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.592908    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8391d48a-ac9c-433a-8dd3-f4854d9c85fa" containerName="istio-proxy"
Sep 15 21:29:49.592961 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.592920    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8391d48a-ac9c-433a-8dd3-f4854d9c85fa" containerName="istio-proxy"
Sep 15 21:29:49.592961 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.592929    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8391d48a-ac9c-433a-8dd3-f4854d9c85fa" containerName="istio-proxy"
Sep 15 21:29:49.592961 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.592935    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8391d48a-ac9c-433a-8dd3-f4854d9c85fa" containerName="istio-proxy"
Sep 15 21:29:49.593055 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.593033    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8391d48a-ac9c-433a-8dd3-f4854d9c85fa" containerName="istio-proxy"
Sep 15 21:29:49.597980 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.597963    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.600217 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.600192    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6f66f7\"/\"e2e-conflict-a-8487c1aa-kserve-self-signed-certs\""
Sep 15 21:29:49.608193 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.608167    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"]
Sep 15 21:29:49.627747 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.627718    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8790c434-1fdc-4dd5-ab28-a803bbc72722-tls-certs\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.627868 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.627753    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpl42\" (UniqueName: \"kubernetes.io/projected/8790c434-1fdc-4dd5-ab28-a803bbc72722-kube-api-access-cpl42\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.627868 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.627782    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-tmp-dir\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.627868 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.627835    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-home\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.627984 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.627913    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-dshm\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.627984 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.627950    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-model-cache\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.729276 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.729243    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-dshm\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.729276 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.729288    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-model-cache\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.729553 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.729373    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8790c434-1fdc-4dd5-ab28-a803bbc72722-tls-certs\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.729553 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.729427    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpl42\" (UniqueName: \"kubernetes.io/projected/8790c434-1fdc-4dd5-ab28-a803bbc72722-kube-api-access-cpl42\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.729553 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.729455    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-tmp-dir\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.729553 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.729478    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-home\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.729798 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.729770    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-model-cache\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.729864 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.729810    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-home\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.729920 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.729891    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-tmp-dir\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.732087 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.732058    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-dshm\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.732312 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.732293    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8790c434-1fdc-4dd5-ab28-a803bbc72722-tls-certs\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.738092 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.738070    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpl42\" (UniqueName: \"kubernetes.io/projected/8790c434-1fdc-4dd5-ab28-a803bbc72722-kube-api-access-cpl42\") pod \"e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") " pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:49.914928 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:49.914825    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:50.193245 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:50.193222    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"]
Sep 15 21:29:50.195274 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:29:50.195242    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8790c434_1fdc_4dd5_ab28_a803bbc72722.slice/crio-72e0ec87ab6ac2131cea32baecfc4294c488ec27d4da681917e67f26ab539daf WatchSource:0}: Error finding container 72e0ec87ab6ac2131cea32baecfc4294c488ec27d4da681917e67f26ab539daf: Status 404 returned error can't find the container with id 72e0ec87ab6ac2131cea32baecfc4294c488ec27d4da681917e67f26ab539daf
Sep 15 21:29:50.803271 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:50.803235    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg" event={"ID":"8790c434-1fdc-4dd5-ab28-a803bbc72722","Type":"ContainerStarted","Data":"5dee47066eefa30a57e4191865fd99cf2ec7314d4924a15916799b0f4347c736"}
Sep 15 21:29:50.803271 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:50.803282    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg" event={"ID":"8790c434-1fdc-4dd5-ab28-a803bbc72722","Type":"ContainerStarted","Data":"72e0ec87ab6ac2131cea32baecfc4294c488ec27d4da681917e67f26ab539daf"}
Sep 15 21:29:50.820594 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:50.820545    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg" podStartSLOduration=1.820529896 podStartE2EDuration="1.820529896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:50.81933514 +0000 UTC m=+1158.681268944" watchObservedRunningTime="2026-09-15 21:29:50.820529896 +0000 UTC m=+1158.682463713"
Sep 15 21:29:59.915122 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:59.915036    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:59.915632 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:59.915121    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:29:59.928045 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:29:59.928022    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:30:00.141179 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.141139    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cfc1f4-29825130-p2zng"]
Sep 15 21:30:00.141905 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.141877    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8391d48a-ac9c-433a-8dd3-f4854d9c85fa" containerName="istio-proxy"
Sep 15 21:30:00.145474 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.145456    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f457e5-29825130-swpnv"]
Sep 15 21:30:00.145591 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.145572    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cfc1f4-29825130-p2zng"
Sep 15 21:30:00.147639 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.147618    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:30:00.148944 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.148924    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-318c22-29825130-mq6kx"]
Sep 15 21:30:00.149199 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.149073    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f457e5-29825130-swpnv"
Sep 15 21:30:00.153023 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.153004    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-318c22-29825130-mq6kx"
Sep 15 21:30:00.156972 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.156945    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cfc1f4-29825130-p2zng"]
Sep 15 21:30:00.159522 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.159506    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cfc1f4-29825130-p2zng"
Sep 15 21:30:00.160291 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.160094    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f457e5-29825130-swpnv"]
Sep 15 21:30:00.171411 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.171392    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f457e5-29825130-swpnv"
Sep 15 21:30:00.172802 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.172784    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6f66f7-29825130-bwltj"]
Sep 15 21:30:00.187493 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.187441    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-318c22-29825130-mq6kx"]
Sep 15 21:30:00.187602 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.187556    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6f66f7-29825130-bwltj"
Sep 15 21:30:00.190022 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.189940    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6f66f7-29825130-bwltj"]
Sep 15 21:30:00.222973 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.222137    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-318c22-29825130-mq6kx"
Sep 15 21:30:00.283165 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.283037    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6f66f7-29825130-bwltj"
Sep 15 21:30:00.504653 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.504577    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cfc1f4-29825130-p2zng"]
Sep 15 21:30:00.507426 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:30:00.507266    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab72d13_937b_4185_ab45_3d5fe40af101.slice/crio-00357f3dc2775916d27c03bb0a45b4f6635764d3dba583ff8fd4248462a894d2 WatchSource:0}: Error finding container 00357f3dc2775916d27c03bb0a45b4f6635764d3dba583ff8fd4248462a894d2: Status 404 returned error can't find the container with id 00357f3dc2775916d27c03bb0a45b4f6635764d3dba583ff8fd4248462a894d2
Sep 15 21:30:00.522585 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.522565    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f457e5-29825130-swpnv"]
Sep 15 21:30:00.524240 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:30:00.524213    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0fafef0_e8a7_4dab_94a6_5efb31d53120.slice/crio-e1ba9f2eee5a445d07dcbf97a497c24601514023115ede350299b2ac382e9513 WatchSource:0}: Error finding container e1ba9f2eee5a445d07dcbf97a497c24601514023115ede350299b2ac382e9513: Status 404 returned error can't find the container with id e1ba9f2eee5a445d07dcbf97a497c24601514023115ede350299b2ac382e9513
Sep 15 21:30:00.583300 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.583278    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-318c22-29825130-mq6kx"]
Sep 15 21:30:00.584509 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:30:00.584479    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85ba6cd_6d34_4e45_9c9b_6cc53fa172bd.slice/crio-d1078c774237a2441b124252ae8d855cabc8075175482939441df088500eba96 WatchSource:0}: Error finding container d1078c774237a2441b124252ae8d855cabc8075175482939441df088500eba96: Status 404 returned error can't find the container with id d1078c774237a2441b124252ae8d855cabc8075175482939441df088500eba96
Sep 15 21:30:00.590178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.590156    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6f66f7-29825130-bwltj"]
Sep 15 21:30:00.591370 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:30:00.591349    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35834631_72e7_45fc_b0c0_7daf6f188eb5.slice/crio-dfa107ed437ac79ef26292ad86addb7b61d7881c3dd38f1b66638b4c69ae9dca WatchSource:0}: Error finding container dfa107ed437ac79ef26292ad86addb7b61d7881c3dd38f1b66638b4c69ae9dca: Status 404 returned error can't find the container with id dfa107ed437ac79ef26292ad86addb7b61d7881c3dd38f1b66638b4c69ae9dca
Sep 15 21:30:00.865130 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.865100    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f457e5-29825130-swpnv" event={"ID":"e0fafef0-e8a7-4dab-94a6-5efb31d53120","Type":"ContainerStarted","Data":"e1ba9f2eee5a445d07dcbf97a497c24601514023115ede350299b2ac382e9513"}
Sep 15 21:30:00.866831 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.866806    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cfc1f4-29825130-p2zng" event={"ID":"9ab72d13-937b-4185-ab45-3d5fe40af101","Type":"ContainerStarted","Data":"c27e01e6b9789de7f1f930df8d380781de597edbb229a373e71b3af556ee754d"}
Sep 15 21:30:00.866928 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.866858    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cfc1f4-29825130-p2zng" event={"ID":"9ab72d13-937b-4185-ab45-3d5fe40af101","Type":"ContainerStarted","Data":"00357f3dc2775916d27c03bb0a45b4f6635764d3dba583ff8fd4248462a894d2"}
Sep 15 21:30:00.868117 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.868095    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6f66f7-29825130-bwltj" event={"ID":"35834631-72e7-45fc-b0c0-7daf6f188eb5","Type":"ContainerStarted","Data":"dfa107ed437ac79ef26292ad86addb7b61d7881c3dd38f1b66638b4c69ae9dca"}
Sep 15 21:30:00.869603 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.869580    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-318c22-29825130-mq6kx" event={"ID":"a85ba6cd-6d34-4e45-9c9b-6cc53fa172bd","Type":"ContainerStarted","Data":"d1078c774237a2441b124252ae8d855cabc8075175482939441df088500eba96"}
Sep 15 21:30:00.881661 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.881640    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:30:00.882643 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.882608    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cfc1f4-29825130-p2zng" podStartSLOduration=0.882596776 podStartE2EDuration="882.596776ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:00.881441141 +0000 UTC m=+1168.743374952" watchObservedRunningTime="2026-09-15 21:30:00.882596776 +0000 UTC m=+1168.744530580"
Sep 15 21:30:00.895222 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:00.895186    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-318c22-29825130-mq6kx" podStartSLOduration=0.895174481 podStartE2EDuration="895.174481ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:00.893603122 +0000 UTC m=+1168.755536928" watchObservedRunningTime="2026-09-15 21:30:00.895174481 +0000 UTC m=+1168.757108285"
Sep 15 21:30:01.877654 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:01.877609    2584 generic.go:356] "Generic (PLEG): container finished" podID="e0fafef0-e8a7-4dab-94a6-5efb31d53120" containerID="234b44fe70f8dca6e5dbb2db8d216d5145066a5537a2a51bb0d54f53d650c3ee" exitCode=0
Sep 15 21:30:01.878264 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:01.877720    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f457e5-29825130-swpnv" event={"ID":"e0fafef0-e8a7-4dab-94a6-5efb31d53120","Type":"ContainerDied","Data":"234b44fe70f8dca6e5dbb2db8d216d5145066a5537a2a51bb0d54f53d650c3ee"}
Sep 15 21:30:01.881168 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:01.880983    2584 generic.go:356] "Generic (PLEG): container finished" podID="9ab72d13-937b-4185-ab45-3d5fe40af101" containerID="c27e01e6b9789de7f1f930df8d380781de597edbb229a373e71b3af556ee754d" exitCode=0
Sep 15 21:30:01.881168 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:01.881020    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cfc1f4-29825130-p2zng" event={"ID":"9ab72d13-937b-4185-ab45-3d5fe40af101","Type":"ContainerDied","Data":"c27e01e6b9789de7f1f930df8d380781de597edbb229a373e71b3af556ee754d"}
Sep 15 21:30:01.883050 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:01.883028    2584 generic.go:356] "Generic (PLEG): container finished" podID="35834631-72e7-45fc-b0c0-7daf6f188eb5" containerID="b787c31a5ebf7f0e7fed1750524ba6f42d7a5e02dd285c34adf60dd738765aee" exitCode=0
Sep 15 21:30:01.883163 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:01.883130    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6f66f7-29825130-bwltj" event={"ID":"35834631-72e7-45fc-b0c0-7daf6f188eb5","Type":"ContainerDied","Data":"b787c31a5ebf7f0e7fed1750524ba6f42d7a5e02dd285c34adf60dd738765aee"}
Sep 15 21:30:01.884724 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:01.884701    2584 generic.go:356] "Generic (PLEG): container finished" podID="a85ba6cd-6d34-4e45-9c9b-6cc53fa172bd" containerID="15064fa418c6677045947afcc0a28b249126398357ad950fe5f0ba8365f06aed" exitCode=0
Sep 15 21:30:01.884818 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:01.884785    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-318c22-29825130-mq6kx" event={"ID":"a85ba6cd-6d34-4e45-9c9b-6cc53fa172bd","Type":"ContainerDied","Data":"15064fa418c6677045947afcc0a28b249126398357ad950fe5f0ba8365f06aed"}
Sep 15 21:30:03.074402 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:03.074343    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cfc1f4-29825130-p2zng"
Sep 15 21:30:03.135218 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:03.135191    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6f66f7-29825130-bwltj"
Sep 15 21:30:03.138467 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:03.138446    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f457e5-29825130-swpnv"
Sep 15 21:30:03.141553 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:03.141537    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-318c22-29825130-mq6kx"
Sep 15 21:30:03.901255 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:03.901230    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f457e5-29825130-swpnv"
Sep 15 21:30:03.901255 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:03.901242    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f457e5-29825130-swpnv" event={"ID":"e0fafef0-e8a7-4dab-94a6-5efb31d53120","Type":"ContainerDied","Data":"e1ba9f2eee5a445d07dcbf97a497c24601514023115ede350299b2ac382e9513"}
Sep 15 21:30:03.901483 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:03.901296    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1ba9f2eee5a445d07dcbf97a497c24601514023115ede350299b2ac382e9513"
Sep 15 21:30:03.902666 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:03.902638    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cfc1f4-29825130-p2zng"
Sep 15 21:30:03.902793 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:03.902680    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cfc1f4-29825130-p2zng" event={"ID":"9ab72d13-937b-4185-ab45-3d5fe40af101","Type":"ContainerDied","Data":"00357f3dc2775916d27c03bb0a45b4f6635764d3dba583ff8fd4248462a894d2"}
Sep 15 21:30:03.902793 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:03.902736    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00357f3dc2775916d27c03bb0a45b4f6635764d3dba583ff8fd4248462a894d2"
Sep 15 21:30:03.904180 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:03.904150    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6f66f7-29825130-bwltj"
Sep 15 21:30:03.904308 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:03.904176    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6f66f7-29825130-bwltj" event={"ID":"35834631-72e7-45fc-b0c0-7daf6f188eb5","Type":"ContainerDied","Data":"dfa107ed437ac79ef26292ad86addb7b61d7881c3dd38f1b66638b4c69ae9dca"}
Sep 15 21:30:03.904308 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:03.904214    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfa107ed437ac79ef26292ad86addb7b61d7881c3dd38f1b66638b4c69ae9dca"
Sep 15 21:30:03.905768 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:03.905748    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-318c22-29825130-mq6kx"
Sep 15 21:30:03.905892 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:03.905783    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-318c22-29825130-mq6kx" event={"ID":"a85ba6cd-6d34-4e45-9c9b-6cc53fa172bd","Type":"ContainerDied","Data":"d1078c774237a2441b124252ae8d855cabc8075175482939441df088500eba96"}
Sep 15 21:30:03.905892 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:03.905815    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1078c774237a2441b124252ae8d855cabc8075175482939441df088500eba96"
Sep 15 21:30:10.687850 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.687817    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"]
Sep 15 21:30:10.688294 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.688281    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ab72d13-937b-4185-ab45-3d5fe40af101" containerName="cleanup"
Sep 15 21:30:10.688335 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.688295    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ab72d13-937b-4185-ab45-3d5fe40af101" containerName="cleanup"
Sep 15 21:30:10.688335 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.688316    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35834631-72e7-45fc-b0c0-7daf6f188eb5" containerName="cleanup"
Sep 15 21:30:10.688335 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.688321    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="35834631-72e7-45fc-b0c0-7daf6f188eb5" containerName="cleanup"
Sep 15 21:30:10.688450 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.688337    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a85ba6cd-6d34-4e45-9c9b-6cc53fa172bd" containerName="cleanup"
Sep 15 21:30:10.688450 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.688343    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a85ba6cd-6d34-4e45-9c9b-6cc53fa172bd" containerName="cleanup"
Sep 15 21:30:10.688450 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.688350    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0fafef0-e8a7-4dab-94a6-5efb31d53120" containerName="cleanup"
Sep 15 21:30:10.688450 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.688355    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0fafef0-e8a7-4dab-94a6-5efb31d53120" containerName="cleanup"
Sep 15 21:30:10.688450 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.688448    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ab72d13-937b-4185-ab45-3d5fe40af101" containerName="cleanup"
Sep 15 21:30:10.688653 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.688455    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0fafef0-e8a7-4dab-94a6-5efb31d53120" containerName="cleanup"
Sep 15 21:30:10.688653 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.688462    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="35834631-72e7-45fc-b0c0-7daf6f188eb5" containerName="cleanup"
Sep 15 21:30:10.688653 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.688470    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a85ba6cd-6d34-4e45-9c9b-6cc53fa172bd" containerName="cleanup"
Sep 15 21:30:10.692755 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.692738    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"
Sep 15 21:30:10.695044 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.695018    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c39135\""
Sep 15 21:30:10.695151 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.695120    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c39135-dockercfg-4dlk7\""
Sep 15 21:30:10.704432 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.704408    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"]
Sep 15 21:30:10.736228 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.736202    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"]
Sep 15 21:30:10.739878 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.739859    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"
Sep 15 21:30:10.751750 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.750450    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"]
Sep 15 21:30:10.875150 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.875115    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/731be20d-8949-42e0-9233-416cbf2a49ac-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs\" (UID: \"731be20d-8949-42e0-9233-416cbf2a49ac\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"
Sep 15 21:30:10.875314 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.875175    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbhr\" (UniqueName: \"kubernetes.io/projected/1de5dc48-089c-4bb3-b40a-6b30c490950e-kube-api-access-6cbhr\") pod \"payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb\" (UID: \"1de5dc48-089c-4bb3-b40a-6b30c490950e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"
Sep 15 21:30:10.875314 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.875238    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz2sh\" (UniqueName: \"kubernetes.io/projected/731be20d-8949-42e0-9233-416cbf2a49ac-kube-api-access-bz2sh\") pod \"payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs\" (UID: \"731be20d-8949-42e0-9233-416cbf2a49ac\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"
Sep 15 21:30:10.875432 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.875352    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de5dc48-089c-4bb3-b40a-6b30c490950e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb\" (UID: \"1de5dc48-089c-4bb3-b40a-6b30c490950e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"
Sep 15 21:30:10.976827 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.976746    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de5dc48-089c-4bb3-b40a-6b30c490950e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb\" (UID: \"1de5dc48-089c-4bb3-b40a-6b30c490950e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"
Sep 15 21:30:10.976990 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.976859    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/731be20d-8949-42e0-9233-416cbf2a49ac-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs\" (UID: \"731be20d-8949-42e0-9233-416cbf2a49ac\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"
Sep 15 21:30:10.976990 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.976887    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbhr\" (UniqueName: \"kubernetes.io/projected/1de5dc48-089c-4bb3-b40a-6b30c490950e-kube-api-access-6cbhr\") pod \"payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb\" (UID: \"1de5dc48-089c-4bb3-b40a-6b30c490950e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"
Sep 15 21:30:10.976990 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.976907    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2sh\" (UniqueName: \"kubernetes.io/projected/731be20d-8949-42e0-9233-416cbf2a49ac-kube-api-access-bz2sh\") pod \"payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs\" (UID: \"731be20d-8949-42e0-9233-416cbf2a49ac\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"
Sep 15 21:30:10.977495 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.977470    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de5dc48-089c-4bb3-b40a-6b30c490950e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb\" (UID: \"1de5dc48-089c-4bb3-b40a-6b30c490950e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"
Sep 15 21:30:10.977687 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.977666    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/731be20d-8949-42e0-9233-416cbf2a49ac-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs\" (UID: \"731be20d-8949-42e0-9233-416cbf2a49ac\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"
Sep 15 21:30:10.987180 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.987147    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz2sh\" (UniqueName: \"kubernetes.io/projected/731be20d-8949-42e0-9233-416cbf2a49ac-kube-api-access-bz2sh\") pod \"payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs\" (UID: \"731be20d-8949-42e0-9233-416cbf2a49ac\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"
Sep 15 21:30:10.987268 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:10.987147    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbhr\" (UniqueName: \"kubernetes.io/projected/1de5dc48-089c-4bb3-b40a-6b30c490950e-kube-api-access-6cbhr\") pod \"payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb\" (UID: \"1de5dc48-089c-4bb3-b40a-6b30c490950e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"
Sep 15 21:30:11.003828 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:11.003806    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"
Sep 15 21:30:11.052830 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:11.052602    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"
Sep 15 21:30:11.281395 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:11.281353    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"]
Sep 15 21:30:11.282791 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:30:11.282771    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731be20d_8949_42e0_9233_416cbf2a49ac.slice/crio-2ae62ae2512ff06019a09788b79f775853a77e1712a08b5fe999a9b0b7f2e0d9 WatchSource:0}: Error finding container 2ae62ae2512ff06019a09788b79f775853a77e1712a08b5fe999a9b0b7f2e0d9: Status 404 returned error can't find the container with id 2ae62ae2512ff06019a09788b79f775853a77e1712a08b5fe999a9b0b7f2e0d9
Sep 15 21:30:11.341105 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:11.341081    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"]
Sep 15 21:30:11.341769 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:30:11.341737    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de5dc48_089c_4bb3_b40a_6b30c490950e.slice/crio-89682061c803742c4d6ccfbdb949203866c614b751e4f4306d119ae6994736f6 WatchSource:0}: Error finding container 89682061c803742c4d6ccfbdb949203866c614b751e4f4306d119ae6994736f6: Status 404 returned error can't find the container with id 89682061c803742c4d6ccfbdb949203866c614b751e4f4306d119ae6994736f6
Sep 15 21:30:11.963799 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:11.963761    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb" event={"ID":"1de5dc48-089c-4bb3-b40a-6b30c490950e","Type":"ContainerStarted","Data":"c617dd1fc7ddc1bd61bc0f477f7bb198f7ac191c1714a6678789cc82e01ce135"}
Sep 15 21:30:11.964271 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:11.963825    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb" event={"ID":"1de5dc48-089c-4bb3-b40a-6b30c490950e","Type":"ContainerStarted","Data":"89682061c803742c4d6ccfbdb949203866c614b751e4f4306d119ae6994736f6"}
Sep 15 21:30:11.964271 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:11.963856    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"
Sep 15 21:30:11.965520 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:11.965494    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs" event={"ID":"731be20d-8949-42e0-9233-416cbf2a49ac","Type":"ContainerStarted","Data":"498a4ea55b97a9588f8983a21442d4165fb8e71dcaa8cdd0fd7b1d64cf40f27a"}
Sep 15 21:30:11.965605 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:11.965549    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs" event={"ID":"731be20d-8949-42e0-9233-416cbf2a49ac","Type":"ContainerStarted","Data":"2ae62ae2512ff06019a09788b79f775853a77e1712a08b5fe999a9b0b7f2e0d9"}
Sep 15 21:30:11.965605 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:11.965594    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"
Sep 15 21:30:11.979981 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:11.979936    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb" podStartSLOduration=1.979921531 podStartE2EDuration="1.979921531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:11.978556713 +0000 UTC m=+1179.840490519" watchObservedRunningTime="2026-09-15 21:30:11.979921531 +0000 UTC m=+1179.841855338"
Sep 15 21:30:11.995561 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:11.995518    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs" podStartSLOduration=1.995504363 podStartE2EDuration="1.995504363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:11.994138752 +0000 UTC m=+1179.856072571" watchObservedRunningTime="2026-09-15 21:30:11.995504363 +0000 UTC m=+1179.857438164"
Sep 15 21:30:22.974272 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:22.974242    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"
Sep 15 21:30:22.974760 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:22.974312    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"
Sep 15 21:30:32.627202 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:32.627170    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:30:32.628630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:32.628610    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:30:32.701398 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:32.701356    2584 scope.go:117] "RemoveContainer" containerID="db6f6f2691b0189eb4663995b96acf45fe8c70514dc5556fb76a9410be405323"
Sep 15 21:30:46.038571 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.038531    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"]
Sep 15 21:30:46.043716 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.043697    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.046962 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.046938    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d1980a0d-openshift-default-dockercfg-kf45c\""
Sep 15 21:30:46.053933 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.053904    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a01b09d-6c2c-4d63-936a-a23306890739-istiod-ca-cert\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.054065 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.053947    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a01b09d-6c2c-4d63-936a-a23306890739-istio-token\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.054065 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.054019    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a01b09d-6c2c-4d63-936a-a23306890739-istio-podinfo\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.054065 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.054060    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-istio-data\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.054234 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.054085    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-istio-envoy\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.054234 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.054136    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-workload-socket\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.054234 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.054191    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49btc\" (UniqueName: \"kubernetes.io/projected/8a01b09d-6c2c-4d63-936a-a23306890739-kube-api-access-49btc\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.054234 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.054214    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-credential-socket\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.054481 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.054240    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-workload-certs\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.067742 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.067711    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"]
Sep 15 21:30:46.154972 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.154935    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-istio-envoy\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.155239 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.155220    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-workload-socket\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.155456 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.155429    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49btc\" (UniqueName: \"kubernetes.io/projected/8a01b09d-6c2c-4d63-936a-a23306890739-kube-api-access-49btc\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.155588 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.155493    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-credential-socket\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.155588 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.155532    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-workload-certs\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.155588 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.155573    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a01b09d-6c2c-4d63-936a-a23306890739-istiod-ca-cert\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.155753 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.155608    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a01b09d-6c2c-4d63-936a-a23306890739-istio-token\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.155753 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.155682    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a01b09d-6c2c-4d63-936a-a23306890739-istio-podinfo\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.155753 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.155741    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-istio-data\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.156206 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.156187    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-istio-data\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.156783 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.156746    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-workload-certs\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.156996 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.156965    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-workload-socket\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.157530 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.157507    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-credential-socket\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.158763 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.158660    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a01b09d-6c2c-4d63-936a-a23306890739-istiod-ca-cert\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.160777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.160757    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-istio-envoy\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.161417 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.161330    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a01b09d-6c2c-4d63-936a-a23306890739-istio-podinfo\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.168226 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.168195    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a01b09d-6c2c-4d63-936a-a23306890739-istio-token\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.170432 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.170406    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49btc\" (UniqueName: \"kubernetes.io/projected/8a01b09d-6c2c-4d63-936a-a23306890739-kube-api-access-49btc\") pod \"e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") " pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.356364 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.356279    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:46.632981 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.632957    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"]
Sep 15 21:30:46.634664 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:30:46.634638    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a01b09d_6c2c_4d63_936a_a23306890739.slice/crio-65f2857e0a163b66540b10fc9302c1f4a257354994f8b3675056e5c5f830b3eb WatchSource:0}: Error finding container 65f2857e0a163b66540b10fc9302c1f4a257354994f8b3675056e5c5f830b3eb: Status 404 returned error can't find the container with id 65f2857e0a163b66540b10fc9302c1f4a257354994f8b3675056e5c5f830b3eb
Sep 15 21:30:46.637116 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.637085    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:30:46.637214 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.637163    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:30:46.637214 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:46.637205    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:30:47.197329 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:47.197290    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk" event={"ID":"8a01b09d-6c2c-4d63-936a-a23306890739","Type":"ContainerStarted","Data":"bc328b0a746f49da36e3d855c31de82a7da052d3bd0a0bf7167da831a4ac1c75"}
Sep 15 21:30:47.197728 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:47.197353    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk" event={"ID":"8a01b09d-6c2c-4d63-936a-a23306890739","Type":"ContainerStarted","Data":"65f2857e0a163b66540b10fc9302c1f4a257354994f8b3675056e5c5f830b3eb"}
Sep 15 21:30:47.222589 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:47.222546    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk" podStartSLOduration=1.222530956 podStartE2EDuration="1.222530956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:47.220150506 +0000 UTC m=+1215.082084314" watchObservedRunningTime="2026-09-15 21:30:47.222530956 +0000 UTC m=+1215.084464761"
Sep 15 21:30:47.356628 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:47.356591    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:47.358501 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:47.358474    2584 patch_prober.go:28] interesting pod/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 15 21:30:47.358634 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:47.358524    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk" podUID="8a01b09d-6c2c-4d63-936a-a23306890739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 15 21:30:48.361065 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:48.361022    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:49.210128 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:49.210099    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:49.211143 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:49.211120    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:30:54.644903 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:54.644866    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"]
Sep 15 21:30:54.650700 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:54.650673    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"
Sep 15 21:30:54.653486 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:54.653463    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d1980a0d-dockercfg-2q28n\""
Sep 15 21:30:54.653652 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:54.653530    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d1980a0d\""
Sep 15 21:30:54.658746 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:54.658701    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"]
Sep 15 21:30:54.754888 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:54.754833    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1889beef-45b3-4c21-b83c-e575f6d44595-plugins-config-volume\") pod \"payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4\" (UID: \"1889beef-45b3-4c21-b83c-e575f6d44595\") " pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"
Sep 15 21:30:54.755074 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:54.755004    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6blpk\" (UniqueName: \"kubernetes.io/projected/1889beef-45b3-4c21-b83c-e575f6d44595-kube-api-access-6blpk\") pod \"payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4\" (UID: \"1889beef-45b3-4c21-b83c-e575f6d44595\") " pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"
Sep 15 21:30:54.856642 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:54.856611    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1889beef-45b3-4c21-b83c-e575f6d44595-plugins-config-volume\") pod \"payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4\" (UID: \"1889beef-45b3-4c21-b83c-e575f6d44595\") " pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"
Sep 15 21:30:54.856840 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:54.856703    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6blpk\" (UniqueName: \"kubernetes.io/projected/1889beef-45b3-4c21-b83c-e575f6d44595-kube-api-access-6blpk\") pod \"payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4\" (UID: \"1889beef-45b3-4c21-b83c-e575f6d44595\") " pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"
Sep 15 21:30:54.857355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:54.857328    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1889beef-45b3-4c21-b83c-e575f6d44595-plugins-config-volume\") pod \"payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4\" (UID: \"1889beef-45b3-4c21-b83c-e575f6d44595\") " pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"
Sep 15 21:30:54.865633 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:54.865604    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6blpk\" (UniqueName: \"kubernetes.io/projected/1889beef-45b3-4c21-b83c-e575f6d44595-kube-api-access-6blpk\") pod \"payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4\" (UID: \"1889beef-45b3-4c21-b83c-e575f6d44595\") " pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"
Sep 15 21:30:54.965980 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:54.965891    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"
Sep 15 21:30:55.238868 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:55.238842    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"]
Sep 15 21:30:55.240446 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:30:55.240415    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1889beef_45b3_4c21_b83c_e575f6d44595.slice/crio-6318e6daf06481500d4edf1d2ec76421d123112ff8ccd9b3a8604860569c5113 WatchSource:0}: Error finding container 6318e6daf06481500d4edf1d2ec76421d123112ff8ccd9b3a8604860569c5113: Status 404 returned error can't find the container with id 6318e6daf06481500d4edf1d2ec76421d123112ff8ccd9b3a8604860569c5113
Sep 15 21:30:55.252045 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:55.252013    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4" event={"ID":"1889beef-45b3-4c21-b83c-e575f6d44595","Type":"ContainerStarted","Data":"6318e6daf06481500d4edf1d2ec76421d123112ff8ccd9b3a8604860569c5113"}
Sep 15 21:30:56.260337 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:56.260304    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4" event={"ID":"1889beef-45b3-4c21-b83c-e575f6d44595","Type":"ContainerStarted","Data":"e8e62f899db390a747b735b95889918c4fb4f6c47421d2d933a2d477e5cd5fff"}
Sep 15 21:30:56.260836 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:56.260372    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"
Sep 15 21:30:56.278994 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:30:56.278939    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4" podStartSLOduration=2.278922219 podStartE2EDuration="2.278922219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:56.278138433 +0000 UTC m=+1224.140072240" watchObservedRunningTime="2026-09-15 21:30:56.278922219 +0000 UTC m=+1224.140856025"
Sep 15 21:31:06.735194 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:06.735157    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b"]
Sep 15 21:31:06.738882 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:06.738861    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b"
Sep 15 21:31:06.741123 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:06.741100    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:31:06.741123 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:06.741114    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:31:06.746032 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:06.746012    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b"]
Sep 15 21:31:06.889093 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:06.889049    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af9dc616-1e69-41a5-a3f1-59a5a77849e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b\" (UID: \"af9dc616-1e69-41a5-a3f1-59a5a77849e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b"
Sep 15 21:31:06.990201 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:06.990093    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af9dc616-1e69-41a5-a3f1-59a5a77849e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b\" (UID: \"af9dc616-1e69-41a5-a3f1-59a5a77849e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b"
Sep 15 21:31:06.990862 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:06.990824    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af9dc616-1e69-41a5-a3f1-59a5a77849e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b\" (UID: \"af9dc616-1e69-41a5-a3f1-59a5a77849e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b"
Sep 15 21:31:07.084888 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:07.084854    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b"
Sep 15 21:31:07.271457 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:07.270588    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"
Sep 15 21:31:07.369607 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:07.369558    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b"]
Sep 15 21:31:07.380503 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:31:07.380467    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9dc616_1e69_41a5_a3f1_59a5a77849e7.slice/crio-2e2e1943cf04d47f9c08685d6e449724166fddc748d49342d6888da83e738020 WatchSource:0}: Error finding container 2e2e1943cf04d47f9c08685d6e449724166fddc748d49342d6888da83e738020: Status 404 returned error can't find the container with id 2e2e1943cf04d47f9c08685d6e449724166fddc748d49342d6888da83e738020
Sep 15 21:31:08.335877 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:08.335838    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b" event={"ID":"af9dc616-1e69-41a5-a3f1-59a5a77849e7","Type":"ContainerStarted","Data":"b2d8ef6449d33b6d11281c64f7b23c98b9ef48dc761ceea7c6270583198cfe79"}
Sep 15 21:31:08.336274 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:08.335903    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b" event={"ID":"af9dc616-1e69-41a5-a3f1-59a5a77849e7","Type":"ContainerStarted","Data":"2e2e1943cf04d47f9c08685d6e449724166fddc748d49342d6888da83e738020"}
Sep 15 21:31:08.351757 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:08.351702    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b" podStartSLOduration=2.351683566 podStartE2EDuration="2.351683566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:08.350891384 +0000 UTC m=+1236.212825184" watchObservedRunningTime="2026-09-15 21:31:08.351683566 +0000 UTC m=+1236.213617412"
Sep 15 21:31:09.343792 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:09.343757    2584 generic.go:356] "Generic (PLEG): container finished" podID="af9dc616-1e69-41a5-a3f1-59a5a77849e7" containerID="b2d8ef6449d33b6d11281c64f7b23c98b9ef48dc761ceea7c6270583198cfe79" exitCode=0
Sep 15 21:31:09.344197 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:09.343840    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b" event={"ID":"af9dc616-1e69-41a5-a3f1-59a5a77849e7","Type":"ContainerDied","Data":"b2d8ef6449d33b6d11281c64f7b23c98b9ef48dc761ceea7c6270583198cfe79"}
Sep 15 21:31:10.542266 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:10.542237    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b"
Sep 15 21:31:10.629226 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:10.629143    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af9dc616-1e69-41a5-a3f1-59a5a77849e7-maas-api-service-ca\") pod \"af9dc616-1e69-41a5-a3f1-59a5a77849e7\" (UID: \"af9dc616-1e69-41a5-a3f1-59a5a77849e7\") "
Sep 15 21:31:10.629549 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:10.629525    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9dc616-1e69-41a5-a3f1-59a5a77849e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af9dc616-1e69-41a5-a3f1-59a5a77849e7" (UID: "af9dc616-1e69-41a5-a3f1-59a5a77849e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:31:10.629679 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:10.629664    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af9dc616-1e69-41a5-a3f1-59a5a77849e7-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:31:11.360071 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:11.360040    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b"
Sep 15 21:31:11.360071 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:11.360054    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b" event={"ID":"af9dc616-1e69-41a5-a3f1-59a5a77849e7","Type":"ContainerDied","Data":"2e2e1943cf04d47f9c08685d6e449724166fddc748d49342d6888da83e738020"}
Sep 15 21:31:11.360284 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:11.360118    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e2e1943cf04d47f9c08685d6e449724166fddc748d49342d6888da83e738020"
Sep 15 21:31:14.381542 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:14.381504    2584 generic.go:356] "Generic (PLEG): container finished" podID="8a01b09d-6c2c-4d63-936a-a23306890739" containerID="bc328b0a746f49da36e3d855c31de82a7da052d3bd0a0bf7167da831a4ac1c75" exitCode=0
Sep 15 21:31:14.382063 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:14.381575    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk" event={"ID":"8a01b09d-6c2c-4d63-936a-a23306890739","Type":"ContainerDied","Data":"bc328b0a746f49da36e3d855c31de82a7da052d3bd0a0bf7167da831a4ac1c75"}
Sep 15 21:31:14.382063 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:14.382015    2584 scope.go:117] "RemoveContainer" containerID="bc328b0a746f49da36e3d855c31de82a7da052d3bd0a0bf7167da831a4ac1c75"
Sep 15 21:31:14.383244 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:14.383212    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:31:14.383330 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:14.383281    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:31:14.383330 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:14.383312    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:31:15.390276 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:15.390245    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk" event={"ID":"8a01b09d-6c2c-4d63-936a-a23306890739","Type":"ContainerStarted","Data":"3cc8a7c025fb20f37d92e321468a4d9218b4783dfb50a8c79079003290802616"}
Sep 15 21:31:16.358200 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:16.358168    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:31:16.359112 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:16.358638    2584 patch_prober.go:28] interesting pod/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 15 21:31:16.359112 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:16.358712    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk" podUID="8a01b09d-6c2c-4d63-936a-a23306890739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 15 21:31:17.357724 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:17.357675    2584 patch_prober.go:28] interesting pod/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 15 21:31:17.358222 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:17.357766    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk" podUID="8a01b09d-6c2c-4d63-936a-a23306890739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 15 21:31:17.582835 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:17.582808    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cfc1f4-29825130-p2zng"]
Sep 15 21:31:17.588194 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:17.588161    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cfc1f4-29825130-p2zng"]
Sep 15 21:31:17.625395 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:17.625304    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"]
Sep 15 21:31:17.625640 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:17.625612    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp" podUID="6e1773c5-716b-42bb-95c9-78428703021e" containerName="payload-processing" containerID="cri-o://674473612b0ac214a6d1622407d894b290c6eb2a3438f207740bd76ec7e394fc" gracePeriod=30
Sep 15 21:31:17.634665 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:17.634639    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"]
Sep 15 21:31:17.637166 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:17.635935    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl" podUID="9c802975-75f6-4098-8469-7fff9d2b77db" containerName="payload-pre-processing" containerID="cri-o://31e6b12ec9979db33144465d44dbbbbcb6cdcd3839a34e9c030e34e8fa099add" gracePeriod=30
Sep 15 21:31:17.811089 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:17.811055    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b"]
Sep 15 21:31:17.815395 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:17.815348    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cfc1f4-67r8b"]
Sep 15 21:31:18.034660 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.034639    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"
Sep 15 21:31:18.037520 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.037502    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"
Sep 15 21:31:18.113888 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.113856    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1773c5-716b-42bb-95c9-78428703021e-plugins-config-volume\") pod \"6e1773c5-716b-42bb-95c9-78428703021e\" (UID: \"6e1773c5-716b-42bb-95c9-78428703021e\") "
Sep 15 21:31:18.114056 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.113925    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c802975-75f6-4098-8469-7fff9d2b77db-plugins-config-volume\") pod \"9c802975-75f6-4098-8469-7fff9d2b77db\" (UID: \"9c802975-75f6-4098-8469-7fff9d2b77db\") "
Sep 15 21:31:18.114056 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.113958    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnkpt\" (UniqueName: \"kubernetes.io/projected/6e1773c5-716b-42bb-95c9-78428703021e-kube-api-access-jnkpt\") pod \"6e1773c5-716b-42bb-95c9-78428703021e\" (UID: \"6e1773c5-716b-42bb-95c9-78428703021e\") "
Sep 15 21:31:18.114056 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.114021    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cnrk\" (UniqueName: \"kubernetes.io/projected/9c802975-75f6-4098-8469-7fff9d2b77db-kube-api-access-8cnrk\") pod \"9c802975-75f6-4098-8469-7fff9d2b77db\" (UID: \"9c802975-75f6-4098-8469-7fff9d2b77db\") "
Sep 15 21:31:18.114268 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.114241    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1773c5-716b-42bb-95c9-78428703021e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e1773c5-716b-42bb-95c9-78428703021e" (UID: "6e1773c5-716b-42bb-95c9-78428703021e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:31:18.114349 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.114323    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c802975-75f6-4098-8469-7fff9d2b77db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c802975-75f6-4098-8469-7fff9d2b77db" (UID: "9c802975-75f6-4098-8469-7fff9d2b77db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:31:18.114513 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.114485    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1773c5-716b-42bb-95c9-78428703021e-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:31:18.114601 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.114512    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c802975-75f6-4098-8469-7fff9d2b77db-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:31:18.116221 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.116195    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c802975-75f6-4098-8469-7fff9d2b77db-kube-api-access-8cnrk" (OuterVolumeSpecName: "kube-api-access-8cnrk") pod "9c802975-75f6-4098-8469-7fff9d2b77db" (UID: "9c802975-75f6-4098-8469-7fff9d2b77db"). InnerVolumeSpecName "kube-api-access-8cnrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:18.116317 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.116201    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1773c5-716b-42bb-95c9-78428703021e-kube-api-access-jnkpt" (OuterVolumeSpecName: "kube-api-access-jnkpt") pod "6e1773c5-716b-42bb-95c9-78428703021e" (UID: "6e1773c5-716b-42bb-95c9-78428703021e"). InnerVolumeSpecName "kube-api-access-jnkpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:18.215897 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.215817    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cnrk\" (UniqueName: \"kubernetes.io/projected/9c802975-75f6-4098-8469-7fff9d2b77db-kube-api-access-8cnrk\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:31:18.215897 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.215846    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnkpt\" (UniqueName: \"kubernetes.io/projected/6e1773c5-716b-42bb-95c9-78428703021e-kube-api-access-jnkpt\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:31:18.356933 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.356889    2584 patch_prober.go:28] interesting pod/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 15 21:31:18.357136 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.356959    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk" podUID="8a01b09d-6c2c-4d63-936a-a23306890739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 15 21:31:18.411333 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.411298    2584 generic.go:356] "Generic (PLEG): container finished" podID="6e1773c5-716b-42bb-95c9-78428703021e" containerID="674473612b0ac214a6d1622407d894b290c6eb2a3438f207740bd76ec7e394fc" exitCode=0
Sep 15 21:31:18.411803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.411342    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"
Sep 15 21:31:18.411803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.411395    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp" event={"ID":"6e1773c5-716b-42bb-95c9-78428703021e","Type":"ContainerDied","Data":"674473612b0ac214a6d1622407d894b290c6eb2a3438f207740bd76ec7e394fc"}
Sep 15 21:31:18.411803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.411444    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp" event={"ID":"6e1773c5-716b-42bb-95c9-78428703021e","Type":"ContainerDied","Data":"a7ae7ee0174541b71226ce84ddc3bd417ff1147e802705aa4e8461667d2ed8c9"}
Sep 15 21:31:18.411803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.411466    2584 scope.go:117] "RemoveContainer" containerID="674473612b0ac214a6d1622407d894b290c6eb2a3438f207740bd76ec7e394fc"
Sep 15 21:31:18.412911 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.412889    2584 generic.go:356] "Generic (PLEG): container finished" podID="9c802975-75f6-4098-8469-7fff9d2b77db" containerID="31e6b12ec9979db33144465d44dbbbbcb6cdcd3839a34e9c030e34e8fa099add" exitCode=0
Sep 15 21:31:18.413020 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.412939    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl" event={"ID":"9c802975-75f6-4098-8469-7fff9d2b77db","Type":"ContainerDied","Data":"31e6b12ec9979db33144465d44dbbbbcb6cdcd3839a34e9c030e34e8fa099add"}
Sep 15 21:31:18.413020 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.412975    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl" event={"ID":"9c802975-75f6-4098-8469-7fff9d2b77db","Type":"ContainerDied","Data":"90a21f3c6e31cd624e9d1aadacc258746f0b00e016b3f73a9892cc591126303d"}
Sep 15 21:31:18.413020 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.412940    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"
Sep 15 21:31:18.422644 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.422628    2584 scope.go:117] "RemoveContainer" containerID="674473612b0ac214a6d1622407d894b290c6eb2a3438f207740bd76ec7e394fc"
Sep 15 21:31:18.422945 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:31:18.422922    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674473612b0ac214a6d1622407d894b290c6eb2a3438f207740bd76ec7e394fc\": container with ID starting with 674473612b0ac214a6d1622407d894b290c6eb2a3438f207740bd76ec7e394fc not found: ID does not exist" containerID="674473612b0ac214a6d1622407d894b290c6eb2a3438f207740bd76ec7e394fc"
Sep 15 21:31:18.423038 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.422953    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674473612b0ac214a6d1622407d894b290c6eb2a3438f207740bd76ec7e394fc"} err="failed to get container status \"674473612b0ac214a6d1622407d894b290c6eb2a3438f207740bd76ec7e394fc\": rpc error: code = NotFound desc = could not find container \"674473612b0ac214a6d1622407d894b290c6eb2a3438f207740bd76ec7e394fc\": container with ID starting with 674473612b0ac214a6d1622407d894b290c6eb2a3438f207740bd76ec7e394fc not found: ID does not exist"
Sep 15 21:31:18.423038 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.422971    2584 scope.go:117] "RemoveContainer" containerID="31e6b12ec9979db33144465d44dbbbbcb6cdcd3839a34e9c030e34e8fa099add"
Sep 15 21:31:18.431194 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.431173    2584 scope.go:117] "RemoveContainer" containerID="31e6b12ec9979db33144465d44dbbbbcb6cdcd3839a34e9c030e34e8fa099add"
Sep 15 21:31:18.431515 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:31:18.431497    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e6b12ec9979db33144465d44dbbbbcb6cdcd3839a34e9c030e34e8fa099add\": container with ID starting with 31e6b12ec9979db33144465d44dbbbbcb6cdcd3839a34e9c030e34e8fa099add not found: ID does not exist" containerID="31e6b12ec9979db33144465d44dbbbbcb6cdcd3839a34e9c030e34e8fa099add"
Sep 15 21:31:18.431583 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.431527    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e6b12ec9979db33144465d44dbbbbcb6cdcd3839a34e9c030e34e8fa099add"} err="failed to get container status \"31e6b12ec9979db33144465d44dbbbbcb6cdcd3839a34e9c030e34e8fa099add\": rpc error: code = NotFound desc = could not find container \"31e6b12ec9979db33144465d44dbbbbcb6cdcd3839a34e9c030e34e8fa099add\": container with ID starting with 31e6b12ec9979db33144465d44dbbbbcb6cdcd3839a34e9c030e34e8fa099add not found: ID does not exist"
Sep 15 21:31:18.436059 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.436038    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"]
Sep 15 21:31:18.438693 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.438673    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cfc1f4-56fc4578c5-r9dwp"]
Sep 15 21:31:18.447553 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.447532    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"]
Sep 15 21:31:18.463406 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.459675    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cfc1f4-87895cd6-mkgkl"]
Sep 15 21:31:18.679064 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.679028    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e1773c5-716b-42bb-95c9-78428703021e" path="/var/lib/kubelet/pods/6e1773c5-716b-42bb-95c9-78428703021e/volumes"
Sep 15 21:31:18.679518 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.679501    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ab72d13-937b-4185-ab45-3d5fe40af101" path="/var/lib/kubelet/pods/9ab72d13-937b-4185-ab45-3d5fe40af101/volumes"
Sep 15 21:31:18.679863 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.679849    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c802975-75f6-4098-8469-7fff9d2b77db" path="/var/lib/kubelet/pods/9c802975-75f6-4098-8469-7fff9d2b77db/volumes"
Sep 15 21:31:18.680157 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:18.680145    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af9dc616-1e69-41a5-a3f1-59a5a77849e7" path="/var/lib/kubelet/pods/af9dc616-1e69-41a5-a3f1-59a5a77849e7/volumes"
Sep 15 21:31:19.210676 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:19.210641    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:31:19.357248 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:19.357206    2584 patch_prober.go:28] interesting pod/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 15 21:31:19.357440 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:19.357269    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk" podUID="8a01b09d-6c2c-4d63-936a-a23306890739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 15 21:31:20.358248 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:20.358211    2584 patch_prober.go:28] interesting pod/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 15 21:31:20.358844 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:20.358811    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk" podUID="8a01b09d-6c2c-4d63-936a-a23306890739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 15 21:31:21.361207 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:21.361171    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:31:21.362293 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:21.362272    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:31:24.526001 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.525971    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"]
Sep 15 21:31:24.526516 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.526233    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj" podUID="0ab13af8-a313-43a0-8447-3b16b24ee632" containerName="istio-proxy" containerID="cri-o://d05515952f5f9c2c5c0cf9a56460f43d70dfae79bf75de6a22585ff16eb0477b" gracePeriod=30
Sep 15 21:31:24.857260 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.857164    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm"]
Sep 15 21:31:24.857778 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.857757    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af9dc616-1e69-41a5-a3f1-59a5a77849e7" containerName="revoke-keys"
Sep 15 21:31:24.857778 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.857779    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="af9dc616-1e69-41a5-a3f1-59a5a77849e7" containerName="revoke-keys"
Sep 15 21:31:24.857959 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.857812    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c802975-75f6-4098-8469-7fff9d2b77db" containerName="payload-pre-processing"
Sep 15 21:31:24.857959 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.857821    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c802975-75f6-4098-8469-7fff9d2b77db" containerName="payload-pre-processing"
Sep 15 21:31:24.857959 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.857834    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1773c5-716b-42bb-95c9-78428703021e" containerName="payload-processing"
Sep 15 21:31:24.857959 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.857842    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1773c5-716b-42bb-95c9-78428703021e" containerName="payload-processing"
Sep 15 21:31:24.858160 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.857983    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="af9dc616-1e69-41a5-a3f1-59a5a77849e7" containerName="revoke-keys"
Sep 15 21:31:24.858160 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.858000    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c802975-75f6-4098-8469-7fff9d2b77db" containerName="payload-pre-processing"
Sep 15 21:31:24.858160 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.858009    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e1773c5-716b-42bb-95c9-78428703021e" containerName="payload-processing"
Sep 15 21:31:24.861741 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.861719    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm"
Sep 15 21:31:24.864177 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.864152    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:31:24.864297 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.864224    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:31:24.867362 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.867342    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm"]
Sep 15 21:31:24.981729 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:24.981698    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df294454-9a04-46b0-985d-5df5f2c6baec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm\" (UID: \"df294454-9a04-46b0-985d-5df5f2c6baec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm"
Sep 15 21:31:25.082509 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:25.082474    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df294454-9a04-46b0-985d-5df5f2c6baec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm\" (UID: \"df294454-9a04-46b0-985d-5df5f2c6baec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm"
Sep 15 21:31:25.083091 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:25.083072    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df294454-9a04-46b0-985d-5df5f2c6baec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm\" (UID: \"df294454-9a04-46b0-985d-5df5f2c6baec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm"
Sep 15 21:31:25.173950 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:25.173848    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm"
Sep 15 21:31:25.447491 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:25.447457    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm"]
Sep 15 21:31:25.447811 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:31:25.447788    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf294454_9a04_46b0_985d_5df5f2c6baec.slice/crio-cf88645767499c65f059f470606f1337045b80373af00f637fc85f549a384f47 WatchSource:0}: Error finding container cf88645767499c65f059f470606f1337045b80373af00f637fc85f549a384f47: Status 404 returned error can't find the container with id cf88645767499c65f059f470606f1337045b80373af00f637fc85f549a384f47
Sep 15 21:31:25.461890 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:25.461864    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm" event={"ID":"df294454-9a04-46b0-985d-5df5f2c6baec","Type":"ContainerStarted","Data":"cf88645767499c65f059f470606f1337045b80373af00f637fc85f549a384f47"}
Sep 15 21:31:26.470264 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:26.470229    2584 generic.go:356] "Generic (PLEG): container finished" podID="df294454-9a04-46b0-985d-5df5f2c6baec" containerID="caadd8f41bf7845b2a0821924522265621c2bb4e31f9e2417c652374d786cc40" exitCode=0
Sep 15 21:31:26.470670 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:26.470319    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm" event={"ID":"df294454-9a04-46b0-985d-5df5f2c6baec","Type":"ContainerDied","Data":"caadd8f41bf7845b2a0821924522265621c2bb4e31f9e2417c652374d786cc40"}
Sep 15 21:31:27.621455 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:27.621432    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm"
Sep 15 21:31:27.714146 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:27.714114    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df294454-9a04-46b0-985d-5df5f2c6baec-maas-api-service-ca\") pod \"df294454-9a04-46b0-985d-5df5f2c6baec\" (UID: \"df294454-9a04-46b0-985d-5df5f2c6baec\") "
Sep 15 21:31:27.714491 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:27.714467    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df294454-9a04-46b0-985d-5df5f2c6baec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "df294454-9a04-46b0-985d-5df5f2c6baec" (UID: "df294454-9a04-46b0-985d-5df5f2c6baec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:31:27.714583 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:27.714571    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df294454-9a04-46b0-985d-5df5f2c6baec-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:31:28.485002 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:28.484969    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm"
Sep 15 21:31:28.485195 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:28.484997    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm" event={"ID":"df294454-9a04-46b0-985d-5df5f2c6baec","Type":"ContainerDied","Data":"cf88645767499c65f059f470606f1337045b80373af00f637fc85f549a384f47"}
Sep 15 21:31:28.485195 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:28.485055    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf88645767499c65f059f470606f1337045b80373af00f637fc85f549a384f47"
Sep 15 21:31:29.820197 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.820175    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:31:29.938116 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938090    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-workload-certs\") pod \"0ab13af8-a313-43a0-8447-3b16b24ee632\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") "
Sep 15 21:31:29.938116 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938121    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-podinfo\") pod \"0ab13af8-a313-43a0-8447-3b16b24ee632\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") "
Sep 15 21:31:29.938331 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938171    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab13af8-a313-43a0-8447-3b16b24ee632-istiod-ca-cert\") pod \"0ab13af8-a313-43a0-8447-3b16b24ee632\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") "
Sep 15 21:31:29.938331 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938190    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-credential-socket\") pod \"0ab13af8-a313-43a0-8447-3b16b24ee632\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") "
Sep 15 21:31:29.938331 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938242    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-envoy\") pod \"0ab13af8-a313-43a0-8447-3b16b24ee632\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") "
Sep 15 21:31:29.938331 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938292    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-workload-socket\") pod \"0ab13af8-a313-43a0-8447-3b16b24ee632\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") "
Sep 15 21:31:29.938331 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938325    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phxcw\" (UniqueName: \"kubernetes.io/projected/0ab13af8-a313-43a0-8447-3b16b24ee632-kube-api-access-phxcw\") pod \"0ab13af8-a313-43a0-8447-3b16b24ee632\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") "
Sep 15 21:31:29.938730 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938362    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-data\") pod \"0ab13af8-a313-43a0-8447-3b16b24ee632\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") "
Sep 15 21:31:29.938730 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938425    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0ab13af8-a313-43a0-8447-3b16b24ee632" (UID: "0ab13af8-a313-43a0-8447-3b16b24ee632"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:29.938730 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938445    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-token\") pod \"0ab13af8-a313-43a0-8447-3b16b24ee632\" (UID: \"0ab13af8-a313-43a0-8447-3b16b24ee632\") "
Sep 15 21:31:29.938730 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938513    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0ab13af8-a313-43a0-8447-3b16b24ee632" (UID: "0ab13af8-a313-43a0-8447-3b16b24ee632"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:29.938730 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938670    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-data" (OuterVolumeSpecName: "istio-data") pod "0ab13af8-a313-43a0-8447-3b16b24ee632" (UID: "0ab13af8-a313-43a0-8447-3b16b24ee632"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:29.938730 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938682    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0ab13af8-a313-43a0-8447-3b16b24ee632" (UID: "0ab13af8-a313-43a0-8447-3b16b24ee632"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:29.938730 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938723    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab13af8-a313-43a0-8447-3b16b24ee632-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0ab13af8-a313-43a0-8447-3b16b24ee632" (UID: "0ab13af8-a313-43a0-8447-3b16b24ee632"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:31:29.939084 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938943    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-workload-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:31:29.939084 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938959    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-data\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:31:29.939084 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938974    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-workload-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:31:29.939084 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938984    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab13af8-a313-43a0-8447-3b16b24ee632-istiod-ca-cert\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:31:29.939084 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.938992    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-credential-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:31:29.940424 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.940399    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0ab13af8-a313-43a0-8447-3b16b24ee632" (UID: "0ab13af8-a313-43a0-8447-3b16b24ee632"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:31:29.940568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.940550    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0ab13af8-a313-43a0-8447-3b16b24ee632" (UID: "0ab13af8-a313-43a0-8447-3b16b24ee632"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:29.940968 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.940947    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab13af8-a313-43a0-8447-3b16b24ee632-kube-api-access-phxcw" (OuterVolumeSpecName: "kube-api-access-phxcw") pod "0ab13af8-a313-43a0-8447-3b16b24ee632" (UID: "0ab13af8-a313-43a0-8447-3b16b24ee632"). InnerVolumeSpecName "kube-api-access-phxcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:29.941027 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:29.940955    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-token" (OuterVolumeSpecName: "istio-token") pod "0ab13af8-a313-43a0-8447-3b16b24ee632" (UID: "0ab13af8-a313-43a0-8447-3b16b24ee632"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:30.039808 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.039766    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phxcw\" (UniqueName: \"kubernetes.io/projected/0ab13af8-a313-43a0-8447-3b16b24ee632-kube-api-access-phxcw\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:31:30.039808 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.039805    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-token\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:31:30.039808 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.039815    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-podinfo\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:31:30.039808 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.039823    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab13af8-a313-43a0-8447-3b16b24ee632-istio-envoy\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:31:30.500734 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.500702    2584 generic.go:356] "Generic (PLEG): container finished" podID="0ab13af8-a313-43a0-8447-3b16b24ee632" containerID="d05515952f5f9c2c5c0cf9a56460f43d70dfae79bf75de6a22585ff16eb0477b" exitCode=0
Sep 15 21:31:30.500899 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.500743    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj" event={"ID":"0ab13af8-a313-43a0-8447-3b16b24ee632","Type":"ContainerDied","Data":"d05515952f5f9c2c5c0cf9a56460f43d70dfae79bf75de6a22585ff16eb0477b"}
Sep 15 21:31:30.500899 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.500753    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"
Sep 15 21:31:30.500899 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.500786    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj" event={"ID":"0ab13af8-a313-43a0-8447-3b16b24ee632","Type":"ContainerDied","Data":"db3190b3832574bebf1812e247d2258fd209f582466b8c9f5fe807bf35ef729f"}
Sep 15 21:31:30.500899 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.500807    2584 scope.go:117] "RemoveContainer" containerID="d05515952f5f9c2c5c0cf9a56460f43d70dfae79bf75de6a22585ff16eb0477b"
Sep 15 21:31:30.511428 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.511409    2584 scope.go:117] "RemoveContainer" containerID="a49805afa90a74c3c8debd5f3e6e865128bf37232fbaa40587055bc0d8d6a74a"
Sep 15 21:31:30.520201 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.520183    2584 scope.go:117] "RemoveContainer" containerID="d05515952f5f9c2c5c0cf9a56460f43d70dfae79bf75de6a22585ff16eb0477b"
Sep 15 21:31:30.520493 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:31:30.520476    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05515952f5f9c2c5c0cf9a56460f43d70dfae79bf75de6a22585ff16eb0477b\": container with ID starting with d05515952f5f9c2c5c0cf9a56460f43d70dfae79bf75de6a22585ff16eb0477b not found: ID does not exist" containerID="d05515952f5f9c2c5c0cf9a56460f43d70dfae79bf75de6a22585ff16eb0477b"
Sep 15 21:31:30.520558 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.520502    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05515952f5f9c2c5c0cf9a56460f43d70dfae79bf75de6a22585ff16eb0477b"} err="failed to get container status \"d05515952f5f9c2c5c0cf9a56460f43d70dfae79bf75de6a22585ff16eb0477b\": rpc error: code = NotFound desc = could not find container \"d05515952f5f9c2c5c0cf9a56460f43d70dfae79bf75de6a22585ff16eb0477b\": container with ID starting with d05515952f5f9c2c5c0cf9a56460f43d70dfae79bf75de6a22585ff16eb0477b not found: ID does not exist"
Sep 15 21:31:30.520558 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.520518    2584 scope.go:117] "RemoveContainer" containerID="a49805afa90a74c3c8debd5f3e6e865128bf37232fbaa40587055bc0d8d6a74a"
Sep 15 21:31:30.520771 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:31:30.520752    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49805afa90a74c3c8debd5f3e6e865128bf37232fbaa40587055bc0d8d6a74a\": container with ID starting with a49805afa90a74c3c8debd5f3e6e865128bf37232fbaa40587055bc0d8d6a74a not found: ID does not exist" containerID="a49805afa90a74c3c8debd5f3e6e865128bf37232fbaa40587055bc0d8d6a74a"
Sep 15 21:31:30.520816 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.520777    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49805afa90a74c3c8debd5f3e6e865128bf37232fbaa40587055bc0d8d6a74a"} err="failed to get container status \"a49805afa90a74c3c8debd5f3e6e865128bf37232fbaa40587055bc0d8d6a74a\": rpc error: code = NotFound desc = could not find container \"a49805afa90a74c3c8debd5f3e6e865128bf37232fbaa40587055bc0d8d6a74a\": container with ID starting with a49805afa90a74c3c8debd5f3e6e865128bf37232fbaa40587055bc0d8d6a74a not found: ID does not exist"
Sep 15 21:31:30.524814 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.524791    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"]
Sep 15 21:31:30.528602 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.528581    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-cfc1f4-openshift-default-595bd84f76-nqjpj"]
Sep 15 21:31:30.672839 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:30.672807    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ab13af8-a313-43a0-8447-3b16b24ee632" path="/var/lib/kubelet/pods/0ab13af8-a313-43a0-8447-3b16b24ee632/volumes"
Sep 15 21:31:32.730798 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:32.730763    2584 scope.go:117] "RemoveContainer" containerID="eef92b13af00cf80b8ef5edf42663f5e309528703f38786e3b9e0b3eae2f4c17"
Sep 15 21:31:32.739628 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:32.739609    2584 scope.go:117] "RemoveContainer" containerID="4c6cdf78a5cd27450936093e777f45cd4768ecfdc206104b9d58122ac86118d7"
Sep 15 21:31:35.361644 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:35.361614    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f457e5-29825130-swpnv"]
Sep 15 21:31:35.365197 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:35.365173    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f457e5-29825130-swpnv"]
Sep 15 21:31:35.556069 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:35.556036    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm"]
Sep 15 21:31:35.559373 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:35.559342    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f457e5-l2pvm"]
Sep 15 21:31:36.673635 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:36.673597    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df294454-9a04-46b0-985d-5df5f2c6baec" path="/var/lib/kubelet/pods/df294454-9a04-46b0-985d-5df5f2c6baec/volumes"
Sep 15 21:31:36.674116 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:36.674045    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0fafef0-e8a7-4dab-94a6-5efb31d53120" path="/var/lib/kubelet/pods/e0fafef0-e8a7-4dab-94a6-5efb31d53120/volumes"
Sep 15 21:31:55.661358 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.661328    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"]
Sep 15 21:31:55.661840 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.661817    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab13af8-a313-43a0-8447-3b16b24ee632" containerName="istio-proxy"
Sep 15 21:31:55.661840 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.661830    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab13af8-a313-43a0-8447-3b16b24ee632" containerName="istio-proxy"
Sep 15 21:31:55.661840 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.661841    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab13af8-a313-43a0-8447-3b16b24ee632" containerName="istio-proxy"
Sep 15 21:31:55.661945 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.661846    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab13af8-a313-43a0-8447-3b16b24ee632" containerName="istio-proxy"
Sep 15 21:31:55.661945 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.661859    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df294454-9a04-46b0-985d-5df5f2c6baec" containerName="revoke-keys"
Sep 15 21:31:55.661945 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.661864    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="df294454-9a04-46b0-985d-5df5f2c6baec" containerName="revoke-keys"
Sep 15 21:31:55.662038 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.661946    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab13af8-a313-43a0-8447-3b16b24ee632" containerName="istio-proxy"
Sep 15 21:31:55.662038 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.661953    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="df294454-9a04-46b0-985d-5df5f2c6baec" containerName="revoke-keys"
Sep 15 21:31:55.662038 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.661963    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab13af8-a313-43a0-8447-3b16b24ee632" containerName="istio-proxy"
Sep 15 21:31:55.666950 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.666932    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"
Sep 15 21:31:55.669643 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.669620    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-f36399\""
Sep 15 21:31:55.669745 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.669690    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-f36399-dockercfg-qz2x9\""
Sep 15 21:31:55.675905 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.675846    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"]
Sep 15 21:31:55.707785 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.707757    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e75854-bfed-4c73-99c6-486d34b60b57-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5\" (UID: \"e5e75854-bfed-4c73-99c6-486d34b60b57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"
Sep 15 21:31:55.707901 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.707799    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62zr5\" (UniqueName: \"kubernetes.io/projected/e5e75854-bfed-4c73-99c6-486d34b60b57-kube-api-access-62zr5\") pod \"payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5\" (UID: \"e5e75854-bfed-4c73-99c6-486d34b60b57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"
Sep 15 21:31:55.809048 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.809013    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e75854-bfed-4c73-99c6-486d34b60b57-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5\" (UID: \"e5e75854-bfed-4c73-99c6-486d34b60b57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"
Sep 15 21:31:55.809279 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.809061    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62zr5\" (UniqueName: \"kubernetes.io/projected/e5e75854-bfed-4c73-99c6-486d34b60b57-kube-api-access-62zr5\") pod \"payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5\" (UID: \"e5e75854-bfed-4c73-99c6-486d34b60b57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"
Sep 15 21:31:55.809849 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.809828    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e75854-bfed-4c73-99c6-486d34b60b57-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5\" (UID: \"e5e75854-bfed-4c73-99c6-486d34b60b57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"
Sep 15 21:31:55.817630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.817593    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62zr5\" (UniqueName: \"kubernetes.io/projected/e5e75854-bfed-4c73-99c6-486d34b60b57-kube-api-access-62zr5\") pod \"payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5\" (UID: \"e5e75854-bfed-4c73-99c6-486d34b60b57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"
Sep 15 21:31:55.980336 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:55.980241    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"
Sep 15 21:31:56.257547 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:56.257525    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"]
Sep 15 21:31:56.260151 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:31:56.260126    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e75854_bfed_4c73_99c6_486d34b60b57.slice/crio-f865931c571c1597eaf5ff817f1c60431213ad72718f3f30b425ffc51cb37e6c WatchSource:0}: Error finding container f865931c571c1597eaf5ff817f1c60431213ad72718f3f30b425ffc51cb37e6c: Status 404 returned error can't find the container with id f865931c571c1597eaf5ff817f1c60431213ad72718f3f30b425ffc51cb37e6c
Sep 15 21:31:56.673114 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:56.673081    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5" event={"ID":"e5e75854-bfed-4c73-99c6-486d34b60b57","Type":"ContainerStarted","Data":"38f2b9f303d979e3567f8c10eba88bc0e832a209f056973f8746b58046996b7b"}
Sep 15 21:31:56.673586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:56.673125    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5" event={"ID":"e5e75854-bfed-4c73-99c6-486d34b60b57","Type":"ContainerStarted","Data":"f865931c571c1597eaf5ff817f1c60431213ad72718f3f30b425ffc51cb37e6c"}
Sep 15 21:31:56.673586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:56.673152    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"
Sep 15 21:31:56.683327 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:31:56.683259    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5" podStartSLOduration=1.683241796 podStartE2EDuration="1.683241796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:56.682263063 +0000 UTC m=+1284.544196869" watchObservedRunningTime="2026-09-15 21:31:56.683241796 +0000 UTC m=+1284.545175602"
Sep 15 21:32:07.676054 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:07.676024    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"
Sep 15 21:32:21.813848 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:21.813813    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"]
Sep 15 21:32:21.817550 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:21.817529    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:21.819910 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:21.819885    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-745dff-openshift-default-dockercfg-wnr6s\""
Sep 15 21:32:21.827962 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:21.827936    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"]
Sep 15 21:32:21.900159 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:21.900124    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-token\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:21.900371 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:21.900173    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istiod-ca-cert\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:21.900371 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:21.900243    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-data\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:21.900371 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:21.900280    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkgxc\" (UniqueName: \"kubernetes.io/projected/4437923b-e05b-4d87-896e-7cc6f9b61b7b-kube-api-access-rkgxc\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:21.900566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:21.900401    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-workload-socket\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:21.900566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:21.900452    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-credential-socket\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:21.900566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:21.900514    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-envoy\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:21.900566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:21.900548    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-workload-certs\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:21.900764 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:21.900623    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-podinfo\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.002157 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.002118    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-envoy\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.002489 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.002454    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-workload-certs\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.002685 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.002669    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-podinfo\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.003259 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.003237    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-token\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.003501 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.003461    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istiod-ca-cert\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.003759 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.003730    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-data\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.004083 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.004062    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkgxc\" (UniqueName: \"kubernetes.io/projected/4437923b-e05b-4d87-896e-7cc6f9b61b7b-kube-api-access-rkgxc\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.004319 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.004301    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-workload-socket\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.004496 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.004477    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-credential-socket\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.004717 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.004486    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istiod-ca-cert\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.005129 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.005032    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-data\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.005325 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.002925    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-workload-certs\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.005682 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.005642    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-credential-socket\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.006012 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.005991    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-workload-socket\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.007021 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.007001    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-envoy\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.007806 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.007786    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-podinfo\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.012556 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.012488    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkgxc\" (UniqueName: \"kubernetes.io/projected/4437923b-e05b-4d87-896e-7cc6f9b61b7b-kube-api-access-rkgxc\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.013766 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.013745    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-token\") pod \"e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") " pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.131054 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.130963    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:22.411214 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.411189    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"]
Sep 15 21:32:22.412533 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:32:22.412504    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4437923b_e05b_4d87_896e_7cc6f9b61b7b.slice/crio-f9a15d04c43519e9893d88ba56b34e769900a737d0deefc6807f86310495d0f0 WatchSource:0}: Error finding container f9a15d04c43519e9893d88ba56b34e769900a737d0deefc6807f86310495d0f0: Status 404 returned error can't find the container with id f9a15d04c43519e9893d88ba56b34e769900a737d0deefc6807f86310495d0f0
Sep 15 21:32:22.415024 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.414984    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:32:22.415118 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.415092    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:32:22.415158 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.415136    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:32:22.832300 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.832267    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p" event={"ID":"4437923b-e05b-4d87-896e-7cc6f9b61b7b","Type":"ContainerStarted","Data":"cc7bbf23d9adf946df284a427adcb78b49a7d9ee700f320ef59f0ceed11fc9b9"}
Sep 15 21:32:22.832729 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.832322    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p" event={"ID":"4437923b-e05b-4d87-896e-7cc6f9b61b7b","Type":"ContainerStarted","Data":"f9a15d04c43519e9893d88ba56b34e769900a737d0deefc6807f86310495d0f0"}
Sep 15 21:32:22.864426 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:22.864365    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p" podStartSLOduration=1.8643481130000001 podStartE2EDuration="1.864348113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:22.86226762 +0000 UTC m=+1310.724201424" watchObservedRunningTime="2026-09-15 21:32:22.864348113 +0000 UTC m=+1310.726281918"
Sep 15 21:32:23.131758 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:23.131666    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:23.137226 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:23.137198    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:23.839072 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:23.839039    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:23.840081 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:23.840067    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:32:30.673877 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:30.673838    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"]
Sep 15 21:32:30.679218 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:30.679182    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"
Sep 15 21:32:30.682223 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:30.682202    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-745dff\""
Sep 15 21:32:30.682498 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:30.682483    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-745dff-dockercfg-gmdzz\""
Sep 15 21:32:30.683996 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:30.683973    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"]
Sep 15 21:32:30.798798 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:30.798767    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b33ab9d-8cf7-4c02-b455-f6f604998bef-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn\" (UID: \"4b33ab9d-8cf7-4c02-b455-f6f604998bef\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"
Sep 15 21:32:30.798992 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:30.798957    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g4gv\" (UniqueName: \"kubernetes.io/projected/4b33ab9d-8cf7-4c02-b455-f6f604998bef-kube-api-access-9g4gv\") pod \"payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn\" (UID: \"4b33ab9d-8cf7-4c02-b455-f6f604998bef\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"
Sep 15 21:32:30.899795 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:30.899760    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b33ab9d-8cf7-4c02-b455-f6f604998bef-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn\" (UID: \"4b33ab9d-8cf7-4c02-b455-f6f604998bef\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"
Sep 15 21:32:30.899943 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:30.899852    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g4gv\" (UniqueName: \"kubernetes.io/projected/4b33ab9d-8cf7-4c02-b455-f6f604998bef-kube-api-access-9g4gv\") pod \"payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn\" (UID: \"4b33ab9d-8cf7-4c02-b455-f6f604998bef\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"
Sep 15 21:32:30.900464 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:30.900446    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b33ab9d-8cf7-4c02-b455-f6f604998bef-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn\" (UID: \"4b33ab9d-8cf7-4c02-b455-f6f604998bef\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"
Sep 15 21:32:30.908810 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:30.908783    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g4gv\" (UniqueName: \"kubernetes.io/projected/4b33ab9d-8cf7-4c02-b455-f6f604998bef-kube-api-access-9g4gv\") pod \"payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn\" (UID: \"4b33ab9d-8cf7-4c02-b455-f6f604998bef\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"
Sep 15 21:32:30.993554 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:30.993458    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"
Sep 15 21:32:31.266397 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:31.266348    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"]
Sep 15 21:32:31.269576 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:32:31.269540    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b33ab9d_8cf7_4c02_b455_f6f604998bef.slice/crio-0e6a2f2aeb4e6d02ea24e37c55bb40b89b53b38687a54908dbff0c4f4f87e2b6 WatchSource:0}: Error finding container 0e6a2f2aeb4e6d02ea24e37c55bb40b89b53b38687a54908dbff0c4f4f87e2b6: Status 404 returned error can't find the container with id 0e6a2f2aeb4e6d02ea24e37c55bb40b89b53b38687a54908dbff0c4f4f87e2b6
Sep 15 21:32:31.898564 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:31.898525    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn" event={"ID":"4b33ab9d-8cf7-4c02-b455-f6f604998bef","Type":"ContainerStarted","Data":"43a83e5fed2d3510a9222cca3d8a52621c13c2450b544c8fe1544ebd9fe9f26d"}
Sep 15 21:32:31.899044 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:31.898579    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn" event={"ID":"4b33ab9d-8cf7-4c02-b455-f6f604998bef","Type":"ContainerStarted","Data":"0e6a2f2aeb4e6d02ea24e37c55bb40b89b53b38687a54908dbff0c4f4f87e2b6"}
Sep 15 21:32:31.899044 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:31.898616    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"
Sep 15 21:32:31.916309 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:31.916261    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn" podStartSLOduration=1.91624382 podStartE2EDuration="1.91624382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:31.913442341 +0000 UTC m=+1319.775376145" watchObservedRunningTime="2026-09-15 21:32:31.91624382 +0000 UTC m=+1319.778177624"
Sep 15 21:32:32.781879 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:32.781848    2584 scope.go:117] "RemoveContainer" containerID="9395b527155194a6a062e70fce9ba9cb944de3b42330e0b17f387768b5056263"
Sep 15 21:32:32.791206 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:32.791184    2584 scope.go:117] "RemoveContainer" containerID="e2b5548ff687698c22a348cfb947c13e13528ab299bbfda3491f7077fe0b4e83"
Sep 15 21:32:32.801646 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:32.801622    2584 scope.go:117] "RemoveContainer" containerID="44f654e00b419accdae3e9a161206d35dc1ff818519a3814618cea65efa34b33"
Sep 15 21:32:42.907162 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:32:42.907132    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"
Sep 15 21:33:01.086660 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:01.086629    2584 generic.go:356] "Generic (PLEG): container finished" podID="4437923b-e05b-4d87-896e-7cc6f9b61b7b" containerID="cc7bbf23d9adf946df284a427adcb78b49a7d9ee700f320ef59f0ceed11fc9b9" exitCode=0
Sep 15 21:33:01.087229 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:01.086707    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p" event={"ID":"4437923b-e05b-4d87-896e-7cc6f9b61b7b","Type":"ContainerDied","Data":"cc7bbf23d9adf946df284a427adcb78b49a7d9ee700f320ef59f0ceed11fc9b9"}
Sep 15 21:33:01.087307 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:01.087287    2584 scope.go:117] "RemoveContainer" containerID="cc7bbf23d9adf946df284a427adcb78b49a7d9ee700f320ef59f0ceed11fc9b9"
Sep 15 21:33:01.088561 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:01.088531    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:33:01.088681 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:01.088624    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:33:01.088681 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:01.088672    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:33:02.095679 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:02.095638    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p" event={"ID":"4437923b-e05b-4d87-896e-7cc6f9b61b7b","Type":"ContainerStarted","Data":"d8ff5fac1c20f6c01377f14fa47337ab6a8e47bbd65061464fd7ba2c833671f1"}
Sep 15 21:33:02.132009 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:02.131980    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:33:02.132168 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:02.132152    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 15 21:33:02.132241 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:02.132205    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p" podUID="4437923b-e05b-4d87-896e-7cc6f9b61b7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 15 21:33:03.132153 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:03.132124    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 15 21:33:03.132560 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:03.132174    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p" podUID="4437923b-e05b-4d87-896e-7cc6f9b61b7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 15 21:33:04.131880 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:04.131843    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 15 21:33:04.132056 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:04.131895    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p" podUID="4437923b-e05b-4d87-896e-7cc6f9b61b7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 15 21:33:05.131573 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:05.131541    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 15 21:33:05.131965 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:05.131594    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p" podUID="4437923b-e05b-4d87-896e-7cc6f9b61b7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 15 21:33:06.132282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:06.132252    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 15 21:33:06.132772 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:06.132313    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p" podUID="4437923b-e05b-4d87-896e-7cc6f9b61b7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 15 21:33:07.135819 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:07.135788    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:33:07.136278 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:07.136178    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:33:07.136729 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:07.136711    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:33:15.525506 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:15.525467    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr"]
Sep 15 21:33:15.528508 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:15.528492    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr"
Sep 15 21:33:15.530456 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:15.530436    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:33:15.530542 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:15.530529    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:33:15.535104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:15.535073    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr"]
Sep 15 21:33:15.588143 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:15.588095    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/063499eb-2c36-4919-8bd6-bb0e1e6eeef5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr\" (UID: \"063499eb-2c36-4919-8bd6-bb0e1e6eeef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr"
Sep 15 21:33:15.689437 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:15.689400    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/063499eb-2c36-4919-8bd6-bb0e1e6eeef5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr\" (UID: \"063499eb-2c36-4919-8bd6-bb0e1e6eeef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr"
Sep 15 21:33:15.690039 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:15.690019    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/063499eb-2c36-4919-8bd6-bb0e1e6eeef5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr\" (UID: \"063499eb-2c36-4919-8bd6-bb0e1e6eeef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr"
Sep 15 21:33:15.841035 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:15.840952    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr"
Sep 15 21:33:16.120187 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:16.120159    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr"]
Sep 15 21:33:16.121787 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:33:16.121760    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063499eb_2c36_4919_8bd6_bb0e1e6eeef5.slice/crio-41c3dc15bb4fcd121ae1a05493acebdb3d781e24203e63d6155a3fe64ba6cdcd WatchSource:0}: Error finding container 41c3dc15bb4fcd121ae1a05493acebdb3d781e24203e63d6155a3fe64ba6cdcd: Status 404 returned error can't find the container with id 41c3dc15bb4fcd121ae1a05493acebdb3d781e24203e63d6155a3fe64ba6cdcd
Sep 15 21:33:16.182762 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:16.182736    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr" event={"ID":"063499eb-2c36-4919-8bd6-bb0e1e6eeef5","Type":"ContainerStarted","Data":"41c3dc15bb4fcd121ae1a05493acebdb3d781e24203e63d6155a3fe64ba6cdcd"}
Sep 15 21:33:17.190457 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:17.190363    2584 generic.go:356] "Generic (PLEG): container finished" podID="063499eb-2c36-4919-8bd6-bb0e1e6eeef5" containerID="ecd116531c520e075aed87e768646ed7a73c5b51b1d1b278ceab547aab0198d4" exitCode=0
Sep 15 21:33:17.190457 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:17.190437    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr" event={"ID":"063499eb-2c36-4919-8bd6-bb0e1e6eeef5","Type":"ContainerDied","Data":"ecd116531c520e075aed87e768646ed7a73c5b51b1d1b278ceab547aab0198d4"}
Sep 15 21:33:18.329279 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:18.329257    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr"
Sep 15 21:33:18.418438 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:18.418412    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/063499eb-2c36-4919-8bd6-bb0e1e6eeef5-maas-api-service-ca\") pod \"063499eb-2c36-4919-8bd6-bb0e1e6eeef5\" (UID: \"063499eb-2c36-4919-8bd6-bb0e1e6eeef5\") "
Sep 15 21:33:18.418746 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:18.418724    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063499eb-2c36-4919-8bd6-bb0e1e6eeef5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "063499eb-2c36-4919-8bd6-bb0e1e6eeef5" (UID: "063499eb-2c36-4919-8bd6-bb0e1e6eeef5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:33:18.519778 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:18.519716    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/063499eb-2c36-4919-8bd6-bb0e1e6eeef5-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:33:19.203998 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:19.203969    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr"
Sep 15 21:33:19.204189 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:19.203969    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr" event={"ID":"063499eb-2c36-4919-8bd6-bb0e1e6eeef5","Type":"ContainerDied","Data":"41c3dc15bb4fcd121ae1a05493acebdb3d781e24203e63d6155a3fe64ba6cdcd"}
Sep 15 21:33:19.204189 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:19.204095    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41c3dc15bb4fcd121ae1a05493acebdb3d781e24203e63d6155a3fe64ba6cdcd"
Sep 15 21:33:31.157919 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:31.157883    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"]
Sep 15 21:33:31.158762 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:31.158732    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4" podUID="1889beef-45b3-4c21-b83c-e575f6d44595" containerName="payload-processing" containerID="cri-o://e8e62f899db390a747b735b95889918c4fb4f6c47421d2d933a2d477e5cd5fff" gracePeriod=30
Sep 15 21:33:31.281300 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:31.281268    2584 generic.go:356] "Generic (PLEG): container finished" podID="1889beef-45b3-4c21-b83c-e575f6d44595" containerID="e8e62f899db390a747b735b95889918c4fb4f6c47421d2d933a2d477e5cd5fff" exitCode=0
Sep 15 21:33:31.281463 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:31.281335    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4" event={"ID":"1889beef-45b3-4c21-b83c-e575f6d44595","Type":"ContainerDied","Data":"e8e62f899db390a747b735b95889918c4fb4f6c47421d2d933a2d477e5cd5fff"}
Sep 15 21:33:31.319757 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:31.319678    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr"]
Sep 15 21:33:31.324092 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:31.324069    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1980a0d-6jqxr"]
Sep 15 21:33:31.446071 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:31.445994    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"
Sep 15 21:33:31.546681 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:31.546653    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6blpk\" (UniqueName: \"kubernetes.io/projected/1889beef-45b3-4c21-b83c-e575f6d44595-kube-api-access-6blpk\") pod \"1889beef-45b3-4c21-b83c-e575f6d44595\" (UID: \"1889beef-45b3-4c21-b83c-e575f6d44595\") "
Sep 15 21:33:31.546855 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:31.546837    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1889beef-45b3-4c21-b83c-e575f6d44595-plugins-config-volume\") pod \"1889beef-45b3-4c21-b83c-e575f6d44595\" (UID: \"1889beef-45b3-4c21-b83c-e575f6d44595\") "
Sep 15 21:33:31.547135 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:31.547108    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1889beef-45b3-4c21-b83c-e575f6d44595-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1889beef-45b3-4c21-b83c-e575f6d44595" (UID: "1889beef-45b3-4c21-b83c-e575f6d44595"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:33:31.547245 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:31.547226    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1889beef-45b3-4c21-b83c-e575f6d44595-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:33:31.548750 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:31.548719    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1889beef-45b3-4c21-b83c-e575f6d44595-kube-api-access-6blpk" (OuterVolumeSpecName: "kube-api-access-6blpk") pod "1889beef-45b3-4c21-b83c-e575f6d44595" (UID: "1889beef-45b3-4c21-b83c-e575f6d44595"). InnerVolumeSpecName "kube-api-access-6blpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:33:31.648766 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:31.648734    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6blpk\" (UniqueName: \"kubernetes.io/projected/1889beef-45b3-4c21-b83c-e575f6d44595-kube-api-access-6blpk\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:33:32.288598 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:32.288566    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"
Sep 15 21:33:32.288994 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:32.288612    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4" event={"ID":"1889beef-45b3-4c21-b83c-e575f6d44595","Type":"ContainerDied","Data":"6318e6daf06481500d4edf1d2ec76421d123112ff8ccd9b3a8604860569c5113"}
Sep 15 21:33:32.288994 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:32.288675    2584 scope.go:117] "RemoveContainer" containerID="e8e62f899db390a747b735b95889918c4fb4f6c47421d2d933a2d477e5cd5fff"
Sep 15 21:33:32.310389 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:32.310360    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"]
Sep 15 21:33:32.313860 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:32.313839    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1980a0d-5594855677-m6qr4"]
Sep 15 21:33:32.677168 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:32.677065    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="063499eb-2c36-4919-8bd6-bb0e1e6eeef5" path="/var/lib/kubelet/pods/063499eb-2c36-4919-8bd6-bb0e1e6eeef5/volumes"
Sep 15 21:33:32.677693 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:32.677668    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1889beef-45b3-4c21-b83c-e575f6d44595" path="/var/lib/kubelet/pods/1889beef-45b3-4c21-b83c-e575f6d44595/volumes"
Sep 15 21:33:51.119778 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:33:51.119740    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"]
Sep 15 21:34:06.139225 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.139185    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg" podUID="8790c434-1fdc-4dd5-ab28-a803bbc72722" containerName="main" containerID="cri-o://5dee47066eefa30a57e4191865fd99cf2ec7314d4924a15916799b0f4347c736" gracePeriod=45
Sep 15 21:34:06.395056 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.394997    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:34:06.479701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.479672    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-home\") pod \"8790c434-1fdc-4dd5-ab28-a803bbc72722\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") "
Sep 15 21:34:06.479842 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.479741    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-model-cache\") pod \"8790c434-1fdc-4dd5-ab28-a803bbc72722\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") "
Sep 15 21:34:06.479842 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.479801    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-tmp-dir\") pod \"8790c434-1fdc-4dd5-ab28-a803bbc72722\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") "
Sep 15 21:34:06.479842 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.479837    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-dshm\") pod \"8790c434-1fdc-4dd5-ab28-a803bbc72722\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") "
Sep 15 21:34:06.480020 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.479893    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8790c434-1fdc-4dd5-ab28-a803bbc72722-tls-certs\") pod \"8790c434-1fdc-4dd5-ab28-a803bbc72722\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") "
Sep 15 21:34:06.480020 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.479942    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpl42\" (UniqueName: \"kubernetes.io/projected/8790c434-1fdc-4dd5-ab28-a803bbc72722-kube-api-access-cpl42\") pod \"8790c434-1fdc-4dd5-ab28-a803bbc72722\" (UID: \"8790c434-1fdc-4dd5-ab28-a803bbc72722\") "
Sep 15 21:34:06.480020 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.479942    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-home" (OuterVolumeSpecName: "home") pod "8790c434-1fdc-4dd5-ab28-a803bbc72722" (UID: "8790c434-1fdc-4dd5-ab28-a803bbc72722"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:34:06.480166 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.480015    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-model-cache" (OuterVolumeSpecName: "model-cache") pod "8790c434-1fdc-4dd5-ab28-a803bbc72722" (UID: "8790c434-1fdc-4dd5-ab28-a803bbc72722"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:34:06.480166 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.480092    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8790c434-1fdc-4dd5-ab28-a803bbc72722" (UID: "8790c434-1fdc-4dd5-ab28-a803bbc72722"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:34:06.480407 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.480361    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-tmp-dir\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:34:06.480538 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.480416    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-home\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:34:06.480538 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.480429    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-model-cache\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:34:06.482048 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.482020    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8790c434-1fdc-4dd5-ab28-a803bbc72722-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8790c434-1fdc-4dd5-ab28-a803bbc72722" (UID: "8790c434-1fdc-4dd5-ab28-a803bbc72722"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:34:06.482048 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.482031    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-dshm" (OuterVolumeSpecName: "dshm") pod "8790c434-1fdc-4dd5-ab28-a803bbc72722" (UID: "8790c434-1fdc-4dd5-ab28-a803bbc72722"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:34:06.482173 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.482114    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8790c434-1fdc-4dd5-ab28-a803bbc72722-kube-api-access-cpl42" (OuterVolumeSpecName: "kube-api-access-cpl42") pod "8790c434-1fdc-4dd5-ab28-a803bbc72722" (UID: "8790c434-1fdc-4dd5-ab28-a803bbc72722"). InnerVolumeSpecName "kube-api-access-cpl42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:34:06.503694 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.503670    2584 generic.go:356] "Generic (PLEG): container finished" podID="8790c434-1fdc-4dd5-ab28-a803bbc72722" containerID="5dee47066eefa30a57e4191865fd99cf2ec7314d4924a15916799b0f4347c736" exitCode=0
Sep 15 21:34:06.503795 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.503710    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"
Sep 15 21:34:06.503795 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.503753    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg" event={"ID":"8790c434-1fdc-4dd5-ab28-a803bbc72722","Type":"ContainerDied","Data":"5dee47066eefa30a57e4191865fd99cf2ec7314d4924a15916799b0f4347c736"}
Sep 15 21:34:06.503912 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.503801    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg" event={"ID":"8790c434-1fdc-4dd5-ab28-a803bbc72722","Type":"ContainerDied","Data":"72e0ec87ab6ac2131cea32baecfc4294c488ec27d4da681917e67f26ab539daf"}
Sep 15 21:34:06.503912 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.503823    2584 scope.go:117] "RemoveContainer" containerID="5dee47066eefa30a57e4191865fd99cf2ec7314d4924a15916799b0f4347c736"
Sep 15 21:34:06.513440 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.513419    2584 scope.go:117] "RemoveContainer" containerID="5dee47066eefa30a57e4191865fd99cf2ec7314d4924a15916799b0f4347c736"
Sep 15 21:34:06.513723 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:34:06.513706    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dee47066eefa30a57e4191865fd99cf2ec7314d4924a15916799b0f4347c736\": container with ID starting with 5dee47066eefa30a57e4191865fd99cf2ec7314d4924a15916799b0f4347c736 not found: ID does not exist" containerID="5dee47066eefa30a57e4191865fd99cf2ec7314d4924a15916799b0f4347c736"
Sep 15 21:34:06.513773 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.513733    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dee47066eefa30a57e4191865fd99cf2ec7314d4924a15916799b0f4347c736"} err="failed to get container status \"5dee47066eefa30a57e4191865fd99cf2ec7314d4924a15916799b0f4347c736\": rpc error: code = NotFound desc = could not find container \"5dee47066eefa30a57e4191865fd99cf2ec7314d4924a15916799b0f4347c736\": container with ID starting with 5dee47066eefa30a57e4191865fd99cf2ec7314d4924a15916799b0f4347c736 not found: ID does not exist"
Sep 15 21:34:06.525228 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.525209    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"]
Sep 15 21:34:06.529115 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.529095    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/e2e-conflict-a-8487c1aa-kserve-85cfdf879b-2d8zg"]
Sep 15 21:34:06.581349 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.581330    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8790c434-1fdc-4dd5-ab28-a803bbc72722-dshm\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:34:06.581349 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.581349    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8790c434-1fdc-4dd5-ab28-a803bbc72722-tls-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:34:06.581491 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.581359    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpl42\" (UniqueName: \"kubernetes.io/projected/8790c434-1fdc-4dd5-ab28-a803bbc72722-kube-api-access-cpl42\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:34:06.674240 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:06.674187    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8790c434-1fdc-4dd5-ab28-a803bbc72722" path="/var/lib/kubelet/pods/8790c434-1fdc-4dd5-ab28-a803bbc72722/volumes"
Sep 15 21:34:12.308431 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.308374    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"]
Sep 15 21:34:12.308921 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.308905    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8790c434-1fdc-4dd5-ab28-a803bbc72722" containerName="main"
Sep 15 21:34:12.308921 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.308921    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8790c434-1fdc-4dd5-ab28-a803bbc72722" containerName="main"
Sep 15 21:34:12.309049 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.308943    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1889beef-45b3-4c21-b83c-e575f6d44595" containerName="payload-processing"
Sep 15 21:34:12.309049 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.308949    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1889beef-45b3-4c21-b83c-e575f6d44595" containerName="payload-processing"
Sep 15 21:34:12.309049 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.308974    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="063499eb-2c36-4919-8bd6-bb0e1e6eeef5" containerName="revoke-keys"
Sep 15 21:34:12.309049 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.308981    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="063499eb-2c36-4919-8bd6-bb0e1e6eeef5" containerName="revoke-keys"
Sep 15 21:34:12.309246 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.309058    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1889beef-45b3-4c21-b83c-e575f6d44595" containerName="payload-processing"
Sep 15 21:34:12.309246 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.309067    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8790c434-1fdc-4dd5-ab28-a803bbc72722" containerName="main"
Sep 15 21:34:12.309246 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.309076    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="063499eb-2c36-4919-8bd6-bb0e1e6eeef5" containerName="revoke-keys"
Sep 15 21:34:12.312066 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.312048    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.314707 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.314688    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6f66f7\"/\"distinct-w4-6f66f7-kserve-self-signed-certs\""
Sep 15 21:34:12.321129 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.321108    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"]
Sep 15 21:34:12.434795 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.434771    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-dshm\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.434929 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.434810    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2dtk\" (UniqueName: \"kubernetes.io/projected/8356eab8-04b6-42b2-8a33-f6069e71ac6b-kube-api-access-n2dtk\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.434929 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.434848    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-home\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.434929 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.434896    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-tmp-dir\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.435036 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.434960    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-model-cache\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.435036 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.435010    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8356eab8-04b6-42b2-8a33-f6069e71ac6b-tls-certs\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.535886 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.535853    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-tmp-dir\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.536093 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.535909    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-model-cache\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.536093 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.535949    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8356eab8-04b6-42b2-8a33-f6069e71ac6b-tls-certs\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.536093 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.535976    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-dshm\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.536093 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.536004    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2dtk\" (UniqueName: \"kubernetes.io/projected/8356eab8-04b6-42b2-8a33-f6069e71ac6b-kube-api-access-n2dtk\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.536093 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.536035    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-home\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.536410 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.536312    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-tmp-dir\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.536410 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.536354    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-model-cache\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.536410 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.536372    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-home\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.538239 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.538217    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-dshm\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.538564 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.538535    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8356eab8-04b6-42b2-8a33-f6069e71ac6b-tls-certs\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.544282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.544260    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2dtk\" (UniqueName: \"kubernetes.io/projected/8356eab8-04b6-42b2-8a33-f6069e71ac6b-kube-api-access-n2dtk\") pod \"distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.621600 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.621551    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:12.893773 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:12.893748    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"]
Sep 15 21:34:12.895677 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:34:12.895652    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8356eab8_04b6_42b2_8a33_f6069e71ac6b.slice/crio-c47223e83952c1eae781edc16288a55e8828d69b67a161c26b295a57b9d5ec63 WatchSource:0}: Error finding container c47223e83952c1eae781edc16288a55e8828d69b67a161c26b295a57b9d5ec63: Status 404 returned error can't find the container with id c47223e83952c1eae781edc16288a55e8828d69b67a161c26b295a57b9d5ec63
Sep 15 21:34:13.560669 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:13.560632    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd" event={"ID":"8356eab8-04b6-42b2-8a33-f6069e71ac6b","Type":"ContainerStarted","Data":"c2d0fe0f2113c05e79208de3f8fda00522031ca097355bb4a97871d879f79ac3"}
Sep 15 21:34:13.561024 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:13.560676    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd" event={"ID":"8356eab8-04b6-42b2-8a33-f6069e71ac6b","Type":"ContainerStarted","Data":"c47223e83952c1eae781edc16288a55e8828d69b67a161c26b295a57b9d5ec63"}
Sep 15 21:34:13.578510 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:13.578462    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd" podStartSLOduration=1.5784468139999999 podStartE2EDuration="1.578446814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:34:13.575771021 +0000 UTC m=+1421.437704825" watchObservedRunningTime="2026-09-15 21:34:13.578446814 +0000 UTC m=+1421.440380655"
Sep 15 21:34:22.622740 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:22.622706    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:22.623200 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:22.622787    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:22.635370 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:22.635346    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:23.337396 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:23.337356    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"]
Sep 15 21:34:23.337655 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:23.337631    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk" podUID="8a01b09d-6c2c-4d63-936a-a23306890739" containerName="istio-proxy" containerID="cri-o://3cc8a7c025fb20f37d92e321468a4d9218b4783dfb50a8c79079003290802616" gracePeriod=30
Sep 15 21:34:23.635294 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:23.635218    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:34:28.616405 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.616327    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:34:28.657474 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.657432    2584 generic.go:356] "Generic (PLEG): container finished" podID="8a01b09d-6c2c-4d63-936a-a23306890739" containerID="3cc8a7c025fb20f37d92e321468a4d9218b4783dfb50a8c79079003290802616" exitCode=0
Sep 15 21:34:28.657646 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.657469    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk" event={"ID":"8a01b09d-6c2c-4d63-936a-a23306890739","Type":"ContainerDied","Data":"3cc8a7c025fb20f37d92e321468a4d9218b4783dfb50a8c79079003290802616"}
Sep 15 21:34:28.657646 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.657508    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"
Sep 15 21:34:28.657646 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.657551    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk" event={"ID":"8a01b09d-6c2c-4d63-936a-a23306890739","Type":"ContainerDied","Data":"65f2857e0a163b66540b10fc9302c1f4a257354994f8b3675056e5c5f830b3eb"}
Sep 15 21:34:28.657646 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.657599    2584 scope.go:117] "RemoveContainer" containerID="3cc8a7c025fb20f37d92e321468a4d9218b4783dfb50a8c79079003290802616"
Sep 15 21:34:28.677695 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.673651    2584 scope.go:117] "RemoveContainer" containerID="bc328b0a746f49da36e3d855c31de82a7da052d3bd0a0bf7167da831a4ac1c75"
Sep 15 21:34:28.689107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.689083    2584 scope.go:117] "RemoveContainer" containerID="3cc8a7c025fb20f37d92e321468a4d9218b4783dfb50a8c79079003290802616"
Sep 15 21:34:28.689448 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:34:28.689426    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc8a7c025fb20f37d92e321468a4d9218b4783dfb50a8c79079003290802616\": container with ID starting with 3cc8a7c025fb20f37d92e321468a4d9218b4783dfb50a8c79079003290802616 not found: ID does not exist" containerID="3cc8a7c025fb20f37d92e321468a4d9218b4783dfb50a8c79079003290802616"
Sep 15 21:34:28.689542 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.689459    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc8a7c025fb20f37d92e321468a4d9218b4783dfb50a8c79079003290802616"} err="failed to get container status \"3cc8a7c025fb20f37d92e321468a4d9218b4783dfb50a8c79079003290802616\": rpc error: code = NotFound desc = could not find container \"3cc8a7c025fb20f37d92e321468a4d9218b4783dfb50a8c79079003290802616\": container with ID starting with 3cc8a7c025fb20f37d92e321468a4d9218b4783dfb50a8c79079003290802616 not found: ID does not exist"
Sep 15 21:34:28.689542 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.689484    2584 scope.go:117] "RemoveContainer" containerID="bc328b0a746f49da36e3d855c31de82a7da052d3bd0a0bf7167da831a4ac1c75"
Sep 15 21:34:28.689878 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:34:28.689847    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc328b0a746f49da36e3d855c31de82a7da052d3bd0a0bf7167da831a4ac1c75\": container with ID starting with bc328b0a746f49da36e3d855c31de82a7da052d3bd0a0bf7167da831a4ac1c75 not found: ID does not exist" containerID="bc328b0a746f49da36e3d855c31de82a7da052d3bd0a0bf7167da831a4ac1c75"
Sep 15 21:34:28.689981 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.689886    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc328b0a746f49da36e3d855c31de82a7da052d3bd0a0bf7167da831a4ac1c75"} err="failed to get container status \"bc328b0a746f49da36e3d855c31de82a7da052d3bd0a0bf7167da831a4ac1c75\": rpc error: code = NotFound desc = could not find container \"bc328b0a746f49da36e3d855c31de82a7da052d3bd0a0bf7167da831a4ac1c75\": container with ID starting with bc328b0a746f49da36e3d855c31de82a7da052d3bd0a0bf7167da831a4ac1c75 not found: ID does not exist"
Sep 15 21:34:28.701768 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.701737    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a01b09d-6c2c-4d63-936a-a23306890739-istiod-ca-cert\") pod \"8a01b09d-6c2c-4d63-936a-a23306890739\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") "
Sep 15 21:34:28.701862 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.701804    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-workload-socket\") pod \"8a01b09d-6c2c-4d63-936a-a23306890739\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") "
Sep 15 21:34:28.701862 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.701830    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a01b09d-6c2c-4d63-936a-a23306890739-istio-token\") pod \"8a01b09d-6c2c-4d63-936a-a23306890739\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") "
Sep 15 21:34:28.701862 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.701856    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-workload-certs\") pod \"8a01b09d-6c2c-4d63-936a-a23306890739\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") "
Sep 15 21:34:28.702020 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.701887    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-credential-socket\") pod \"8a01b09d-6c2c-4d63-936a-a23306890739\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") "
Sep 15 21:34:28.702020 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.701971    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-istio-envoy\") pod \"8a01b09d-6c2c-4d63-936a-a23306890739\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") "
Sep 15 21:34:28.702020 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.702000    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49btc\" (UniqueName: \"kubernetes.io/projected/8a01b09d-6c2c-4d63-936a-a23306890739-kube-api-access-49btc\") pod \"8a01b09d-6c2c-4d63-936a-a23306890739\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") "
Sep 15 21:34:28.702169 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.702017    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8a01b09d-6c2c-4d63-936a-a23306890739" (UID: "8a01b09d-6c2c-4d63-936a-a23306890739"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:34:28.702169 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.702051    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-istio-data\") pod \"8a01b09d-6c2c-4d63-936a-a23306890739\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") "
Sep 15 21:34:28.702169 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.702114    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a01b09d-6c2c-4d63-936a-a23306890739-istio-podinfo\") pod \"8a01b09d-6c2c-4d63-936a-a23306890739\" (UID: \"8a01b09d-6c2c-4d63-936a-a23306890739\") "
Sep 15 21:34:28.702169 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.702116    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a01b09d-6c2c-4d63-936a-a23306890739-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8a01b09d-6c2c-4d63-936a-a23306890739" (UID: "8a01b09d-6c2c-4d63-936a-a23306890739"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:34:28.702169 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.702126    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8a01b09d-6c2c-4d63-936a-a23306890739" (UID: "8a01b09d-6c2c-4d63-936a-a23306890739"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:34:28.702454 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.702408    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-istio-data" (OuterVolumeSpecName: "istio-data") pod "8a01b09d-6c2c-4d63-936a-a23306890739" (UID: "8a01b09d-6c2c-4d63-936a-a23306890739"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:34:28.702514 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.702493    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a01b09d-6c2c-4d63-936a-a23306890739-istiod-ca-cert\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:34:28.702567 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.702513    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-workload-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:34:28.702567 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.702528    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-workload-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:34:28.702567 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.702540    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-istio-data\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:34:28.702718 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.702606    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8a01b09d-6c2c-4d63-936a-a23306890739" (UID: "8a01b09d-6c2c-4d63-936a-a23306890739"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:34:28.704182 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.704157    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a01b09d-6c2c-4d63-936a-a23306890739-istio-token" (OuterVolumeSpecName: "istio-token") pod "8a01b09d-6c2c-4d63-936a-a23306890739" (UID: "8a01b09d-6c2c-4d63-936a-a23306890739"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:34:28.704322 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.704300    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a01b09d-6c2c-4d63-936a-a23306890739-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8a01b09d-6c2c-4d63-936a-a23306890739" (UID: "8a01b09d-6c2c-4d63-936a-a23306890739"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:34:28.704520 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.704493    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8a01b09d-6c2c-4d63-936a-a23306890739" (UID: "8a01b09d-6c2c-4d63-936a-a23306890739"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:34:28.704632 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.704547    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a01b09d-6c2c-4d63-936a-a23306890739-kube-api-access-49btc" (OuterVolumeSpecName: "kube-api-access-49btc") pod "8a01b09d-6c2c-4d63-936a-a23306890739" (UID: "8a01b09d-6c2c-4d63-936a-a23306890739"). InnerVolumeSpecName "kube-api-access-49btc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:34:28.804075 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.804039    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a01b09d-6c2c-4d63-936a-a23306890739-istio-token\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:34:28.804075 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.804077    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-credential-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:34:28.804313 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.804092    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a01b09d-6c2c-4d63-936a-a23306890739-istio-envoy\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:34:28.804313 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.804106    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49btc\" (UniqueName: \"kubernetes.io/projected/8a01b09d-6c2c-4d63-936a-a23306890739-kube-api-access-49btc\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:34:28.804313 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.804120    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a01b09d-6c2c-4d63-936a-a23306890739-istio-podinfo\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:34:28.981544 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.981507    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"]
Sep 15 21:34:28.986720 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:28.986696    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d1980a0d-openshift-default-6f4c9b6bd-tn8kk"]
Sep 15 21:34:30.674352 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:34:30.674310    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a01b09d-6c2c-4d63-936a-a23306890739" path="/var/lib/kubelet/pods/8a01b09d-6c2c-4d63-936a-a23306890739/volumes"
Sep 15 21:35:07.110530 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.110493    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"]
Sep 15 21:35:07.110986 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.110964    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a01b09d-6c2c-4d63-936a-a23306890739" containerName="istio-proxy"
Sep 15 21:35:07.110986 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.110977    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a01b09d-6c2c-4d63-936a-a23306890739" containerName="istio-proxy"
Sep 15 21:35:07.111072 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.111061    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a01b09d-6c2c-4d63-936a-a23306890739" containerName="istio-proxy"
Sep 15 21:35:07.114139 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.114126    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.116311 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.116288    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6f66f7\"/\"distinct-2-w4-6f66f7-kserve-self-signed-certs\""
Sep 15 21:35:07.121720 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.121697    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"]
Sep 15 21:35:07.275246 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.275207    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad652e4b-78a3-496e-9cc5-06736f0c6204-tls-certs\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.275436 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.275253    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-tmp-dir\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.275436 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.275319    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbp9d\" (UniqueName: \"kubernetes.io/projected/ad652e4b-78a3-496e-9cc5-06736f0c6204-kube-api-access-nbp9d\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.275436 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.275338    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-dshm\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.275436 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.275429    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-home\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.275587 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.275455    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-model-cache\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.376956 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.376859    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-home\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.377556 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.377529    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-home\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.377734 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.377551    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-model-cache\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.377871 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.377847    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-model-cache\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.378028 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.378012    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad652e4b-78a3-496e-9cc5-06736f0c6204-tls-certs\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.378150 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.378138    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-tmp-dir\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.379022 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.379002    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbp9d\" (UniqueName: \"kubernetes.io/projected/ad652e4b-78a3-496e-9cc5-06736f0c6204-kube-api-access-nbp9d\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.379201 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.379176    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-dshm\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.379408 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.379006    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-tmp-dir\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.383237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.383215    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad652e4b-78a3-496e-9cc5-06736f0c6204-tls-certs\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.384870 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.384840    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-dshm\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.388574 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.388554    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbp9d\" (UniqueName: \"kubernetes.io/projected/ad652e4b-78a3-496e-9cc5-06736f0c6204-kube-api-access-nbp9d\") pod \"distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") " pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.423810 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.423782    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:07.699834 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.699806    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"]
Sep 15 21:35:07.701426 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:35:07.701398    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad652e4b_78a3_496e_9cc5_06736f0c6204.slice/crio-1b96bb8d743f1883f31acc579edebb638ec260e6287d4de1b5140a68dcc90d54 WatchSource:0}: Error finding container 1b96bb8d743f1883f31acc579edebb638ec260e6287d4de1b5140a68dcc90d54: Status 404 returned error can't find the container with id 1b96bb8d743f1883f31acc579edebb638ec260e6287d4de1b5140a68dcc90d54
Sep 15 21:35:07.892786 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.892750    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh" event={"ID":"ad652e4b-78a3-496e-9cc5-06736f0c6204","Type":"ContainerStarted","Data":"e381fc3b3d3914df7ea79c3e1fe56babc3545a0a24ddfee077a4e8b0e5e39fd0"}
Sep 15 21:35:07.892971 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.892798    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh" event={"ID":"ad652e4b-78a3-496e-9cc5-06736f0c6204","Type":"ContainerStarted","Data":"1b96bb8d743f1883f31acc579edebb638ec260e6287d4de1b5140a68dcc90d54"}
Sep 15 21:35:07.917565 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:07.917515    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh" podStartSLOduration=0.917498889 podStartE2EDuration="917.498889ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:07.915795589 +0000 UTC m=+1475.777729393" watchObservedRunningTime="2026-09-15 21:35:07.917498889 +0000 UTC m=+1475.779432693"
Sep 15 21:35:17.424580 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:17.424542    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:17.424580 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:17.424589    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:17.438182 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:17.438152    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:17.966171 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:17.966138    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:35:27.252334 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:27.252290    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"]
Sep 15 21:35:27.253018 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:27.252991    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a01b09d-6c2c-4d63-936a-a23306890739" containerName="istio-proxy"
Sep 15 21:35:27.253018 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:27.253014    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a01b09d-6c2c-4d63-936a-a23306890739" containerName="istio-proxy"
Sep 15 21:35:27.253170 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:27.253114    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a01b09d-6c2c-4d63-936a-a23306890739" containerName="istio-proxy"
Sep 15 21:35:27.256580 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:27.256565    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"
Sep 15 21:35:27.258973 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:27.258951    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d8f9d1\""
Sep 15 21:35:27.259104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:27.259021    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d8f9d1-dockercfg-9vwnb\""
Sep 15 21:35:27.267475 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:27.267450    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"]
Sep 15 21:35:27.385324 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:27.385288    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk4ck\" (UniqueName: \"kubernetes.io/projected/9a6da525-c9a4-4161-85ec-85e06da90622-kube-api-access-mk4ck\") pod \"payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx\" (UID: \"9a6da525-c9a4-4161-85ec-85e06da90622\") " pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"
Sep 15 21:35:27.385518 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:27.385403    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6da525-c9a4-4161-85ec-85e06da90622-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx\" (UID: \"9a6da525-c9a4-4161-85ec-85e06da90622\") " pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"
Sep 15 21:35:27.487073 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:27.487041    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk4ck\" (UniqueName: \"kubernetes.io/projected/9a6da525-c9a4-4161-85ec-85e06da90622-kube-api-access-mk4ck\") pod \"payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx\" (UID: \"9a6da525-c9a4-4161-85ec-85e06da90622\") " pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"
Sep 15 21:35:27.487228 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:27.487082    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6da525-c9a4-4161-85ec-85e06da90622-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx\" (UID: \"9a6da525-c9a4-4161-85ec-85e06da90622\") " pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"
Sep 15 21:35:27.487748 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:27.487729    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6da525-c9a4-4161-85ec-85e06da90622-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx\" (UID: \"9a6da525-c9a4-4161-85ec-85e06da90622\") " pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"
Sep 15 21:35:27.496456 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:27.496433    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk4ck\" (UniqueName: \"kubernetes.io/projected/9a6da525-c9a4-4161-85ec-85e06da90622-kube-api-access-mk4ck\") pod \"payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx\" (UID: \"9a6da525-c9a4-4161-85ec-85e06da90622\") " pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"
Sep 15 21:35:27.568534 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:27.568507    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"
Sep 15 21:35:27.840786 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:27.840765    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"]
Sep 15 21:35:27.842131 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:35:27.842105    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6da525_c9a4_4161_85ec_85e06da90622.slice/crio-b7ae64efc886ada8bb885db627d1753d7bbd068305814f3ce6fffd43a2debb63 WatchSource:0}: Error finding container b7ae64efc886ada8bb885db627d1753d7bbd068305814f3ce6fffd43a2debb63: Status 404 returned error can't find the container with id b7ae64efc886ada8bb885db627d1753d7bbd068305814f3ce6fffd43a2debb63
Sep 15 21:35:28.018971 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:28.018930    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx" event={"ID":"9a6da525-c9a4-4161-85ec-85e06da90622","Type":"ContainerStarted","Data":"0a27d80d49178860a8efa8aff944c4f7a435b26e7c17baec44f30fdc0732abe0"}
Sep 15 21:35:28.019156 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:28.019084    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"
Sep 15 21:35:28.019156 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:28.019117    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx" event={"ID":"9a6da525-c9a4-4161-85ec-85e06da90622","Type":"ContainerStarted","Data":"b7ae64efc886ada8bb885db627d1753d7bbd068305814f3ce6fffd43a2debb63"}
Sep 15 21:35:28.039560 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:28.039519    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx" podStartSLOduration=1.03950563 podStartE2EDuration="1.03950563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:28.036227604 +0000 UTC m=+1495.898161408" watchObservedRunningTime="2026-09-15 21:35:28.03950563 +0000 UTC m=+1495.901439434"
Sep 15 21:35:32.676664 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:32.676633    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:35:32.677617 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:32.677593    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:35:32.869824 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:32.869795    2584 scope.go:117] "RemoveContainer" containerID="1301e734ada6e555a45b39ed39933560d78800ecb23e0526a9308cb017df85b0"
Sep 15 21:35:32.879510 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:32.879356    2584 scope.go:117] "RemoveContainer" containerID="de5bfd393205cc5a8f009c3e2144a9eaea79096ca1a9514e38d16e82219f0321"
Sep 15 21:35:39.027182 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:39.027151    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"
Sep 15 21:35:52.047429 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.047397    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"]
Sep 15 21:35:52.051868 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.051845    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.054819 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.054795    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-119834-openshift-default-dockercfg-gr4gc\""
Sep 15 21:35:52.065357 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.065333    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"]
Sep 15 21:35:52.231875 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.231835    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-credential-socket\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.232054 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.231881    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9wnk\" (UniqueName: \"kubernetes.io/projected/41daf450-ffee-4338-99f0-bc55135c42d9-kube-api-access-v9wnk\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.232054 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.231924    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41daf450-ffee-4338-99f0-bc55135c42d9-istio-podinfo\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.232054 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.232008    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-istio-data\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.232196 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.232083    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41daf450-ffee-4338-99f0-bc55135c42d9-istio-token\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.232196 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.232126    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41daf450-ffee-4338-99f0-bc55135c42d9-istiod-ca-cert\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.232196 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.232159    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-workload-certs\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.232303 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.232219    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-workload-socket\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.232303 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.232258    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-istio-envoy\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.333502 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.333400    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41daf450-ffee-4338-99f0-bc55135c42d9-istiod-ca-cert\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.333502 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.333460    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-workload-certs\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.333502 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.333504    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-workload-socket\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.333816 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.333550    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-istio-envoy\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.333816 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.333593    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-credential-socket\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.333816 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.333629    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9wnk\" (UniqueName: \"kubernetes.io/projected/41daf450-ffee-4338-99f0-bc55135c42d9-kube-api-access-v9wnk\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.333816 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.333665    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41daf450-ffee-4338-99f0-bc55135c42d9-istio-podinfo\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.333816 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.333735    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-istio-data\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.333816 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.333794    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41daf450-ffee-4338-99f0-bc55135c42d9-istio-token\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.334163 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.333875    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-workload-certs\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.334163 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.334107    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41daf450-ffee-4338-99f0-bc55135c42d9-istiod-ca-cert\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.334329 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.334278    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-credential-socket\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.334329 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.334318    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-istio-data\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.334662 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.334636    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-workload-socket\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.336285 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.336264    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-istio-envoy\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.336799 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.336778    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41daf450-ffee-4338-99f0-bc55135c42d9-istio-podinfo\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.348677 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.348651    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41daf450-ffee-4338-99f0-bc55135c42d9-istio-token\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.349634 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.349611    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9wnk\" (UniqueName: \"kubernetes.io/projected/41daf450-ffee-4338-99f0-bc55135c42d9-kube-api-access-v9wnk\") pod \"e2e-api-b-119834-openshift-default-694cb45d4b-t26m5\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") " pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.363288 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.363261    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:52.653633 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.653593    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"]
Sep 15 21:35:52.655255 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:35:52.655222    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41daf450_ffee_4338_99f0_bc55135c42d9.slice/crio-0f16abcd98f0f8cb63f217f94cc92ab31fc70d8fa5b3872765f054aeb61e51de WatchSource:0}: Error finding container 0f16abcd98f0f8cb63f217f94cc92ab31fc70d8fa5b3872765f054aeb61e51de: Status 404 returned error can't find the container with id 0f16abcd98f0f8cb63f217f94cc92ab31fc70d8fa5b3872765f054aeb61e51de
Sep 15 21:35:52.657792 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.657755    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:35:52.657894 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.657839    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:35:52.657894 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:52.657878    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:35:53.177106 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:53.177074    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5" event={"ID":"41daf450-ffee-4338-99f0-bc55135c42d9","Type":"ContainerStarted","Data":"e25e7c2088b134b68eb09c22cd545d343d9747e5f813d45fb815a9c725e241a5"}
Sep 15 21:35:53.177479 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:53.177137    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5" event={"ID":"41daf450-ffee-4338-99f0-bc55135c42d9","Type":"ContainerStarted","Data":"0f16abcd98f0f8cb63f217f94cc92ab31fc70d8fa5b3872765f054aeb61e51de"}
Sep 15 21:35:53.200564 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:53.200518    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5" podStartSLOduration=1.200504972 podStartE2EDuration="1.200504972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:53.19754936 +0000 UTC m=+1521.059483165" watchObservedRunningTime="2026-09-15 21:35:53.200504972 +0000 UTC m=+1521.062438775"
Sep 15 21:35:53.363842 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:53.363812    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:53.368900 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:53.368877    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:54.183266 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:54.183230    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:35:54.184254 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:35:54.184236    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:36:32.906201 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:32.906168    2584 scope.go:117] "RemoveContainer" containerID="c27e01e6b9789de7f1f930df8d380781de597edbb229a373e71b3af556ee754d"
Sep 15 21:36:32.915275 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:32.915255    2584 scope.go:117] "RemoveContainer" containerID="234b44fe70f8dca6e5dbb2db8d216d5145066a5537a2a51bb0d54f53d650c3ee"
Sep 15 21:36:33.213848 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:33.213767    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2"]
Sep 15 21:36:33.219069 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:33.219052    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2"
Sep 15 21:36:33.221631 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:33.221606    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:36:33.221896 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:33.221864    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:36:33.224470 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:33.224450    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2"]
Sep 15 21:36:33.242734 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:33.242701    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f169e28-8a0e-4022-80f3-30457725f2e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2\" (UID: \"3f169e28-8a0e-4022-80f3-30457725f2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2"
Sep 15 21:36:33.343988 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:33.343950    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f169e28-8a0e-4022-80f3-30457725f2e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2\" (UID: \"3f169e28-8a0e-4022-80f3-30457725f2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2"
Sep 15 21:36:33.344825 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:33.344703    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f169e28-8a0e-4022-80f3-30457725f2e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2\" (UID: \"3f169e28-8a0e-4022-80f3-30457725f2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2"
Sep 15 21:36:33.531083 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:33.531059    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2"
Sep 15 21:36:33.838055 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:33.837787    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2"]
Sep 15 21:36:33.840570 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:36:33.840542    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f169e28_8a0e_4022_80f3_30457725f2e7.slice/crio-10015e200c8ecc6c48e7c8424b5bcf77e25cede3f6639af1cdbe57fcfce42111 WatchSource:0}: Error finding container 10015e200c8ecc6c48e7c8424b5bcf77e25cede3f6639af1cdbe57fcfce42111: Status 404 returned error can't find the container with id 10015e200c8ecc6c48e7c8424b5bcf77e25cede3f6639af1cdbe57fcfce42111
Sep 15 21:36:34.446777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:34.446689    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2" event={"ID":"3f169e28-8a0e-4022-80f3-30457725f2e7","Type":"ContainerStarted","Data":"5ba0d9d833650b99987b38823dca6b4182d39eb287665c21939a5fe56f2c8b95"}
Sep 15 21:36:34.446777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:34.446744    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2" event={"ID":"3f169e28-8a0e-4022-80f3-30457725f2e7","Type":"ContainerStarted","Data":"10015e200c8ecc6c48e7c8424b5bcf77e25cede3f6639af1cdbe57fcfce42111"}
Sep 15 21:36:34.465179 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:34.465127    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2" podStartSLOduration=1.465107927 podStartE2EDuration="1.465107927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:34.462876824 +0000 UTC m=+1562.324810626" watchObservedRunningTime="2026-09-15 21:36:34.465107927 +0000 UTC m=+1562.327041733"
Sep 15 21:36:35.457017 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:35.456975    2584 generic.go:356] "Generic (PLEG): container finished" podID="3f169e28-8a0e-4022-80f3-30457725f2e7" containerID="5ba0d9d833650b99987b38823dca6b4182d39eb287665c21939a5fe56f2c8b95" exitCode=0
Sep 15 21:36:35.457500 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:35.457185    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2" event={"ID":"3f169e28-8a0e-4022-80f3-30457725f2e7","Type":"ContainerDied","Data":"5ba0d9d833650b99987b38823dca6b4182d39eb287665c21939a5fe56f2c8b95"}
Sep 15 21:36:36.642837 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:36.642815    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2"
Sep 15 21:36:36.700916 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:36.700888    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f169e28-8a0e-4022-80f3-30457725f2e7-maas-api-service-ca\") pod \"3f169e28-8a0e-4022-80f3-30457725f2e7\" (UID: \"3f169e28-8a0e-4022-80f3-30457725f2e7\") "
Sep 15 21:36:36.701351 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:36.701325    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f169e28-8a0e-4022-80f3-30457725f2e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f169e28-8a0e-4022-80f3-30457725f2e7" (UID: "3f169e28-8a0e-4022-80f3-30457725f2e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:36:36.802097 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:36.802061    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f169e28-8a0e-4022-80f3-30457725f2e7-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:36:37.478548 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:37.478521    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2"
Sep 15 21:36:37.478832 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:37.478563    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2" event={"ID":"3f169e28-8a0e-4022-80f3-30457725f2e7","Type":"ContainerDied","Data":"10015e200c8ecc6c48e7c8424b5bcf77e25cede3f6639af1cdbe57fcfce42111"}
Sep 15 21:36:37.478920 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:37.478858    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10015e200c8ecc6c48e7c8424b5bcf77e25cede3f6639af1cdbe57fcfce42111"
Sep 15 21:36:40.224579 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:40.224538    2584 patch_prober.go:28] interesting pod/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 21:36:40.225126 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:40.224642    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5" podUID="41daf450-ffee-4338-99f0-bc55135c42d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 21:36:41.507027 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:41.506991    2584 generic.go:356] "Generic (PLEG): container finished" podID="41daf450-ffee-4338-99f0-bc55135c42d9" containerID="e25e7c2088b134b68eb09c22cd545d343d9747e5f813d45fb815a9c725e241a5" exitCode=0
Sep 15 21:36:41.507481 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:41.507059    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5" event={"ID":"41daf450-ffee-4338-99f0-bc55135c42d9","Type":"ContainerDied","Data":"e25e7c2088b134b68eb09c22cd545d343d9747e5f813d45fb815a9c725e241a5"}
Sep 15 21:36:41.507481 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:41.507473    2584 scope.go:117] "RemoveContainer" containerID="e25e7c2088b134b68eb09c22cd545d343d9747e5f813d45fb815a9c725e241a5"
Sep 15 21:36:41.508737 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:41.508711    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:36:41.508813 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:41.508775    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:36:41.508813 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:41.508806    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:36:42.519185 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:42.519151    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5" event={"ID":"41daf450-ffee-4338-99f0-bc55135c42d9","Type":"ContainerStarted","Data":"04749c0a6a3621cb42edc69ac660b37e66fae055dcae3d432d72ccf7d6fc24cc"}
Sep 15 21:36:43.364090 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:43.364057    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:36:43.364436 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:43.364369    2584 patch_prober.go:28] interesting pod/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 15 21:36:43.364549 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:43.364465    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5" podUID="41daf450-ffee-4338-99f0-bc55135c42d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 15 21:36:44.364786 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:44.364318    2584 patch_prober.go:28] interesting pod/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 15 21:36:44.364786 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:44.364401    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5" podUID="41daf450-ffee-4338-99f0-bc55135c42d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 15 21:36:45.364524 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:45.364482    2584 patch_prober.go:28] interesting pod/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 15 21:36:45.364748 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:45.364546    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5" podUID="41daf450-ffee-4338-99f0-bc55135c42d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 15 21:36:46.363906 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:46.363869    2584 patch_prober.go:28] interesting pod/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 15 21:36:46.364502 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:46.364468    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5" podUID="41daf450-ffee-4338-99f0-bc55135c42d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 15 21:36:47.368318 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:47.368286    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:36:47.368809 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:47.368664    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:36:47.369354 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:47.369334    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:36:49.046144 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.046107    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"]
Sep 15 21:36:49.046655 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.046361    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx" podUID="9a6da525-c9a4-4161-85ec-85e06da90622" containerName="payload-processing" containerID="cri-o://0a27d80d49178860a8efa8aff944c4f7a435b26e7c17baec44f30fdc0732abe0" gracePeriod=30
Sep 15 21:36:49.310141 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.310118    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2"]
Sep 15 21:36:49.315869 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.315849    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d8f9d1-2h2x2"]
Sep 15 21:36:49.331824 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.331807    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"
Sep 15 21:36:49.446617 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.446586    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk4ck\" (UniqueName: \"kubernetes.io/projected/9a6da525-c9a4-4161-85ec-85e06da90622-kube-api-access-mk4ck\") pod \"9a6da525-c9a4-4161-85ec-85e06da90622\" (UID: \"9a6da525-c9a4-4161-85ec-85e06da90622\") "
Sep 15 21:36:49.446617 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.446624    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6da525-c9a4-4161-85ec-85e06da90622-plugins-config-volume\") pod \"9a6da525-c9a4-4161-85ec-85e06da90622\" (UID: \"9a6da525-c9a4-4161-85ec-85e06da90622\") "
Sep 15 21:36:49.447049 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.447016    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6da525-c9a4-4161-85ec-85e06da90622-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9a6da525-c9a4-4161-85ec-85e06da90622" (UID: "9a6da525-c9a4-4161-85ec-85e06da90622"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:36:49.448847 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.448815    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6da525-c9a4-4161-85ec-85e06da90622-kube-api-access-mk4ck" (OuterVolumeSpecName: "kube-api-access-mk4ck") pod "9a6da525-c9a4-4161-85ec-85e06da90622" (UID: "9a6da525-c9a4-4161-85ec-85e06da90622"). InnerVolumeSpecName "kube-api-access-mk4ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:36:49.547789 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.547764    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk4ck\" (UniqueName: \"kubernetes.io/projected/9a6da525-c9a4-4161-85ec-85e06da90622-kube-api-access-mk4ck\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:36:49.547789 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.547785    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6da525-c9a4-4161-85ec-85e06da90622-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:36:49.564358 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.564334    2584 generic.go:356] "Generic (PLEG): container finished" podID="9a6da525-c9a4-4161-85ec-85e06da90622" containerID="0a27d80d49178860a8efa8aff944c4f7a435b26e7c17baec44f30fdc0732abe0" exitCode=0
Sep 15 21:36:49.564496 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.564373    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx" event={"ID":"9a6da525-c9a4-4161-85ec-85e06da90622","Type":"ContainerDied","Data":"0a27d80d49178860a8efa8aff944c4f7a435b26e7c17baec44f30fdc0732abe0"}
Sep 15 21:36:49.564496 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.564446    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx" event={"ID":"9a6da525-c9a4-4161-85ec-85e06da90622","Type":"ContainerDied","Data":"b7ae64efc886ada8bb885db627d1753d7bbd068305814f3ce6fffd43a2debb63"}
Sep 15 21:36:49.564496 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.564480    2584 scope.go:117] "RemoveContainer" containerID="0a27d80d49178860a8efa8aff944c4f7a435b26e7c17baec44f30fdc0732abe0"
Sep 15 21:36:49.564496 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.564485    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"
Sep 15 21:36:49.574130 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.574109    2584 scope.go:117] "RemoveContainer" containerID="0a27d80d49178860a8efa8aff944c4f7a435b26e7c17baec44f30fdc0732abe0"
Sep 15 21:36:49.574523 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:36:49.574496    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a27d80d49178860a8efa8aff944c4f7a435b26e7c17baec44f30fdc0732abe0\": container with ID starting with 0a27d80d49178860a8efa8aff944c4f7a435b26e7c17baec44f30fdc0732abe0 not found: ID does not exist" containerID="0a27d80d49178860a8efa8aff944c4f7a435b26e7c17baec44f30fdc0732abe0"
Sep 15 21:36:49.574697 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.574533    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a27d80d49178860a8efa8aff944c4f7a435b26e7c17baec44f30fdc0732abe0"} err="failed to get container status \"0a27d80d49178860a8efa8aff944c4f7a435b26e7c17baec44f30fdc0732abe0\": rpc error: code = NotFound desc = could not find container \"0a27d80d49178860a8efa8aff944c4f7a435b26e7c17baec44f30fdc0732abe0\": container with ID starting with 0a27d80d49178860a8efa8aff944c4f7a435b26e7c17baec44f30fdc0732abe0 not found: ID does not exist"
Sep 15 21:36:49.589184 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.589159    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"]
Sep 15 21:36:49.593618 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:49.593600    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d8f9d1-5c9bc76d7-5r8gx"]
Sep 15 21:36:50.573223 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:50.573190    2584 generic.go:356] "Generic (PLEG): container finished" podID="fe5a30ee-14a2-4dec-a16b-2a44bd005c6b" containerID="5acdc19d015248d4a85932fa6d46c5088cad6b7f1e9f9c373cdba0d220b9c7ce" exitCode=137
Sep 15 21:36:50.573655 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:50.573267    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg" event={"ID":"fe5a30ee-14a2-4dec-a16b-2a44bd005c6b","Type":"ContainerDied","Data":"5acdc19d015248d4a85932fa6d46c5088cad6b7f1e9f9c373cdba0d220b9c7ce"}
Sep 15 21:36:50.573729 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:50.573698    2584 scope.go:117] "RemoveContainer" containerID="5acdc19d015248d4a85932fa6d46c5088cad6b7f1e9f9c373cdba0d220b9c7ce"
Sep 15 21:36:50.674624 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:50.674600    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f169e28-8a0e-4022-80f3-30457725f2e7" path="/var/lib/kubelet/pods/3f169e28-8a0e-4022-80f3-30457725f2e7/volumes"
Sep 15 21:36:50.674980 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:50.674967    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a6da525-c9a4-4161-85ec-85e06da90622" path="/var/lib/kubelet/pods/9a6da525-c9a4-4161-85ec-85e06da90622/volumes"
Sep 15 21:36:51.582006 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:51.581972    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg" event={"ID":"fe5a30ee-14a2-4dec-a16b-2a44bd005c6b","Type":"ContainerStarted","Data":"9df2f00b2b99d5bc3c8b2d1fcb12e67320925357e691a67676ac6e97e2f06fa1"}
Sep 15 21:36:51.582372 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:51.582175    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"
Sep 15 21:36:57.310179 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:57.310143    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-119834-glfcj"]
Sep 15 21:36:57.310666 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:57.310651    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f169e28-8a0e-4022-80f3-30457725f2e7" containerName="revoke-keys"
Sep 15 21:36:57.310666 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:57.310667    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f169e28-8a0e-4022-80f3-30457725f2e7" containerName="revoke-keys"
Sep 15 21:36:57.310741 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:57.310686    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a6da525-c9a4-4161-85ec-85e06da90622" containerName="payload-processing"
Sep 15 21:36:57.310741 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:57.310692    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a6da525-c9a4-4161-85ec-85e06da90622" containerName="payload-processing"
Sep 15 21:36:57.310805 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:57.310768    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f169e28-8a0e-4022-80f3-30457725f2e7" containerName="revoke-keys"
Sep 15 21:36:57.310805 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:57.310777    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a6da525-c9a4-4161-85ec-85e06da90622" containerName="payload-processing"
Sep 15 21:36:57.314908 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:57.314891    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-119834-glfcj"
Sep 15 21:36:57.317780 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:57.317757    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:36:57.317910 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:57.317764    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:36:57.320645 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:57.320621    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-119834-glfcj"]
Sep 15 21:36:57.324940 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:57.324915    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35c893a3-2681-42e4-97dc-04d8e91cdb65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-119834-glfcj\" (UID: \"35c893a3-2681-42e4-97dc-04d8e91cdb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-119834-glfcj"
Sep 15 21:36:57.425676 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:57.425651    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35c893a3-2681-42e4-97dc-04d8e91cdb65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-119834-glfcj\" (UID: \"35c893a3-2681-42e4-97dc-04d8e91cdb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-119834-glfcj"
Sep 15 21:36:57.426228 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:57.426209    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35c893a3-2681-42e4-97dc-04d8e91cdb65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-119834-glfcj\" (UID: \"35c893a3-2681-42e4-97dc-04d8e91cdb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-119834-glfcj"
Sep 15 21:36:57.627540 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:57.627456    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-119834-glfcj"
Sep 15 21:36:57.899421 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:57.899395    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-119834-glfcj"]
Sep 15 21:36:57.900921 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:36:57.900894    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c893a3_2681_42e4_97dc_04d8e91cdb65.slice/crio-af9ce94f15ae019d887906c484c9a2c6b0ecb23bacd3076f555b151c96853f31 WatchSource:0}: Error finding container af9ce94f15ae019d887906c484c9a2c6b0ecb23bacd3076f555b151c96853f31: Status 404 returned error can't find the container with id af9ce94f15ae019d887906c484c9a2c6b0ecb23bacd3076f555b151c96853f31
Sep 15 21:36:58.625338 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:58.625308    2584 generic.go:356] "Generic (PLEG): container finished" podID="35c893a3-2681-42e4-97dc-04d8e91cdb65" containerID="65d15d45e3def5c795d134859addffaaf53708e05d044ea5f05ccd57ccd1a3b7" exitCode=0
Sep 15 21:36:58.625745 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:58.625409    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-119834-glfcj" event={"ID":"35c893a3-2681-42e4-97dc-04d8e91cdb65","Type":"ContainerDied","Data":"65d15d45e3def5c795d134859addffaaf53708e05d044ea5f05ccd57ccd1a3b7"}
Sep 15 21:36:58.625745 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:58.625460    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-119834-glfcj" event={"ID":"35c893a3-2681-42e4-97dc-04d8e91cdb65","Type":"ContainerStarted","Data":"af9ce94f15ae019d887906c484c9a2c6b0ecb23bacd3076f555b151c96853f31"}
Sep 15 21:36:59.773657 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:59.773631    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-119834-glfcj"
Sep 15 21:36:59.850783 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:59.850750    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35c893a3-2681-42e4-97dc-04d8e91cdb65-maas-api-service-ca\") pod \"35c893a3-2681-42e4-97dc-04d8e91cdb65\" (UID: \"35c893a3-2681-42e4-97dc-04d8e91cdb65\") "
Sep 15 21:36:59.851072 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:59.851051    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c893a3-2681-42e4-97dc-04d8e91cdb65-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "35c893a3-2681-42e4-97dc-04d8e91cdb65" (UID: "35c893a3-2681-42e4-97dc-04d8e91cdb65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:36:59.952402 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:36:59.952313    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35c893a3-2681-42e4-97dc-04d8e91cdb65-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:00.638512 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:00.638481    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-119834-glfcj"
Sep 15 21:37:00.638678 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:00.638524    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-119834-glfcj" event={"ID":"35c893a3-2681-42e4-97dc-04d8e91cdb65","Type":"ContainerDied","Data":"af9ce94f15ae019d887906c484c9a2c6b0ecb23bacd3076f555b151c96853f31"}
Sep 15 21:37:00.638678 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:00.638608    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af9ce94f15ae019d887906c484c9a2c6b0ecb23bacd3076f555b151c96853f31"
Sep 15 21:37:02.590899 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:02.590870    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"
Sep 15 21:37:13.148200 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:13.148169    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-119834-glfcj"]
Sep 15 21:37:13.150637 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:13.150614    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-119834-glfcj"]
Sep 15 21:37:14.674581 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:14.674553    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35c893a3-2681-42e4-97dc-04d8e91cdb65" path="/var/lib/kubelet/pods/35c893a3-2681-42e4-97dc-04d8e91cdb65/volumes"
Sep 15 21:37:20.020107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:20.020071    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"]
Sep 15 21:37:20.020570 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:20.020288    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5" podUID="41daf450-ffee-4338-99f0-bc55135c42d9" containerName="istio-proxy" containerID="cri-o://04749c0a6a3621cb42edc69ac660b37e66fae055dcae3d432d72ccf7d6fc24cc" gracePeriod=30
Sep 15 21:37:20.385053 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:20.385019    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r"]
Sep 15 21:37:20.385553 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:20.385539    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35c893a3-2681-42e4-97dc-04d8e91cdb65" containerName="revoke-keys"
Sep 15 21:37:20.385604 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:20.385555    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="35c893a3-2681-42e4-97dc-04d8e91cdb65" containerName="revoke-keys"
Sep 15 21:37:20.385640 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:20.385631    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="35c893a3-2681-42e4-97dc-04d8e91cdb65" containerName="revoke-keys"
Sep 15 21:37:20.388783 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:20.388766    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r"
Sep 15 21:37:20.391000 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:20.390970    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:37:20.391110 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:20.391071    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:37:20.397093 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:20.397071    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r"]
Sep 15 21:37:20.453977 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:20.453947    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c4e952-8205-4e6a-80ca-c988caa9fffc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r\" (UID: \"55c4e952-8205-4e6a-80ca-c988caa9fffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r"
Sep 15 21:37:20.555170 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:20.555129    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c4e952-8205-4e6a-80ca-c988caa9fffc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r\" (UID: \"55c4e952-8205-4e6a-80ca-c988caa9fffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r"
Sep 15 21:37:20.556018 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:20.555986    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c4e952-8205-4e6a-80ca-c988caa9fffc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r\" (UID: \"55c4e952-8205-4e6a-80ca-c988caa9fffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r"
Sep 15 21:37:20.702131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:20.702058    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r"
Sep 15 21:37:21.008551 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:21.008513    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r"]
Sep 15 21:37:21.011838 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:37:21.011807    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c4e952_8205_4e6a_80ca_c988caa9fffc.slice/crio-3e9e5ea733eabc9f06783024c2e8ae228812e1afff2847ed3a437439c4eeced7 WatchSource:0}: Error finding container 3e9e5ea733eabc9f06783024c2e8ae228812e1afff2847ed3a437439c4eeced7: Status 404 returned error can't find the container with id 3e9e5ea733eabc9f06783024c2e8ae228812e1afff2847ed3a437439c4eeced7
Sep 15 21:37:21.779022 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:21.778989    2584 generic.go:356] "Generic (PLEG): container finished" podID="55c4e952-8205-4e6a-80ca-c988caa9fffc" containerID="91e7316b32110cf4e89fe011a79fdfc586c8ecf4560670097580a8de8ccb0ee4" exitCode=0
Sep 15 21:37:21.779424 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:21.779026    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r" event={"ID":"55c4e952-8205-4e6a-80ca-c988caa9fffc","Type":"ContainerDied","Data":"91e7316b32110cf4e89fe011a79fdfc586c8ecf4560670097580a8de8ccb0ee4"}
Sep 15 21:37:21.779424 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:21.779091    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r" event={"ID":"55c4e952-8205-4e6a-80ca-c988caa9fffc","Type":"ContainerStarted","Data":"3e9e5ea733eabc9f06783024c2e8ae228812e1afff2847ed3a437439c4eeced7"}
Sep 15 21:37:22.932651 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:22.932627    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r"
Sep 15 21:37:23.085803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:23.085727    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c4e952-8205-4e6a-80ca-c988caa9fffc-maas-api-service-ca\") pod \"55c4e952-8205-4e6a-80ca-c988caa9fffc\" (UID: \"55c4e952-8205-4e6a-80ca-c988caa9fffc\") "
Sep 15 21:37:23.086088 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:23.086063    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c4e952-8205-4e6a-80ca-c988caa9fffc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55c4e952-8205-4e6a-80ca-c988caa9fffc" (UID: "55c4e952-8205-4e6a-80ca-c988caa9fffc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:23.086200 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:23.086183    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c4e952-8205-4e6a-80ca-c988caa9fffc-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:23.792450 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:23.792420    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r"
Sep 15 21:37:23.792450 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:23.792438    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r" event={"ID":"55c4e952-8205-4e6a-80ca-c988caa9fffc","Type":"ContainerDied","Data":"3e9e5ea733eabc9f06783024c2e8ae228812e1afff2847ed3a437439c4eeced7"}
Sep 15 21:37:23.792665 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:23.792490    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e9e5ea733eabc9f06783024c2e8ae228812e1afff2847ed3a437439c4eeced7"
Sep 15 21:37:25.284668 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.284586    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:37:25.408341 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.408310    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-istio-data\") pod \"41daf450-ffee-4338-99f0-bc55135c42d9\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") "
Sep 15 21:37:25.408509 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.408366    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41daf450-ffee-4338-99f0-bc55135c42d9-istiod-ca-cert\") pod \"41daf450-ffee-4338-99f0-bc55135c42d9\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") "
Sep 15 21:37:25.408509 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.408420    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41daf450-ffee-4338-99f0-bc55135c42d9-istio-token\") pod \"41daf450-ffee-4338-99f0-bc55135c42d9\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") "
Sep 15 21:37:25.408509 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.408495    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9wnk\" (UniqueName: \"kubernetes.io/projected/41daf450-ffee-4338-99f0-bc55135c42d9-kube-api-access-v9wnk\") pod \"41daf450-ffee-4338-99f0-bc55135c42d9\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") "
Sep 15 21:37:25.408648 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.408515    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-istio-data" (OuterVolumeSpecName: "istio-data") pod "41daf450-ffee-4338-99f0-bc55135c42d9" (UID: "41daf450-ffee-4338-99f0-bc55135c42d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:37:25.408648 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.408524    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-workload-socket\") pod \"41daf450-ffee-4338-99f0-bc55135c42d9\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") "
Sep 15 21:37:25.408743 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.408653    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41daf450-ffee-4338-99f0-bc55135c42d9-istio-podinfo\") pod \"41daf450-ffee-4338-99f0-bc55135c42d9\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") "
Sep 15 21:37:25.408743 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.408703    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-workload-certs\") pod \"41daf450-ffee-4338-99f0-bc55135c42d9\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") "
Sep 15 21:37:25.408820 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.408779    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "41daf450-ffee-4338-99f0-bc55135c42d9" (UID: "41daf450-ffee-4338-99f0-bc55135c42d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:37:25.408873 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.408822    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41daf450-ffee-4338-99f0-bc55135c42d9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "41daf450-ffee-4338-99f0-bc55135c42d9" (UID: "41daf450-ffee-4338-99f0-bc55135c42d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:25.408873 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.408846    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-istio-envoy\") pod \"41daf450-ffee-4338-99f0-bc55135c42d9\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") "
Sep 15 21:37:25.408980 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.408913    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-credential-socket\") pod \"41daf450-ffee-4338-99f0-bc55135c42d9\" (UID: \"41daf450-ffee-4338-99f0-bc55135c42d9\") "
Sep 15 21:37:25.408980 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.408937    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "41daf450-ffee-4338-99f0-bc55135c42d9" (UID: "41daf450-ffee-4338-99f0-bc55135c42d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:37:25.409223 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.409196    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "41daf450-ffee-4338-99f0-bc55135c42d9" (UID: "41daf450-ffee-4338-99f0-bc55135c42d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:37:25.409359 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.409350    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41daf450-ffee-4338-99f0-bc55135c42d9-istiod-ca-cert\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:25.409458 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.409367    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-workload-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:25.409458 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.409397    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-workload-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:25.409458 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.409411    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-credential-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:25.409458 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.409424    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-istio-data\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:25.410707 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.410684    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41daf450-ffee-4338-99f0-bc55135c42d9-istio-token" (OuterVolumeSpecName: "istio-token") pod "41daf450-ffee-4338-99f0-bc55135c42d9" (UID: "41daf450-ffee-4338-99f0-bc55135c42d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:25.410824 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.410771    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41daf450-ffee-4338-99f0-bc55135c42d9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "41daf450-ffee-4338-99f0-bc55135c42d9" (UID: "41daf450-ffee-4338-99f0-bc55135c42d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:37:25.410824 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.410785    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "41daf450-ffee-4338-99f0-bc55135c42d9" (UID: "41daf450-ffee-4338-99f0-bc55135c42d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:37:25.410945 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.410849    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41daf450-ffee-4338-99f0-bc55135c42d9-kube-api-access-v9wnk" (OuterVolumeSpecName: "kube-api-access-v9wnk") pod "41daf450-ffee-4338-99f0-bc55135c42d9" (UID: "41daf450-ffee-4338-99f0-bc55135c42d9"). InnerVolumeSpecName "kube-api-access-v9wnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:25.510245 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.510216    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9wnk\" (UniqueName: \"kubernetes.io/projected/41daf450-ffee-4338-99f0-bc55135c42d9-kube-api-access-v9wnk\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:25.510245 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.510241    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41daf450-ffee-4338-99f0-bc55135c42d9-istio-podinfo\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:25.510245 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.510250    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41daf450-ffee-4338-99f0-bc55135c42d9-istio-envoy\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:25.510459 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.510258    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41daf450-ffee-4338-99f0-bc55135c42d9-istio-token\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:25.806439 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.806410    2584 generic.go:356] "Generic (PLEG): container finished" podID="41daf450-ffee-4338-99f0-bc55135c42d9" containerID="04749c0a6a3621cb42edc69ac660b37e66fae055dcae3d432d72ccf7d6fc24cc" exitCode=0
Sep 15 21:37:25.806595 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.806453    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5" event={"ID":"41daf450-ffee-4338-99f0-bc55135c42d9","Type":"ContainerDied","Data":"04749c0a6a3621cb42edc69ac660b37e66fae055dcae3d432d72ccf7d6fc24cc"}
Sep 15 21:37:25.806595 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.806496    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5" event={"ID":"41daf450-ffee-4338-99f0-bc55135c42d9","Type":"ContainerDied","Data":"0f16abcd98f0f8cb63f217f94cc92ab31fc70d8fa5b3872765f054aeb61e51de"}
Sep 15 21:37:25.806595 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.806507    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"
Sep 15 21:37:25.806595 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.806517    2584 scope.go:117] "RemoveContainer" containerID="04749c0a6a3621cb42edc69ac660b37e66fae055dcae3d432d72ccf7d6fc24cc"
Sep 15 21:37:25.817321 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.817301    2584 scope.go:117] "RemoveContainer" containerID="e25e7c2088b134b68eb09c22cd545d343d9747e5f813d45fb815a9c725e241a5"
Sep 15 21:37:25.826590 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.826571    2584 scope.go:117] "RemoveContainer" containerID="04749c0a6a3621cb42edc69ac660b37e66fae055dcae3d432d72ccf7d6fc24cc"
Sep 15 21:37:25.826877 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:37:25.826857    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04749c0a6a3621cb42edc69ac660b37e66fae055dcae3d432d72ccf7d6fc24cc\": container with ID starting with 04749c0a6a3621cb42edc69ac660b37e66fae055dcae3d432d72ccf7d6fc24cc not found: ID does not exist" containerID="04749c0a6a3621cb42edc69ac660b37e66fae055dcae3d432d72ccf7d6fc24cc"
Sep 15 21:37:25.826931 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.826886    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04749c0a6a3621cb42edc69ac660b37e66fae055dcae3d432d72ccf7d6fc24cc"} err="failed to get container status \"04749c0a6a3621cb42edc69ac660b37e66fae055dcae3d432d72ccf7d6fc24cc\": rpc error: code = NotFound desc = could not find container \"04749c0a6a3621cb42edc69ac660b37e66fae055dcae3d432d72ccf7d6fc24cc\": container with ID starting with 04749c0a6a3621cb42edc69ac660b37e66fae055dcae3d432d72ccf7d6fc24cc not found: ID does not exist"
Sep 15 21:37:25.826931 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.826902    2584 scope.go:117] "RemoveContainer" containerID="e25e7c2088b134b68eb09c22cd545d343d9747e5f813d45fb815a9c725e241a5"
Sep 15 21:37:25.827170 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:37:25.827148    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25e7c2088b134b68eb09c22cd545d343d9747e5f813d45fb815a9c725e241a5\": container with ID starting with e25e7c2088b134b68eb09c22cd545d343d9747e5f813d45fb815a9c725e241a5 not found: ID does not exist" containerID="e25e7c2088b134b68eb09c22cd545d343d9747e5f813d45fb815a9c725e241a5"
Sep 15 21:37:25.827236 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.827180    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25e7c2088b134b68eb09c22cd545d343d9747e5f813d45fb815a9c725e241a5"} err="failed to get container status \"e25e7c2088b134b68eb09c22cd545d343d9747e5f813d45fb815a9c725e241a5\": rpc error: code = NotFound desc = could not find container \"e25e7c2088b134b68eb09c22cd545d343d9747e5f813d45fb815a9c725e241a5\": container with ID starting with e25e7c2088b134b68eb09c22cd545d343d9747e5f813d45fb815a9c725e241a5 not found: ID does not exist"
Sep 15 21:37:25.832623 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.832594    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"]
Sep 15 21:37:25.836885 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:25.836864    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-119834-openshift-default-694cb45d4b-t26m5"]
Sep 15 21:37:26.673918 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:26.673877    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41daf450-ffee-4338-99f0-bc55135c42d9" path="/var/lib/kubelet/pods/41daf450-ffee-4338-99f0-bc55135c42d9/volumes"
Sep 15 21:37:31.065110 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.065070    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"]
Sep 15 21:37:31.065828 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.065797    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627" podUID="1986b9b7-1c49-455a-b608-39889c14d670" containerName="payload-processing" containerID="cri-o://a06ed5fc48ab806b83098681d30f644ce8c7f47b2dabc4beed531295822f4c22" gracePeriod=30
Sep 15 21:37:31.069726 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.069695    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"]
Sep 15 21:37:31.070157 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.069887    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2" podUID="915666e3-94e8-4820-ba19-b25b136f686b" containerName="payload-pre-processing" containerID="cri-o://d7ce632f8770208e0ed41170f1aedc66b08449090436d806e59c029e7bc44e9e" gracePeriod=30
Sep 15 21:37:31.244087 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.244054    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r"]
Sep 15 21:37:31.248013 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.247993    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dbcc8f-cxk5r"]
Sep 15 21:37:31.363306 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.363286    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"
Sep 15 21:37:31.366006 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.365984    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"
Sep 15 21:37:31.376341 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.376317    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrkm4\" (UniqueName: \"kubernetes.io/projected/915666e3-94e8-4820-ba19-b25b136f686b-kube-api-access-xrkm4\") pod \"915666e3-94e8-4820-ba19-b25b136f686b\" (UID: \"915666e3-94e8-4820-ba19-b25b136f686b\") "
Sep 15 21:37:31.376449 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.376359    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915666e3-94e8-4820-ba19-b25b136f686b-plugins-config-volume\") pod \"915666e3-94e8-4820-ba19-b25b136f686b\" (UID: \"915666e3-94e8-4820-ba19-b25b136f686b\") "
Sep 15 21:37:31.376449 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.376416    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2qlw\" (UniqueName: \"kubernetes.io/projected/1986b9b7-1c49-455a-b608-39889c14d670-kube-api-access-t2qlw\") pod \"1986b9b7-1c49-455a-b608-39889c14d670\" (UID: \"1986b9b7-1c49-455a-b608-39889c14d670\") "
Sep 15 21:37:31.376449 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.376447    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1986b9b7-1c49-455a-b608-39889c14d670-plugins-config-volume\") pod \"1986b9b7-1c49-455a-b608-39889c14d670\" (UID: \"1986b9b7-1c49-455a-b608-39889c14d670\") "
Sep 15 21:37:31.376787 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.376762    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/915666e3-94e8-4820-ba19-b25b136f686b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "915666e3-94e8-4820-ba19-b25b136f686b" (UID: "915666e3-94e8-4820-ba19-b25b136f686b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:31.376859 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.376810    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1986b9b7-1c49-455a-b608-39889c14d670-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1986b9b7-1c49-455a-b608-39889c14d670" (UID: "1986b9b7-1c49-455a-b608-39889c14d670"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:31.378392 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.378360    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1986b9b7-1c49-455a-b608-39889c14d670-kube-api-access-t2qlw" (OuterVolumeSpecName: "kube-api-access-t2qlw") pod "1986b9b7-1c49-455a-b608-39889c14d670" (UID: "1986b9b7-1c49-455a-b608-39889c14d670"). InnerVolumeSpecName "kube-api-access-t2qlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:31.378476 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.378396    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915666e3-94e8-4820-ba19-b25b136f686b-kube-api-access-xrkm4" (OuterVolumeSpecName: "kube-api-access-xrkm4") pod "915666e3-94e8-4820-ba19-b25b136f686b" (UID: "915666e3-94e8-4820-ba19-b25b136f686b"). InnerVolumeSpecName "kube-api-access-xrkm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:31.477327 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.477290    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2qlw\" (UniqueName: \"kubernetes.io/projected/1986b9b7-1c49-455a-b608-39889c14d670-kube-api-access-t2qlw\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:31.477327 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.477325    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1986b9b7-1c49-455a-b608-39889c14d670-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:31.477327 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.477334    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrkm4\" (UniqueName: \"kubernetes.io/projected/915666e3-94e8-4820-ba19-b25b136f686b-kube-api-access-xrkm4\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:31.477595 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.477343    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915666e3-94e8-4820-ba19-b25b136f686b-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:31.846513 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.846477    2584 generic.go:356] "Generic (PLEG): container finished" podID="1986b9b7-1c49-455a-b608-39889c14d670" containerID="a06ed5fc48ab806b83098681d30f644ce8c7f47b2dabc4beed531295822f4c22" exitCode=0
Sep 15 21:37:31.846704 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.846520    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"
Sep 15 21:37:31.846704 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.846561    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627" event={"ID":"1986b9b7-1c49-455a-b608-39889c14d670","Type":"ContainerDied","Data":"a06ed5fc48ab806b83098681d30f644ce8c7f47b2dabc4beed531295822f4c22"}
Sep 15 21:37:31.846704 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.846606    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627" event={"ID":"1986b9b7-1c49-455a-b608-39889c14d670","Type":"ContainerDied","Data":"641c5eab1e56dd784a7189a2f28be8f2f95c1b1aa8f01c115ddd540993cef99d"}
Sep 15 21:37:31.846704 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.846632    2584 scope.go:117] "RemoveContainer" containerID="a06ed5fc48ab806b83098681d30f644ce8c7f47b2dabc4beed531295822f4c22"
Sep 15 21:37:31.848001 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.847978    2584 generic.go:356] "Generic (PLEG): container finished" podID="915666e3-94e8-4820-ba19-b25b136f686b" containerID="d7ce632f8770208e0ed41170f1aedc66b08449090436d806e59c029e7bc44e9e" exitCode=0
Sep 15 21:37:31.848127 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.848014    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2" event={"ID":"915666e3-94e8-4820-ba19-b25b136f686b","Type":"ContainerDied","Data":"d7ce632f8770208e0ed41170f1aedc66b08449090436d806e59c029e7bc44e9e"}
Sep 15 21:37:31.848127 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.848029    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"
Sep 15 21:37:31.848127 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.848052    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2" event={"ID":"915666e3-94e8-4820-ba19-b25b136f686b","Type":"ContainerDied","Data":"52678bcb68a606a11efb70d96a74e92037b161510d2295f2c1175302b836eca4"}
Sep 15 21:37:31.857779 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.857755    2584 scope.go:117] "RemoveContainer" containerID="a06ed5fc48ab806b83098681d30f644ce8c7f47b2dabc4beed531295822f4c22"
Sep 15 21:37:31.858294 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:37:31.858052    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06ed5fc48ab806b83098681d30f644ce8c7f47b2dabc4beed531295822f4c22\": container with ID starting with a06ed5fc48ab806b83098681d30f644ce8c7f47b2dabc4beed531295822f4c22 not found: ID does not exist" containerID="a06ed5fc48ab806b83098681d30f644ce8c7f47b2dabc4beed531295822f4c22"
Sep 15 21:37:31.858294 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.858079    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06ed5fc48ab806b83098681d30f644ce8c7f47b2dabc4beed531295822f4c22"} err="failed to get container status \"a06ed5fc48ab806b83098681d30f644ce8c7f47b2dabc4beed531295822f4c22\": rpc error: code = NotFound desc = could not find container \"a06ed5fc48ab806b83098681d30f644ce8c7f47b2dabc4beed531295822f4c22\": container with ID starting with a06ed5fc48ab806b83098681d30f644ce8c7f47b2dabc4beed531295822f4c22 not found: ID does not exist"
Sep 15 21:37:31.858294 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.858095    2584 scope.go:117] "RemoveContainer" containerID="d7ce632f8770208e0ed41170f1aedc66b08449090436d806e59c029e7bc44e9e"
Sep 15 21:37:31.866634 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.866616    2584 scope.go:117] "RemoveContainer" containerID="d7ce632f8770208e0ed41170f1aedc66b08449090436d806e59c029e7bc44e9e"
Sep 15 21:37:31.866889 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:37:31.866873    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ce632f8770208e0ed41170f1aedc66b08449090436d806e59c029e7bc44e9e\": container with ID starting with d7ce632f8770208e0ed41170f1aedc66b08449090436d806e59c029e7bc44e9e not found: ID does not exist" containerID="d7ce632f8770208e0ed41170f1aedc66b08449090436d806e59c029e7bc44e9e"
Sep 15 21:37:31.866936 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.866896    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ce632f8770208e0ed41170f1aedc66b08449090436d806e59c029e7bc44e9e"} err="failed to get container status \"d7ce632f8770208e0ed41170f1aedc66b08449090436d806e59c029e7bc44e9e\": rpc error: code = NotFound desc = could not find container \"d7ce632f8770208e0ed41170f1aedc66b08449090436d806e59c029e7bc44e9e\": container with ID starting with d7ce632f8770208e0ed41170f1aedc66b08449090436d806e59c029e7bc44e9e not found: ID does not exist"
Sep 15 21:37:31.874473 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.874453    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"]
Sep 15 21:37:31.881667 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.881648    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dbcc8f-6c7c57b965-mf627"]
Sep 15 21:37:31.893010 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.892992    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"]
Sep 15 21:37:31.899341 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:31.899318    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbcc8f-78c5d5c586-5mrm2"]
Sep 15 21:37:32.674848 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:32.674814    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1986b9b7-1c49-455a-b608-39889c14d670" path="/var/lib/kubelet/pods/1986b9b7-1c49-455a-b608-39889c14d670/volumes"
Sep 15 21:37:32.675232 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:32.675216    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55c4e952-8205-4e6a-80ca-c988caa9fffc" path="/var/lib/kubelet/pods/55c4e952-8205-4e6a-80ca-c988caa9fffc/volumes"
Sep 15 21:37:32.675603 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:32.675590    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="915666e3-94e8-4820-ba19-b25b136f686b" path="/var/lib/kubelet/pods/915666e3-94e8-4820-ba19-b25b136f686b/volumes"
Sep 15 21:37:32.941140 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:32.941056    2584 scope.go:117] "RemoveContainer" containerID="caadd8f41bf7845b2a0821924522265621c2bb4e31f9e2417c652374d786cc40"
Sep 15 21:37:32.950213 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:32.950190    2584 scope.go:117] "RemoveContainer" containerID="b2d8ef6449d33b6d11281c64f7b23c98b9ef48dc761ceea7c6270583198cfe79"
Sep 15 21:37:38.532434 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.532373    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk"]
Sep 15 21:37:38.533124 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.533102    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c4e952-8205-4e6a-80ca-c988caa9fffc" containerName="revoke-keys"
Sep 15 21:37:38.533220 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.533126    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c4e952-8205-4e6a-80ca-c988caa9fffc" containerName="revoke-keys"
Sep 15 21:37:38.533220 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.533142    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41daf450-ffee-4338-99f0-bc55135c42d9" containerName="istio-proxy"
Sep 15 21:37:38.533220 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.533150    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="41daf450-ffee-4338-99f0-bc55135c42d9" containerName="istio-proxy"
Sep 15 21:37:38.533220 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.533163    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="915666e3-94e8-4820-ba19-b25b136f686b" containerName="payload-pre-processing"
Sep 15 21:37:38.533220 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.533172    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="915666e3-94e8-4820-ba19-b25b136f686b" containerName="payload-pre-processing"
Sep 15 21:37:38.533220 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.533194    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41daf450-ffee-4338-99f0-bc55135c42d9" containerName="istio-proxy"
Sep 15 21:37:38.533220 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.533201    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="41daf450-ffee-4338-99f0-bc55135c42d9" containerName="istio-proxy"
Sep 15 21:37:38.533603 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.533245    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1986b9b7-1c49-455a-b608-39889c14d670" containerName="payload-processing"
Sep 15 21:37:38.533603 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.533255    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1986b9b7-1c49-455a-b608-39889c14d670" containerName="payload-processing"
Sep 15 21:37:38.533603 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.533405    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="41daf450-ffee-4338-99f0-bc55135c42d9" containerName="istio-proxy"
Sep 15 21:37:38.533603 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.533422    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c4e952-8205-4e6a-80ca-c988caa9fffc" containerName="revoke-keys"
Sep 15 21:37:38.533603 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.533432    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="915666e3-94e8-4820-ba19-b25b136f686b" containerName="payload-pre-processing"
Sep 15 21:37:38.533603 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.533444    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1986b9b7-1c49-455a-b608-39889c14d670" containerName="payload-processing"
Sep 15 21:37:38.536940 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.536918    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk"
Sep 15 21:37:38.540164 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.540142    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:37:38.540350 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.540336    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:37:38.546271 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.546250    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk"]
Sep 15 21:37:38.649951 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.649919    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d296f7fe-1892-411b-818c-fbe193e38a7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk\" (UID: \"d296f7fe-1892-411b-818c-fbe193e38a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk"
Sep 15 21:37:38.751332 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.751295    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d296f7fe-1892-411b-818c-fbe193e38a7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk\" (UID: \"d296f7fe-1892-411b-818c-fbe193e38a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk"
Sep 15 21:37:38.751907 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.751892    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d296f7fe-1892-411b-818c-fbe193e38a7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk\" (UID: \"d296f7fe-1892-411b-818c-fbe193e38a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk"
Sep 15 21:37:38.848420 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:38.848320    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk"
Sep 15 21:37:39.126962 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:39.126931    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk"]
Sep 15 21:37:39.128759 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:37:39.128734    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd296f7fe_1892_411b_818c_fbe193e38a7c.slice/crio-0d0bb5140d08f839586cbf1ed6a306420f4136e1d72dea86d01301bc530e0fc5 WatchSource:0}: Error finding container 0d0bb5140d08f839586cbf1ed6a306420f4136e1d72dea86d01301bc530e0fc5: Status 404 returned error can't find the container with id 0d0bb5140d08f839586cbf1ed6a306420f4136e1d72dea86d01301bc530e0fc5
Sep 15 21:37:39.904028 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:39.903993    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk" event={"ID":"d296f7fe-1892-411b-818c-fbe193e38a7c","Type":"ContainerStarted","Data":"796742c7b626bc144c68e2a7f3f2e0054507be6a61812f6929c78c6f72c87052"}
Sep 15 21:37:39.904445 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:39.904048    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk" event={"ID":"d296f7fe-1892-411b-818c-fbe193e38a7c","Type":"ContainerStarted","Data":"0d0bb5140d08f839586cbf1ed6a306420f4136e1d72dea86d01301bc530e0fc5"}
Sep 15 21:37:39.923033 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:39.922967    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk" podStartSLOduration=1.922948361 podStartE2EDuration="1.922948361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:37:39.920375161 +0000 UTC m=+1627.782308967" watchObservedRunningTime="2026-09-15 21:37:39.922948361 +0000 UTC m=+1627.784882168"
Sep 15 21:37:40.145062 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:40.145022    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm"]
Sep 15 21:37:40.145625 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:40.145610    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="41daf450-ffee-4338-99f0-bc55135c42d9" containerName="istio-proxy"
Sep 15 21:37:40.148807 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:40.148788    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm"
Sep 15 21:37:40.157140 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:40.156723    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm"]
Sep 15 21:37:40.266912 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:40.266880    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3965e249-51e9-41fc-a33a-cb8716d8e0bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm\" (UID: \"3965e249-51e9-41fc-a33a-cb8716d8e0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm"
Sep 15 21:37:40.368498 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:40.368453    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3965e249-51e9-41fc-a33a-cb8716d8e0bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm\" (UID: \"3965e249-51e9-41fc-a33a-cb8716d8e0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm"
Sep 15 21:37:40.369123 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:40.369104    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3965e249-51e9-41fc-a33a-cb8716d8e0bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm\" (UID: \"3965e249-51e9-41fc-a33a-cb8716d8e0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm"
Sep 15 21:37:40.461983 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:40.461889    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm"
Sep 15 21:37:40.735116 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:40.735083    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm"]
Sep 15 21:37:40.736702 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:37:40.736670    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3965e249_51e9_41fc_a33a_cb8716d8e0bb.slice/crio-862040eea386ce6e92604bd3d0f957cb6ddde6f5b4d6cd2044fe9a33be3e5235 WatchSource:0}: Error finding container 862040eea386ce6e92604bd3d0f957cb6ddde6f5b4d6cd2044fe9a33be3e5235: Status 404 returned error can't find the container with id 862040eea386ce6e92604bd3d0f957cb6ddde6f5b4d6cd2044fe9a33be3e5235
Sep 15 21:37:40.911795 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:40.911760    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm" event={"ID":"3965e249-51e9-41fc-a33a-cb8716d8e0bb","Type":"ContainerStarted","Data":"862040eea386ce6e92604bd3d0f957cb6ddde6f5b4d6cd2044fe9a33be3e5235"}
Sep 15 21:37:40.913218 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:40.913192    2584 generic.go:356] "Generic (PLEG): container finished" podID="d296f7fe-1892-411b-818c-fbe193e38a7c" containerID="796742c7b626bc144c68e2a7f3f2e0054507be6a61812f6929c78c6f72c87052" exitCode=0
Sep 15 21:37:40.913344 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:40.913281    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk" event={"ID":"d296f7fe-1892-411b-818c-fbe193e38a7c","Type":"ContainerDied","Data":"796742c7b626bc144c68e2a7f3f2e0054507be6a61812f6929c78c6f72c87052"}
Sep 15 21:37:41.920803 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:41.920774    2584 generic.go:356] "Generic (PLEG): container finished" podID="3965e249-51e9-41fc-a33a-cb8716d8e0bb" containerID="ad9e411953681fcf872fa3157c42782ddb14d9936ee11c0b6290f4c72574b65e" exitCode=0
Sep 15 21:37:41.921268 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:41.920846    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm" event={"ID":"3965e249-51e9-41fc-a33a-cb8716d8e0bb","Type":"ContainerDied","Data":"ad9e411953681fcf872fa3157c42782ddb14d9936ee11c0b6290f4c72574b65e"}
Sep 15 21:37:42.071407 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:42.071369    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk"
Sep 15 21:37:42.187963 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:42.187890    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d296f7fe-1892-411b-818c-fbe193e38a7c-maas-api-service-ca\") pod \"d296f7fe-1892-411b-818c-fbe193e38a7c\" (UID: \"d296f7fe-1892-411b-818c-fbe193e38a7c\") "
Sep 15 21:37:42.188276 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:42.188254    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d296f7fe-1892-411b-818c-fbe193e38a7c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d296f7fe-1892-411b-818c-fbe193e38a7c" (UID: "d296f7fe-1892-411b-818c-fbe193e38a7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:42.188391 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:42.188363    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d296f7fe-1892-411b-818c-fbe193e38a7c-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:42.928214 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:42.928182    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk"
Sep 15 21:37:42.928701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:42.928219    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk" event={"ID":"d296f7fe-1892-411b-818c-fbe193e38a7c","Type":"ContainerDied","Data":"0d0bb5140d08f839586cbf1ed6a306420f4136e1d72dea86d01301bc530e0fc5"}
Sep 15 21:37:42.928701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:42.928270    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d0bb5140d08f839586cbf1ed6a306420f4136e1d72dea86d01301bc530e0fc5"
Sep 15 21:37:43.080999 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:43.080977    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm"
Sep 15 21:37:43.199368 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:43.199338    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3965e249-51e9-41fc-a33a-cb8716d8e0bb-maas-api-service-ca\") pod \"3965e249-51e9-41fc-a33a-cb8716d8e0bb\" (UID: \"3965e249-51e9-41fc-a33a-cb8716d8e0bb\") "
Sep 15 21:37:43.199737 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:43.199711    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3965e249-51e9-41fc-a33a-cb8716d8e0bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3965e249-51e9-41fc-a33a-cb8716d8e0bb" (UID: "3965e249-51e9-41fc-a33a-cb8716d8e0bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:43.300352 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:43.300317    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3965e249-51e9-41fc-a33a-cb8716d8e0bb-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:43.942339 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:43.942314    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm"
Sep 15 21:37:43.942762 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:43.942306    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm" event={"ID":"3965e249-51e9-41fc-a33a-cb8716d8e0bb","Type":"ContainerDied","Data":"862040eea386ce6e92604bd3d0f957cb6ddde6f5b4d6cd2044fe9a33be3e5235"}
Sep 15 21:37:43.942762 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:43.942466    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="862040eea386ce6e92604bd3d0f957cb6ddde6f5b4d6cd2044fe9a33be3e5235"
Sep 15 21:37:49.298948 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.298907    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"]
Sep 15 21:37:49.299404 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.299179    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs" podUID="731be20d-8949-42e0-9233-416cbf2a49ac" containerName="payload-processing" containerID="cri-o://498a4ea55b97a9588f8983a21442d4165fb8e71dcaa8cdd0fd7b1d64cf40f27a" gracePeriod=30
Sep 15 21:37:49.323566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.323077    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"]
Sep 15 21:37:49.323566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.323440    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb" podUID="1de5dc48-089c-4bb3-b40a-6b30c490950e" containerName="payload-pre-processing" containerID="cri-o://c617dd1fc7ddc1bd61bc0f477f7bb198f7ac191c1714a6678789cc82e01ce135" gracePeriod=30
Sep 15 21:37:49.465014 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.464979    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk"]
Sep 15 21:37:49.467888 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.467858    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c39135-j6jfk"]
Sep 15 21:37:49.593324 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.593298    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"
Sep 15 21:37:49.613813 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.613784    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"
Sep 15 21:37:49.667848 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.667728    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/731be20d-8949-42e0-9233-416cbf2a49ac-plugins-config-volume\") pod \"731be20d-8949-42e0-9233-416cbf2a49ac\" (UID: \"731be20d-8949-42e0-9233-416cbf2a49ac\") "
Sep 15 21:37:49.668066 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.667886    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz2sh\" (UniqueName: \"kubernetes.io/projected/731be20d-8949-42e0-9233-416cbf2a49ac-kube-api-access-bz2sh\") pod \"731be20d-8949-42e0-9233-416cbf2a49ac\" (UID: \"731be20d-8949-42e0-9233-416cbf2a49ac\") "
Sep 15 21:37:49.668181 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.668150    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/731be20d-8949-42e0-9233-416cbf2a49ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "731be20d-8949-42e0-9233-416cbf2a49ac" (UID: "731be20d-8949-42e0-9233-416cbf2a49ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:49.668351 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.668331    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/731be20d-8949-42e0-9233-416cbf2a49ac-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:49.670435 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.670411    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731be20d-8949-42e0-9233-416cbf2a49ac-kube-api-access-bz2sh" (OuterVolumeSpecName: "kube-api-access-bz2sh") pod "731be20d-8949-42e0-9233-416cbf2a49ac" (UID: "731be20d-8949-42e0-9233-416cbf2a49ac"). InnerVolumeSpecName "kube-api-access-bz2sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:49.769506 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.769475    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de5dc48-089c-4bb3-b40a-6b30c490950e-plugins-config-volume\") pod \"1de5dc48-089c-4bb3-b40a-6b30c490950e\" (UID: \"1de5dc48-089c-4bb3-b40a-6b30c490950e\") "
Sep 15 21:37:49.769676 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.769566    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cbhr\" (UniqueName: \"kubernetes.io/projected/1de5dc48-089c-4bb3-b40a-6b30c490950e-kube-api-access-6cbhr\") pod \"1de5dc48-089c-4bb3-b40a-6b30c490950e\" (UID: \"1de5dc48-089c-4bb3-b40a-6b30c490950e\") "
Sep 15 21:37:49.769891 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.769866    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de5dc48-089c-4bb3-b40a-6b30c490950e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1de5dc48-089c-4bb3-b40a-6b30c490950e" (UID: "1de5dc48-089c-4bb3-b40a-6b30c490950e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:49.770005 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.769992    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz2sh\" (UniqueName: \"kubernetes.io/projected/731be20d-8949-42e0-9233-416cbf2a49ac-kube-api-access-bz2sh\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:49.770042 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.770008    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de5dc48-089c-4bb3-b40a-6b30c490950e-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:49.771541 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.771516    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de5dc48-089c-4bb3-b40a-6b30c490950e-kube-api-access-6cbhr" (OuterVolumeSpecName: "kube-api-access-6cbhr") pod "1de5dc48-089c-4bb3-b40a-6b30c490950e" (UID: "1de5dc48-089c-4bb3-b40a-6b30c490950e"). InnerVolumeSpecName "kube-api-access-6cbhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:49.871749 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.871660    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cbhr\" (UniqueName: \"kubernetes.io/projected/1de5dc48-089c-4bb3-b40a-6b30c490950e-kube-api-access-6cbhr\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:49.984944 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.984912    2584 generic.go:356] "Generic (PLEG): container finished" podID="731be20d-8949-42e0-9233-416cbf2a49ac" containerID="498a4ea55b97a9588f8983a21442d4165fb8e71dcaa8cdd0fd7b1d64cf40f27a" exitCode=0
Sep 15 21:37:49.985093 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.984961    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"
Sep 15 21:37:49.985093 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.985006    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs" event={"ID":"731be20d-8949-42e0-9233-416cbf2a49ac","Type":"ContainerDied","Data":"498a4ea55b97a9588f8983a21442d4165fb8e71dcaa8cdd0fd7b1d64cf40f27a"}
Sep 15 21:37:49.985093 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.985060    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs" event={"ID":"731be20d-8949-42e0-9233-416cbf2a49ac","Type":"ContainerDied","Data":"2ae62ae2512ff06019a09788b79f775853a77e1712a08b5fe999a9b0b7f2e0d9"}
Sep 15 21:37:49.985093 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.985081    2584 scope.go:117] "RemoveContainer" containerID="498a4ea55b97a9588f8983a21442d4165fb8e71dcaa8cdd0fd7b1d64cf40f27a"
Sep 15 21:37:49.986515 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.986496    2584 generic.go:356] "Generic (PLEG): container finished" podID="1de5dc48-089c-4bb3-b40a-6b30c490950e" containerID="c617dd1fc7ddc1bd61bc0f477f7bb198f7ac191c1714a6678789cc82e01ce135" exitCode=0
Sep 15 21:37:49.986623 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.986523    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb" event={"ID":"1de5dc48-089c-4bb3-b40a-6b30c490950e","Type":"ContainerDied","Data":"c617dd1fc7ddc1bd61bc0f477f7bb198f7ac191c1714a6678789cc82e01ce135"}
Sep 15 21:37:49.986623 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.986547    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"
Sep 15 21:37:49.986623 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.986574    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb" event={"ID":"1de5dc48-089c-4bb3-b40a-6b30c490950e","Type":"ContainerDied","Data":"89682061c803742c4d6ccfbdb949203866c614b751e4f4306d119ae6994736f6"}
Sep 15 21:37:49.996620 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.996594    2584 scope.go:117] "RemoveContainer" containerID="498a4ea55b97a9588f8983a21442d4165fb8e71dcaa8cdd0fd7b1d64cf40f27a"
Sep 15 21:37:49.996943 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:37:49.996926    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"498a4ea55b97a9588f8983a21442d4165fb8e71dcaa8cdd0fd7b1d64cf40f27a\": container with ID starting with 498a4ea55b97a9588f8983a21442d4165fb8e71dcaa8cdd0fd7b1d64cf40f27a not found: ID does not exist" containerID="498a4ea55b97a9588f8983a21442d4165fb8e71dcaa8cdd0fd7b1d64cf40f27a"
Sep 15 21:37:49.996991 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.996952    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498a4ea55b97a9588f8983a21442d4165fb8e71dcaa8cdd0fd7b1d64cf40f27a"} err="failed to get container status \"498a4ea55b97a9588f8983a21442d4165fb8e71dcaa8cdd0fd7b1d64cf40f27a\": rpc error: code = NotFound desc = could not find container \"498a4ea55b97a9588f8983a21442d4165fb8e71dcaa8cdd0fd7b1d64cf40f27a\": container with ID starting with 498a4ea55b97a9588f8983a21442d4165fb8e71dcaa8cdd0fd7b1d64cf40f27a not found: ID does not exist"
Sep 15 21:37:49.996991 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:49.996969    2584 scope.go:117] "RemoveContainer" containerID="c617dd1fc7ddc1bd61bc0f477f7bb198f7ac191c1714a6678789cc82e01ce135"
Sep 15 21:37:50.010284 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:50.010218    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"]
Sep 15 21:37:50.012849 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:50.012810    2584 scope.go:117] "RemoveContainer" containerID="c617dd1fc7ddc1bd61bc0f477f7bb198f7ac191c1714a6678789cc82e01ce135"
Sep 15 21:37:50.012979 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:50.012933    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c39135-84944494cf-vvrrs"]
Sep 15 21:37:50.013197 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:37:50.013174    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c617dd1fc7ddc1bd61bc0f477f7bb198f7ac191c1714a6678789cc82e01ce135\": container with ID starting with c617dd1fc7ddc1bd61bc0f477f7bb198f7ac191c1714a6678789cc82e01ce135 not found: ID does not exist" containerID="c617dd1fc7ddc1bd61bc0f477f7bb198f7ac191c1714a6678789cc82e01ce135"
Sep 15 21:37:50.013260 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:50.013208    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c617dd1fc7ddc1bd61bc0f477f7bb198f7ac191c1714a6678789cc82e01ce135"} err="failed to get container status \"c617dd1fc7ddc1bd61bc0f477f7bb198f7ac191c1714a6678789cc82e01ce135\": rpc error: code = NotFound desc = could not find container \"c617dd1fc7ddc1bd61bc0f477f7bb198f7ac191c1714a6678789cc82e01ce135\": container with ID starting with c617dd1fc7ddc1bd61bc0f477f7bb198f7ac191c1714a6678789cc82e01ce135 not found: ID does not exist"
Sep 15 21:37:50.023861 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:50.023834    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"]
Sep 15 21:37:50.026116 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:50.026096    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c39135-6fff9b5fbb-7ngjb"]
Sep 15 21:37:50.673325 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:50.673247    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1de5dc48-089c-4bb3-b40a-6b30c490950e" path="/var/lib/kubelet/pods/1de5dc48-089c-4bb3-b40a-6b30c490950e/volumes"
Sep 15 21:37:50.673697 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:50.673688    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="731be20d-8949-42e0-9233-416cbf2a49ac" path="/var/lib/kubelet/pods/731be20d-8949-42e0-9233-416cbf2a49ac/volumes"
Sep 15 21:37:50.673994 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:50.673980    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d296f7fe-1892-411b-818c-fbe193e38a7c" path="/var/lib/kubelet/pods/d296f7fe-1892-411b-818c-fbe193e38a7c/volumes"
Sep 15 21:37:57.943731 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:57.943695    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-318c22-29825130-mq6kx"]
Sep 15 21:37:57.948166 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:57.948130    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-318c22-29825130-mq6kx"]
Sep 15 21:37:57.974649 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:57.974614    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"]
Sep 15 21:37:57.974903 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:57.974867    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h" podUID="70dd81f7-e79e-4f09-aba4-b92d32d6a31b" containerName="payload-processing" containerID="cri-o://b17940b5c1b3963cefed2f5db165bcc39dbf3e1583c892455a4f683215fc7eed" gracePeriod=30
Sep 15 21:37:58.191347 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:58.191316    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm"]
Sep 15 21:37:58.195218 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:58.195160    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-318c22-fx6bm"]
Sep 15 21:37:58.232284 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:58.232257    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"
Sep 15 21:37:58.365401 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:58.364958    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70dd81f7-e79e-4f09-aba4-b92d32d6a31b-plugins-config-volume\") pod \"70dd81f7-e79e-4f09-aba4-b92d32d6a31b\" (UID: \"70dd81f7-e79e-4f09-aba4-b92d32d6a31b\") "
Sep 15 21:37:58.365401 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:58.365143    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ns9x\" (UniqueName: \"kubernetes.io/projected/70dd81f7-e79e-4f09-aba4-b92d32d6a31b-kube-api-access-8ns9x\") pod \"70dd81f7-e79e-4f09-aba4-b92d32d6a31b\" (UID: \"70dd81f7-e79e-4f09-aba4-b92d32d6a31b\") "
Sep 15 21:37:58.365401 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:58.365354    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70dd81f7-e79e-4f09-aba4-b92d32d6a31b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70dd81f7-e79e-4f09-aba4-b92d32d6a31b" (UID: "70dd81f7-e79e-4f09-aba4-b92d32d6a31b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:58.365761 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:58.365598    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70dd81f7-e79e-4f09-aba4-b92d32d6a31b-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:58.368301 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:58.367911    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70dd81f7-e79e-4f09-aba4-b92d32d6a31b-kube-api-access-8ns9x" (OuterVolumeSpecName: "kube-api-access-8ns9x") pod "70dd81f7-e79e-4f09-aba4-b92d32d6a31b" (UID: "70dd81f7-e79e-4f09-aba4-b92d32d6a31b"). InnerVolumeSpecName "kube-api-access-8ns9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:58.466478 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:58.466400    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ns9x\" (UniqueName: \"kubernetes.io/projected/70dd81f7-e79e-4f09-aba4-b92d32d6a31b-kube-api-access-8ns9x\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:37:58.674367 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:58.674321    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3965e249-51e9-41fc-a33a-cb8716d8e0bb" path="/var/lib/kubelet/pods/3965e249-51e9-41fc-a33a-cb8716d8e0bb/volumes"
Sep 15 21:37:58.674912 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:58.674888    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a85ba6cd-6d34-4e45-9c9b-6cc53fa172bd" path="/var/lib/kubelet/pods/a85ba6cd-6d34-4e45-9c9b-6cc53fa172bd/volumes"
Sep 15 21:37:59.053884 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:59.053847    2584 generic.go:356] "Generic (PLEG): container finished" podID="70dd81f7-e79e-4f09-aba4-b92d32d6a31b" containerID="b17940b5c1b3963cefed2f5db165bcc39dbf3e1583c892455a4f683215fc7eed" exitCode=0
Sep 15 21:37:59.054300 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:59.053927    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"
Sep 15 21:37:59.054300 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:59.053933    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h" event={"ID":"70dd81f7-e79e-4f09-aba4-b92d32d6a31b","Type":"ContainerDied","Data":"b17940b5c1b3963cefed2f5db165bcc39dbf3e1583c892455a4f683215fc7eed"}
Sep 15 21:37:59.054300 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:59.053989    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h" event={"ID":"70dd81f7-e79e-4f09-aba4-b92d32d6a31b","Type":"ContainerDied","Data":"64aabe209623c988c2d10ef97ff135d5e690b8a15354841c05208283e2b48827"}
Sep 15 21:37:59.054300 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:59.054012    2584 scope.go:117] "RemoveContainer" containerID="b17940b5c1b3963cefed2f5db165bcc39dbf3e1583c892455a4f683215fc7eed"
Sep 15 21:37:59.063184 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:59.063165    2584 scope.go:117] "RemoveContainer" containerID="b17940b5c1b3963cefed2f5db165bcc39dbf3e1583c892455a4f683215fc7eed"
Sep 15 21:37:59.063473 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:37:59.063446    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17940b5c1b3963cefed2f5db165bcc39dbf3e1583c892455a4f683215fc7eed\": container with ID starting with b17940b5c1b3963cefed2f5db165bcc39dbf3e1583c892455a4f683215fc7eed not found: ID does not exist" containerID="b17940b5c1b3963cefed2f5db165bcc39dbf3e1583c892455a4f683215fc7eed"
Sep 15 21:37:59.063531 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:59.063481    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17940b5c1b3963cefed2f5db165bcc39dbf3e1583c892455a4f683215fc7eed"} err="failed to get container status \"b17940b5c1b3963cefed2f5db165bcc39dbf3e1583c892455a4f683215fc7eed\": rpc error: code = NotFound desc = could not find container \"b17940b5c1b3963cefed2f5db165bcc39dbf3e1583c892455a4f683215fc7eed\": container with ID starting with b17940b5c1b3963cefed2f5db165bcc39dbf3e1583c892455a4f683215fc7eed not found: ID does not exist"
Sep 15 21:37:59.072270 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:59.072249    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"]
Sep 15 21:37:59.074500 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:37:59.074479    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-318c22-55b69c99f7-xp76h"]
Sep 15 21:38:00.674205 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:00.674172    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70dd81f7-e79e-4f09-aba4-b92d32d6a31b" path="/var/lib/kubelet/pods/70dd81f7-e79e-4f09-aba4-b92d32d6a31b/volumes"
Sep 15 21:38:04.566372 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:04.566343    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"]
Sep 15 21:38:04.566842 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:04.566574    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs" podUID="0425206a-e3e3-4644-9066-69c1fa315c2f" containerName="istio-proxy" containerID="cri-o://31f9d7254b9c2ff9c74d38cf115e55aaacd5f88d333a21dab36f98435a3ce95f" gracePeriod=30
Sep 15 21:38:09.829545 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.829518    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:38:09.879894 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.879807    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-podinfo\") pod \"0425206a-e3e3-4644-9066-69c1fa315c2f\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") "
Sep 15 21:38:09.880065 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.879903    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-credential-socket\") pod \"0425206a-e3e3-4644-9066-69c1fa315c2f\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") "
Sep 15 21:38:09.880065 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.879935    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-envoy\") pod \"0425206a-e3e3-4644-9066-69c1fa315c2f\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") "
Sep 15 21:38:09.880065 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.879959    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-workload-socket\") pod \"0425206a-e3e3-4644-9066-69c1fa315c2f\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") "
Sep 15 21:38:09.880065 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.880001    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-workload-certs\") pod \"0425206a-e3e3-4644-9066-69c1fa315c2f\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") "
Sep 15 21:38:09.880065 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.880032    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-token\") pod \"0425206a-e3e3-4644-9066-69c1fa315c2f\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") "
Sep 15 21:38:09.880327 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.880103    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0425206a-e3e3-4644-9066-69c1fa315c2f-istiod-ca-cert\") pod \"0425206a-e3e3-4644-9066-69c1fa315c2f\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") "
Sep 15 21:38:09.880327 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.880166    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-data\") pod \"0425206a-e3e3-4644-9066-69c1fa315c2f\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") "
Sep 15 21:38:09.880327 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.880197    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw4xt\" (UniqueName: \"kubernetes.io/projected/0425206a-e3e3-4644-9066-69c1fa315c2f-kube-api-access-cw4xt\") pod \"0425206a-e3e3-4644-9066-69c1fa315c2f\" (UID: \"0425206a-e3e3-4644-9066-69c1fa315c2f\") "
Sep 15 21:38:09.880327 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.880239    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0425206a-e3e3-4644-9066-69c1fa315c2f" (UID: "0425206a-e3e3-4644-9066-69c1fa315c2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:09.880552 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.880452    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0425206a-e3e3-4644-9066-69c1fa315c2f" (UID: "0425206a-e3e3-4644-9066-69c1fa315c2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:09.880621 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.880576    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-data" (OuterVolumeSpecName: "istio-data") pod "0425206a-e3e3-4644-9066-69c1fa315c2f" (UID: "0425206a-e3e3-4644-9066-69c1fa315c2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:09.880703 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.880682    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0425206a-e3e3-4644-9066-69c1fa315c2f" (UID: "0425206a-e3e3-4644-9066-69c1fa315c2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:09.880775 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.880726    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-data\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:09.880775 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.880744    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-credential-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:09.880775 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.880759    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-workload-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:09.880942 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.880834    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0425206a-e3e3-4644-9066-69c1fa315c2f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0425206a-e3e3-4644-9066-69c1fa315c2f" (UID: "0425206a-e3e3-4644-9066-69c1fa315c2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:38:09.882600 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.882551    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0425206a-e3e3-4644-9066-69c1fa315c2f-kube-api-access-cw4xt" (OuterVolumeSpecName: "kube-api-access-cw4xt") pod "0425206a-e3e3-4644-9066-69c1fa315c2f" (UID: "0425206a-e3e3-4644-9066-69c1fa315c2f"). InnerVolumeSpecName "kube-api-access-cw4xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:09.882726 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.882633    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0425206a-e3e3-4644-9066-69c1fa315c2f" (UID: "0425206a-e3e3-4644-9066-69c1fa315c2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:09.882726 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.882633    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0425206a-e3e3-4644-9066-69c1fa315c2f" (UID: "0425206a-e3e3-4644-9066-69c1fa315c2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:38:09.882892 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.882873    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-token" (OuterVolumeSpecName: "istio-token") pod "0425206a-e3e3-4644-9066-69c1fa315c2f" (UID: "0425206a-e3e3-4644-9066-69c1fa315c2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:09.981541 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.981499    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-envoy\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:09.981541 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.981533    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0425206a-e3e3-4644-9066-69c1fa315c2f-workload-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:09.981541 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.981543    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-token\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:09.981541 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.981551    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0425206a-e3e3-4644-9066-69c1fa315c2f-istiod-ca-cert\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:09.981813 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.981560    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw4xt\" (UniqueName: \"kubernetes.io/projected/0425206a-e3e3-4644-9066-69c1fa315c2f-kube-api-access-cw4xt\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:09.981813 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:09.981569    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0425206a-e3e3-4644-9066-69c1fa315c2f-istio-podinfo\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:10.128158 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:10.128123    2584 generic.go:356] "Generic (PLEG): container finished" podID="0425206a-e3e3-4644-9066-69c1fa315c2f" containerID="31f9d7254b9c2ff9c74d38cf115e55aaacd5f88d333a21dab36f98435a3ce95f" exitCode=0
Sep 15 21:38:10.128330 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:10.128170    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"
Sep 15 21:38:10.128330 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:10.128178    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs" event={"ID":"0425206a-e3e3-4644-9066-69c1fa315c2f","Type":"ContainerDied","Data":"31f9d7254b9c2ff9c74d38cf115e55aaacd5f88d333a21dab36f98435a3ce95f"}
Sep 15 21:38:10.128330 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:10.128220    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs" event={"ID":"0425206a-e3e3-4644-9066-69c1fa315c2f","Type":"ContainerDied","Data":"f831ed38ec5f31deeeab32326281be5afc111aa18ac562075894cacd511815a1"}
Sep 15 21:38:10.128330 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:10.128240    2584 scope.go:117] "RemoveContainer" containerID="31f9d7254b9c2ff9c74d38cf115e55aaacd5f88d333a21dab36f98435a3ce95f"
Sep 15 21:38:10.137989 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:10.137971    2584 scope.go:117] "RemoveContainer" containerID="bdd22441729cc8e014465ba0a490f61ef090a1311e987a351872750031f68ddd"
Sep 15 21:38:10.146134 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:10.146114    2584 scope.go:117] "RemoveContainer" containerID="31f9d7254b9c2ff9c74d38cf115e55aaacd5f88d333a21dab36f98435a3ce95f"
Sep 15 21:38:10.146422 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:38:10.146403    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f9d7254b9c2ff9c74d38cf115e55aaacd5f88d333a21dab36f98435a3ce95f\": container with ID starting with 31f9d7254b9c2ff9c74d38cf115e55aaacd5f88d333a21dab36f98435a3ce95f not found: ID does not exist" containerID="31f9d7254b9c2ff9c74d38cf115e55aaacd5f88d333a21dab36f98435a3ce95f"
Sep 15 21:38:10.146496 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:10.146433    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f9d7254b9c2ff9c74d38cf115e55aaacd5f88d333a21dab36f98435a3ce95f"} err="failed to get container status \"31f9d7254b9c2ff9c74d38cf115e55aaacd5f88d333a21dab36f98435a3ce95f\": rpc error: code = NotFound desc = could not find container \"31f9d7254b9c2ff9c74d38cf115e55aaacd5f88d333a21dab36f98435a3ce95f\": container with ID starting with 31f9d7254b9c2ff9c74d38cf115e55aaacd5f88d333a21dab36f98435a3ce95f not found: ID does not exist"
Sep 15 21:38:10.146496 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:10.146450    2584 scope.go:117] "RemoveContainer" containerID="bdd22441729cc8e014465ba0a490f61ef090a1311e987a351872750031f68ddd"
Sep 15 21:38:10.146705 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:38:10.146688    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd22441729cc8e014465ba0a490f61ef090a1311e987a351872750031f68ddd\": container with ID starting with bdd22441729cc8e014465ba0a490f61ef090a1311e987a351872750031f68ddd not found: ID does not exist" containerID="bdd22441729cc8e014465ba0a490f61ef090a1311e987a351872750031f68ddd"
Sep 15 21:38:10.146754 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:10.146711    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd22441729cc8e014465ba0a490f61ef090a1311e987a351872750031f68ddd"} err="failed to get container status \"bdd22441729cc8e014465ba0a490f61ef090a1311e987a351872750031f68ddd\": rpc error: code = NotFound desc = could not find container \"bdd22441729cc8e014465ba0a490f61ef090a1311e987a351872750031f68ddd\": container with ID starting with bdd22441729cc8e014465ba0a490f61ef090a1311e987a351872750031f68ddd not found: ID does not exist"
Sep 15 21:38:10.150908 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:10.150883    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"]
Sep 15 21:38:10.157292 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:10.157265    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-318c22-openshift-default-6f8fc57498-7rbjs"]
Sep 15 21:38:10.680763 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:10.680731    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0425206a-e3e3-4644-9066-69c1fa315c2f" path="/var/lib/kubelet/pods/0425206a-e3e3-4644-9066-69c1fa315c2f/volumes"
Sep 15 21:38:28.999531 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:28.999496    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv"]
Sep 15 21:38:28.999962 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:28.999922    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de5dc48-089c-4bb3-b40a-6b30c490950e" containerName="payload-pre-processing"
Sep 15 21:38:28.999962 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:28.999933    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de5dc48-089c-4bb3-b40a-6b30c490950e" containerName="payload-pre-processing"
Sep 15 21:38:28.999962 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:28.999945    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3965e249-51e9-41fc-a33a-cb8716d8e0bb" containerName="revoke-keys"
Sep 15 21:38:28.999962 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:28.999951    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3965e249-51e9-41fc-a33a-cb8716d8e0bb" containerName="revoke-keys"
Sep 15 21:38:28.999962 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:28.999963    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="731be20d-8949-42e0-9233-416cbf2a49ac" containerName="payload-processing"
Sep 15 21:38:29.000131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:28.999970    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="731be20d-8949-42e0-9233-416cbf2a49ac" containerName="payload-processing"
Sep 15 21:38:29.000131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:28.999981    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70dd81f7-e79e-4f09-aba4-b92d32d6a31b" containerName="payload-processing"
Sep 15 21:38:29.000131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:28.999986    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="70dd81f7-e79e-4f09-aba4-b92d32d6a31b" containerName="payload-processing"
Sep 15 21:38:29.000131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:28.999994    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0425206a-e3e3-4644-9066-69c1fa315c2f" containerName="istio-proxy"
Sep 15 21:38:29.000131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:28.999999    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0425206a-e3e3-4644-9066-69c1fa315c2f" containerName="istio-proxy"
Sep 15 21:38:29.000131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.000005    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0425206a-e3e3-4644-9066-69c1fa315c2f" containerName="istio-proxy"
Sep 15 21:38:29.000131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.000011    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0425206a-e3e3-4644-9066-69c1fa315c2f" containerName="istio-proxy"
Sep 15 21:38:29.000131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.000027    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d296f7fe-1892-411b-818c-fbe193e38a7c" containerName="revoke-keys"
Sep 15 21:38:29.000131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.000032    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d296f7fe-1892-411b-818c-fbe193e38a7c" containerName="revoke-keys"
Sep 15 21:38:29.000131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.000096    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0425206a-e3e3-4644-9066-69c1fa315c2f" containerName="istio-proxy"
Sep 15 21:38:29.000131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.000103    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0425206a-e3e3-4644-9066-69c1fa315c2f" containerName="istio-proxy"
Sep 15 21:38:29.000131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.000109    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="731be20d-8949-42e0-9233-416cbf2a49ac" containerName="payload-processing"
Sep 15 21:38:29.000131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.000116    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="70dd81f7-e79e-4f09-aba4-b92d32d6a31b" containerName="payload-processing"
Sep 15 21:38:29.000131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.000122    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de5dc48-089c-4bb3-b40a-6b30c490950e" containerName="payload-pre-processing"
Sep 15 21:38:29.000131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.000127    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d296f7fe-1892-411b-818c-fbe193e38a7c" containerName="revoke-keys"
Sep 15 21:38:29.000131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.000132    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3965e249-51e9-41fc-a33a-cb8716d8e0bb" containerName="revoke-keys"
Sep 15 21:38:29.004539 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.004523    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv"
Sep 15 21:38:29.006984 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.006939    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:38:29.007773 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.007748    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:38:29.010949 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.010925    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv"]
Sep 15 21:38:29.079418 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.079357    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87d7733-ca98-4d75-9972-43387262625c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv\" (UID: \"b87d7733-ca98-4d75-9972-43387262625c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv"
Sep 15 21:38:29.180295 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.180256    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87d7733-ca98-4d75-9972-43387262625c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv\" (UID: \"b87d7733-ca98-4d75-9972-43387262625c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv"
Sep 15 21:38:29.180911 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.180891    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87d7733-ca98-4d75-9972-43387262625c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv\" (UID: \"b87d7733-ca98-4d75-9972-43387262625c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv"
Sep 15 21:38:29.316439 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.316414    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv"
Sep 15 21:38:29.590742 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:29.590717    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv"]
Sep 15 21:38:29.592570 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:38:29.592528    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87d7733_ca98_4d75_9972_43387262625c.slice/crio-9917c2f5efda66a912fdfcafaf71e8d9af32430974738b1f7f7c5322ec8b8c06 WatchSource:0}: Error finding container 9917c2f5efda66a912fdfcafaf71e8d9af32430974738b1f7f7c5322ec8b8c06: Status 404 returned error can't find the container with id 9917c2f5efda66a912fdfcafaf71e8d9af32430974738b1f7f7c5322ec8b8c06
Sep 15 21:38:30.250333 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:30.250238    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv" event={"ID":"b87d7733-ca98-4d75-9972-43387262625c","Type":"ContainerStarted","Data":"c75eff9916ff0a5b69e13e6bc9ec5616eb53d79dfeccd8e4dfa18c6814a80b35"}
Sep 15 21:38:30.250706 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:30.250333    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv" event={"ID":"b87d7733-ca98-4d75-9972-43387262625c","Type":"ContainerStarted","Data":"9917c2f5efda66a912fdfcafaf71e8d9af32430974738b1f7f7c5322ec8b8c06"}
Sep 15 21:38:30.266108 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:30.266064    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv" podStartSLOduration=2.26604969 podStartE2EDuration="2.26604969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:38:30.264258761 +0000 UTC m=+1678.126192565" watchObservedRunningTime="2026-09-15 21:38:30.26604969 +0000 UTC m=+1678.127983495"
Sep 15 21:38:31.259269 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:31.259229    2584 generic.go:356] "Generic (PLEG): container finished" podID="b87d7733-ca98-4d75-9972-43387262625c" containerID="c75eff9916ff0a5b69e13e6bc9ec5616eb53d79dfeccd8e4dfa18c6814a80b35" exitCode=0
Sep 15 21:38:31.259692 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:31.259344    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv" event={"ID":"b87d7733-ca98-4d75-9972-43387262625c","Type":"ContainerDied","Data":"c75eff9916ff0a5b69e13e6bc9ec5616eb53d79dfeccd8e4dfa18c6814a80b35"}
Sep 15 21:38:32.409640 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:32.409615    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv"
Sep 15 21:38:32.513587 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:32.513563    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87d7733-ca98-4d75-9972-43387262625c-maas-api-service-ca\") pod \"b87d7733-ca98-4d75-9972-43387262625c\" (UID: \"b87d7733-ca98-4d75-9972-43387262625c\") "
Sep 15 21:38:32.513913 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:32.513886    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87d7733-ca98-4d75-9972-43387262625c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b87d7733-ca98-4d75-9972-43387262625c" (UID: "b87d7733-ca98-4d75-9972-43387262625c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:38:32.614834 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:32.614802    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87d7733-ca98-4d75-9972-43387262625c-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:32.993574 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:32.993497    2584 scope.go:117] "RemoveContainer" containerID="15064fa418c6677045947afcc0a28b249126398357ad950fe5f0ba8365f06aed"
Sep 15 21:38:33.272110 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:33.272080    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv" event={"ID":"b87d7733-ca98-4d75-9972-43387262625c","Type":"ContainerDied","Data":"9917c2f5efda66a912fdfcafaf71e8d9af32430974738b1f7f7c5322ec8b8c06"}
Sep 15 21:38:33.272245 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:33.272115    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv"
Sep 15 21:38:33.272245 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:33.272138    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9917c2f5efda66a912fdfcafaf71e8d9af32430974738b1f7f7c5322ec8b8c06"
Sep 15 21:38:33.866848 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:33.866809    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"]
Sep 15 21:38:33.873602 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:33.873569    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"]
Sep 15 21:38:33.883299 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:33.883273    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"]
Sep 15 21:38:44.500398 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:44.500343    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6f66f7-29825130-bwltj"]
Sep 15 21:38:44.503389 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:44.503348    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6f66f7-29825130-bwltj"]
Sep 15 21:38:44.535130 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:44.535102    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"]
Sep 15 21:38:44.535422 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:44.535359    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w" podUID="3d1794ad-3eb4-450e-aaa9-424d0fe894eb" containerName="payload-pre-processing" containerID="cri-o://913c2bd312451fc05cce53e109b8ab6a1ae0a55fd995ba5d36b0ce49f79c8f02" gracePeriod=30
Sep 15 21:38:44.674901 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:44.674867    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35834631-72e7-45fc-b0c0-7daf6f188eb5" path="/var/lib/kubelet/pods/35834631-72e7-45fc-b0c0-7daf6f188eb5/volumes"
Sep 15 21:38:44.702769 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:44.702737    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv"]
Sep 15 21:38:44.705008 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:44.704986    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6f66f7-h6xfv"]
Sep 15 21:38:44.816936 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:44.816908    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"
Sep 15 21:38:44.833333 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:44.833300    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1794ad-3eb4-450e-aaa9-424d0fe894eb-plugins-config-volume\") pod \"3d1794ad-3eb4-450e-aaa9-424d0fe894eb\" (UID: \"3d1794ad-3eb4-450e-aaa9-424d0fe894eb\") "
Sep 15 21:38:44.833540 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:44.833406    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtb4d\" (UniqueName: \"kubernetes.io/projected/3d1794ad-3eb4-450e-aaa9-424d0fe894eb-kube-api-access-xtb4d\") pod \"3d1794ad-3eb4-450e-aaa9-424d0fe894eb\" (UID: \"3d1794ad-3eb4-450e-aaa9-424d0fe894eb\") "
Sep 15 21:38:44.834105 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:44.834073    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1794ad-3eb4-450e-aaa9-424d0fe894eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d1794ad-3eb4-450e-aaa9-424d0fe894eb" (UID: "3d1794ad-3eb4-450e-aaa9-424d0fe894eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:38:44.836279 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:44.836251    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1794ad-3eb4-450e-aaa9-424d0fe894eb-kube-api-access-xtb4d" (OuterVolumeSpecName: "kube-api-access-xtb4d") pod "3d1794ad-3eb4-450e-aaa9-424d0fe894eb" (UID: "3d1794ad-3eb4-450e-aaa9-424d0fe894eb"). InnerVolumeSpecName "kube-api-access-xtb4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:44.934289 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:44.934260    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtb4d\" (UniqueName: \"kubernetes.io/projected/3d1794ad-3eb4-450e-aaa9-424d0fe894eb-kube-api-access-xtb4d\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:44.934289 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:44.934286    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1794ad-3eb4-450e-aaa9-424d0fe894eb-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:45.344833 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:45.344794    2584 generic.go:356] "Generic (PLEG): container finished" podID="3d1794ad-3eb4-450e-aaa9-424d0fe894eb" containerID="913c2bd312451fc05cce53e109b8ab6a1ae0a55fd995ba5d36b0ce49f79c8f02" exitCode=0
Sep 15 21:38:45.345001 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:45.344842    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"
Sep 15 21:38:45.345001 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:45.344880    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w" event={"ID":"3d1794ad-3eb4-450e-aaa9-424d0fe894eb","Type":"ContainerDied","Data":"913c2bd312451fc05cce53e109b8ab6a1ae0a55fd995ba5d36b0ce49f79c8f02"}
Sep 15 21:38:45.345001 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:45.344937    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w" event={"ID":"3d1794ad-3eb4-450e-aaa9-424d0fe894eb","Type":"ContainerDied","Data":"39007c822b8d9ccbd0d92c5dee460f695fe14f09a32a40bdb8ef358b1c0ba8bf"}
Sep 15 21:38:45.345001 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:45.344960    2584 scope.go:117] "RemoveContainer" containerID="913c2bd312451fc05cce53e109b8ab6a1ae0a55fd995ba5d36b0ce49f79c8f02"
Sep 15 21:38:45.356546 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:45.356518    2584 scope.go:117] "RemoveContainer" containerID="913c2bd312451fc05cce53e109b8ab6a1ae0a55fd995ba5d36b0ce49f79c8f02"
Sep 15 21:38:45.356840 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:38:45.356820    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"913c2bd312451fc05cce53e109b8ab6a1ae0a55fd995ba5d36b0ce49f79c8f02\": container with ID starting with 913c2bd312451fc05cce53e109b8ab6a1ae0a55fd995ba5d36b0ce49f79c8f02 not found: ID does not exist" containerID="913c2bd312451fc05cce53e109b8ab6a1ae0a55fd995ba5d36b0ce49f79c8f02"
Sep 15 21:38:45.356912 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:45.356848    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913c2bd312451fc05cce53e109b8ab6a1ae0a55fd995ba5d36b0ce49f79c8f02"} err="failed to get container status \"913c2bd312451fc05cce53e109b8ab6a1ae0a55fd995ba5d36b0ce49f79c8f02\": rpc error: code = NotFound desc = could not find container \"913c2bd312451fc05cce53e109b8ab6a1ae0a55fd995ba5d36b0ce49f79c8f02\": container with ID starting with 913c2bd312451fc05cce53e109b8ab6a1ae0a55fd995ba5d36b0ce49f79c8f02 not found: ID does not exist"
Sep 15 21:38:45.370040 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:45.370019    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"]
Sep 15 21:38:45.374550 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:45.374531    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-6f66f7-7d76676486-hzb6w"]
Sep 15 21:38:46.673327 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:46.673294    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d1794ad-3eb4-450e-aaa9-424d0fe894eb" path="/var/lib/kubelet/pods/3d1794ad-3eb4-450e-aaa9-424d0fe894eb/volumes"
Sep 15 21:38:46.673699 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:46.673650    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b87d7733-ca98-4d75-9972-43387262625c" path="/var/lib/kubelet/pods/b87d7733-ca98-4d75-9972-43387262625c/volumes"
Sep 15 21:38:48.887474 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:48.887438    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh" podUID="ad652e4b-78a3-496e-9cc5-06736f0c6204" containerName="main" containerID="cri-o://e381fc3b3d3914df7ea79c3e1fe56babc3545a0a24ddfee077a4e8b0e5e39fd0" gracePeriod=45
Sep 15 21:38:48.898722 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:48.898692    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd" podUID="8356eab8-04b6-42b2-8a33-f6069e71ac6b" containerName="main" containerID="cri-o://c2d0fe0f2113c05e79208de3f8fda00522031ca097355bb4a97871d879f79ac3" gracePeriod=45
Sep 15 21:38:48.903126 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:48.903101    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt" podUID="ef0ed4d5-9eca-43a8-b817-18f79a35d213" containerName="main" containerID="cri-o://6e743941d0a1f495d5ffe2a03932adc03095fdf523a04eb00b3eebf51fb57781" gracePeriod=45
Sep 15 21:38:49.213047 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.213018    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:38:49.216238 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.216218    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:38:49.220047 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.220028    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:38:49.278121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278078    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-home\") pod \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") "
Sep 15 21:38:49.278121 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278119    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-home\") pod \"ad652e4b-78a3-496e-9cc5-06736f0c6204\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") "
Sep 15 21:38:49.278362 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278169    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-model-cache\") pod \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") "
Sep 15 21:38:49.278362 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278196    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2dtk\" (UniqueName: \"kubernetes.io/projected/8356eab8-04b6-42b2-8a33-f6069e71ac6b-kube-api-access-n2dtk\") pod \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") "
Sep 15 21:38:49.278362 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278228    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-tmp-dir\") pod \"ad652e4b-78a3-496e-9cc5-06736f0c6204\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") "
Sep 15 21:38:49.278362 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278257    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbp9d\" (UniqueName: \"kubernetes.io/projected/ad652e4b-78a3-496e-9cc5-06736f0c6204-kube-api-access-nbp9d\") pod \"ad652e4b-78a3-496e-9cc5-06736f0c6204\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") "
Sep 15 21:38:49.278362 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278287    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8356eab8-04b6-42b2-8a33-f6069e71ac6b-tls-certs\") pod \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") "
Sep 15 21:38:49.278362 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278324    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad652e4b-78a3-496e-9cc5-06736f0c6204-tls-certs\") pod \"ad652e4b-78a3-496e-9cc5-06736f0c6204\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") "
Sep 15 21:38:49.278362 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278361    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-dshm\") pod \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") "
Sep 15 21:38:49.278777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278392    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-home" (OuterVolumeSpecName: "home") pod "ef0ed4d5-9eca-43a8-b817-18f79a35d213" (UID: "ef0ed4d5-9eca-43a8-b817-18f79a35d213"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:49.278777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278402    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-home" (OuterVolumeSpecName: "home") pod "ad652e4b-78a3-496e-9cc5-06736f0c6204" (UID: "ad652e4b-78a3-496e-9cc5-06736f0c6204"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:49.278777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278405    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-dshm\") pod \"ad652e4b-78a3-496e-9cc5-06736f0c6204\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") "
Sep 15 21:38:49.278777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278468    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "8356eab8-04b6-42b2-8a33-f6069e71ac6b" (UID: "8356eab8-04b6-42b2-8a33-f6069e71ac6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:49.278777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278492    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qmft\" (UniqueName: \"kubernetes.io/projected/ef0ed4d5-9eca-43a8-b817-18f79a35d213-kube-api-access-5qmft\") pod \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") "
Sep 15 21:38:49.278777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278523    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-tmp-dir\") pod \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") "
Sep 15 21:38:49.278777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278561    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-home\") pod \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") "
Sep 15 21:38:49.278777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278605    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef0ed4d5-9eca-43a8-b817-18f79a35d213-tls-certs\") pod \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") "
Sep 15 21:38:49.278777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278690    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-model-cache\") pod \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") "
Sep 15 21:38:49.278777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278713    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-model-cache\") pod \"ad652e4b-78a3-496e-9cc5-06736f0c6204\" (UID: \"ad652e4b-78a3-496e-9cc5-06736f0c6204\") "
Sep 15 21:38:49.278777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278745    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-dshm\") pod \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\" (UID: \"8356eab8-04b6-42b2-8a33-f6069e71ac6b\") "
Sep 15 21:38:49.278777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.278772    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-tmp-dir\") pod \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\" (UID: \"ef0ed4d5-9eca-43a8-b817-18f79a35d213\") "
Sep 15 21:38:49.279484 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.279188    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-home\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.279484 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.279206    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-home\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.279484 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.279218    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-model-cache\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.279484 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.279431    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef0ed4d5-9eca-43a8-b817-18f79a35d213" (UID: "ef0ed4d5-9eca-43a8-b817-18f79a35d213"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:49.281166 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.281001    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad652e4b-78a3-496e-9cc5-06736f0c6204" (UID: "ad652e4b-78a3-496e-9cc5-06736f0c6204"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:49.281291 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.281188    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8356eab8-04b6-42b2-8a33-f6069e71ac6b" (UID: "8356eab8-04b6-42b2-8a33-f6069e71ac6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:49.281472 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.281445    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad652e4b-78a3-496e-9cc5-06736f0c6204" (UID: "ad652e4b-78a3-496e-9cc5-06736f0c6204"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:49.281753 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.281727    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef0ed4d5-9eca-43a8-b817-18f79a35d213" (UID: "ef0ed4d5-9eca-43a8-b817-18f79a35d213"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:49.283308 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.283281    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad652e4b-78a3-496e-9cc5-06736f0c6204-kube-api-access-nbp9d" (OuterVolumeSpecName: "kube-api-access-nbp9d") pod "ad652e4b-78a3-496e-9cc5-06736f0c6204" (UID: "ad652e4b-78a3-496e-9cc5-06736f0c6204"). InnerVolumeSpecName "kube-api-access-nbp9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:49.283565 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.283535    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-home" (OuterVolumeSpecName: "home") pod "8356eab8-04b6-42b2-8a33-f6069e71ac6b" (UID: "8356eab8-04b6-42b2-8a33-f6069e71ac6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:49.283660 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.283561    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-dshm" (OuterVolumeSpecName: "dshm") pod "ef0ed4d5-9eca-43a8-b817-18f79a35d213" (UID: "ef0ed4d5-9eca-43a8-b817-18f79a35d213"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:49.283968 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.283889    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8356eab8-04b6-42b2-8a33-f6069e71ac6b-kube-api-access-n2dtk" (OuterVolumeSpecName: "kube-api-access-n2dtk") pod "8356eab8-04b6-42b2-8a33-f6069e71ac6b" (UID: "8356eab8-04b6-42b2-8a33-f6069e71ac6b"). InnerVolumeSpecName "kube-api-access-n2dtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:49.284039 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.284015    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef0ed4d5-9eca-43a8-b817-18f79a35d213-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef0ed4d5-9eca-43a8-b817-18f79a35d213" (UID: "ef0ed4d5-9eca-43a8-b817-18f79a35d213"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:49.284126 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.284106    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef0ed4d5-9eca-43a8-b817-18f79a35d213-kube-api-access-5qmft" (OuterVolumeSpecName: "kube-api-access-5qmft") pod "ef0ed4d5-9eca-43a8-b817-18f79a35d213" (UID: "ef0ed4d5-9eca-43a8-b817-18f79a35d213"). InnerVolumeSpecName "kube-api-access-5qmft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:49.284197 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.284181    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-dshm" (OuterVolumeSpecName: "dshm") pod "8356eab8-04b6-42b2-8a33-f6069e71ac6b" (UID: "8356eab8-04b6-42b2-8a33-f6069e71ac6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:49.284255 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.284199    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-dshm" (OuterVolumeSpecName: "dshm") pod "ad652e4b-78a3-496e-9cc5-06736f0c6204" (UID: "ad652e4b-78a3-496e-9cc5-06736f0c6204"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:49.284438 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.284412    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad652e4b-78a3-496e-9cc5-06736f0c6204-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad652e4b-78a3-496e-9cc5-06736f0c6204" (UID: "ad652e4b-78a3-496e-9cc5-06736f0c6204"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:49.285478 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.285452    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8356eab8-04b6-42b2-8a33-f6069e71ac6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8356eab8-04b6-42b2-8a33-f6069e71ac6b" (UID: "8356eab8-04b6-42b2-8a33-f6069e71ac6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:49.372078 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.372047    2584 generic.go:356] "Generic (PLEG): container finished" podID="ad652e4b-78a3-496e-9cc5-06736f0c6204" containerID="e381fc3b3d3914df7ea79c3e1fe56babc3545a0a24ddfee077a4e8b0e5e39fd0" exitCode=0
Sep 15 21:38:49.372246 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.372107    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"
Sep 15 21:38:49.372246 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.372139    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh" event={"ID":"ad652e4b-78a3-496e-9cc5-06736f0c6204","Type":"ContainerDied","Data":"e381fc3b3d3914df7ea79c3e1fe56babc3545a0a24ddfee077a4e8b0e5e39fd0"}
Sep 15 21:38:49.372246 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.372185    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh" event={"ID":"ad652e4b-78a3-496e-9cc5-06736f0c6204","Type":"ContainerDied","Data":"1b96bb8d743f1883f31acc579edebb638ec260e6287d4de1b5140a68dcc90d54"}
Sep 15 21:38:49.372246 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.372207    2584 scope.go:117] "RemoveContainer" containerID="e381fc3b3d3914df7ea79c3e1fe56babc3545a0a24ddfee077a4e8b0e5e39fd0"
Sep 15 21:38:49.373711 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.373686    2584 generic.go:356] "Generic (PLEG): container finished" podID="8356eab8-04b6-42b2-8a33-f6069e71ac6b" containerID="c2d0fe0f2113c05e79208de3f8fda00522031ca097355bb4a97871d879f79ac3" exitCode=0
Sep 15 21:38:49.373843 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.373717    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd" event={"ID":"8356eab8-04b6-42b2-8a33-f6069e71ac6b","Type":"ContainerDied","Data":"c2d0fe0f2113c05e79208de3f8fda00522031ca097355bb4a97871d879f79ac3"}
Sep 15 21:38:49.373843 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.373743    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"
Sep 15 21:38:49.373843 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.373752    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd" event={"ID":"8356eab8-04b6-42b2-8a33-f6069e71ac6b","Type":"ContainerDied","Data":"c47223e83952c1eae781edc16288a55e8828d69b67a161c26b295a57b9d5ec63"}
Sep 15 21:38:49.375226 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.375199    2584 generic.go:356] "Generic (PLEG): container finished" podID="ef0ed4d5-9eca-43a8-b817-18f79a35d213" containerID="6e743941d0a1f495d5ffe2a03932adc03095fdf523a04eb00b3eebf51fb57781" exitCode=0
Sep 15 21:38:49.375407 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.375258    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"
Sep 15 21:38:49.375407 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.375274    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt" event={"ID":"ef0ed4d5-9eca-43a8-b817-18f79a35d213","Type":"ContainerDied","Data":"6e743941d0a1f495d5ffe2a03932adc03095fdf523a04eb00b3eebf51fb57781"}
Sep 15 21:38:49.375407 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.375314    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt" event={"ID":"ef0ed4d5-9eca-43a8-b817-18f79a35d213","Type":"ContainerDied","Data":"1c36a8588644150aad0dbca611f76527215056404df7964cffa947f1e6b1d942"}
Sep 15 21:38:49.379908 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.379887    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-model-cache\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.379908 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.379910    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-model-cache\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.380086 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.379923    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-dshm\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.380086 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.379934    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-tmp-dir\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.380086 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.379947    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2dtk\" (UniqueName: \"kubernetes.io/projected/8356eab8-04b6-42b2-8a33-f6069e71ac6b-kube-api-access-n2dtk\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.380086 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.379961    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-tmp-dir\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.380086 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.379975    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbp9d\" (UniqueName: \"kubernetes.io/projected/ad652e4b-78a3-496e-9cc5-06736f0c6204-kube-api-access-nbp9d\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.380086 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.379986    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8356eab8-04b6-42b2-8a33-f6069e71ac6b-tls-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.380086 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.379998    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad652e4b-78a3-496e-9cc5-06736f0c6204-tls-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.380086 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.380008    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef0ed4d5-9eca-43a8-b817-18f79a35d213-dshm\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.380086 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.380019    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad652e4b-78a3-496e-9cc5-06736f0c6204-dshm\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.380086 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.380030    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qmft\" (UniqueName: \"kubernetes.io/projected/ef0ed4d5-9eca-43a8-b817-18f79a35d213-kube-api-access-5qmft\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.380086 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.380043    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-tmp-dir\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.380086 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.380055    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8356eab8-04b6-42b2-8a33-f6069e71ac6b-home\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.380086 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.380066    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef0ed4d5-9eca-43a8-b817-18f79a35d213-tls-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:49.381908 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.381894    2584 scope.go:117] "RemoveContainer" containerID="e381fc3b3d3914df7ea79c3e1fe56babc3545a0a24ddfee077a4e8b0e5e39fd0"
Sep 15 21:38:49.382186 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:38:49.382166    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e381fc3b3d3914df7ea79c3e1fe56babc3545a0a24ddfee077a4e8b0e5e39fd0\": container with ID starting with e381fc3b3d3914df7ea79c3e1fe56babc3545a0a24ddfee077a4e8b0e5e39fd0 not found: ID does not exist" containerID="e381fc3b3d3914df7ea79c3e1fe56babc3545a0a24ddfee077a4e8b0e5e39fd0"
Sep 15 21:38:49.382259 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.382194    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e381fc3b3d3914df7ea79c3e1fe56babc3545a0a24ddfee077a4e8b0e5e39fd0"} err="failed to get container status \"e381fc3b3d3914df7ea79c3e1fe56babc3545a0a24ddfee077a4e8b0e5e39fd0\": rpc error: code = NotFound desc = could not find container \"e381fc3b3d3914df7ea79c3e1fe56babc3545a0a24ddfee077a4e8b0e5e39fd0\": container with ID starting with e381fc3b3d3914df7ea79c3e1fe56babc3545a0a24ddfee077a4e8b0e5e39fd0 not found: ID does not exist"
Sep 15 21:38:49.382259 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.382209    2584 scope.go:117] "RemoveContainer" containerID="c2d0fe0f2113c05e79208de3f8fda00522031ca097355bb4a97871d879f79ac3"
Sep 15 21:38:49.390781 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.390761    2584 scope.go:117] "RemoveContainer" containerID="c2d0fe0f2113c05e79208de3f8fda00522031ca097355bb4a97871d879f79ac3"
Sep 15 21:38:49.391045 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:38:49.391028    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d0fe0f2113c05e79208de3f8fda00522031ca097355bb4a97871d879f79ac3\": container with ID starting with c2d0fe0f2113c05e79208de3f8fda00522031ca097355bb4a97871d879f79ac3 not found: ID does not exist" containerID="c2d0fe0f2113c05e79208de3f8fda00522031ca097355bb4a97871d879f79ac3"
Sep 15 21:38:49.391091 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.391053    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d0fe0f2113c05e79208de3f8fda00522031ca097355bb4a97871d879f79ac3"} err="failed to get container status \"c2d0fe0f2113c05e79208de3f8fda00522031ca097355bb4a97871d879f79ac3\": rpc error: code = NotFound desc = could not find container \"c2d0fe0f2113c05e79208de3f8fda00522031ca097355bb4a97871d879f79ac3\": container with ID starting with c2d0fe0f2113c05e79208de3f8fda00522031ca097355bb4a97871d879f79ac3 not found: ID does not exist"
Sep 15 21:38:49.391091 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.391068    2584 scope.go:117] "RemoveContainer" containerID="6e743941d0a1f495d5ffe2a03932adc03095fdf523a04eb00b3eebf51fb57781"
Sep 15 21:38:49.399844 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.399754    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"]
Sep 15 21:38:49.399899 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.399872    2584 scope.go:117] "RemoveContainer" containerID="6e743941d0a1f495d5ffe2a03932adc03095fdf523a04eb00b3eebf51fb57781"
Sep 15 21:38:49.400131 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:38:49.400113    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e743941d0a1f495d5ffe2a03932adc03095fdf523a04eb00b3eebf51fb57781\": container with ID starting with 6e743941d0a1f495d5ffe2a03932adc03095fdf523a04eb00b3eebf51fb57781 not found: ID does not exist" containerID="6e743941d0a1f495d5ffe2a03932adc03095fdf523a04eb00b3eebf51fb57781"
Sep 15 21:38:49.400183 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.400134    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e743941d0a1f495d5ffe2a03932adc03095fdf523a04eb00b3eebf51fb57781"} err="failed to get container status \"6e743941d0a1f495d5ffe2a03932adc03095fdf523a04eb00b3eebf51fb57781\": rpc error: code = NotFound desc = could not find container \"6e743941d0a1f495d5ffe2a03932adc03095fdf523a04eb00b3eebf51fb57781\": container with ID starting with 6e743941d0a1f495d5ffe2a03932adc03095fdf523a04eb00b3eebf51fb57781 not found: ID does not exist"
Sep 15 21:38:49.404251 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.404231    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/distinct-w4-6f66f7-kserve-cf476d7b8-tf5fd"]
Sep 15 21:38:49.414080 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.414060    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"]
Sep 15 21:38:49.420250 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.420230    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/premium-w4-6f66f7-kserve-5bf58b849c-5pznt"]
Sep 15 21:38:49.430663 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.430619    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"]
Sep 15 21:38:49.434433 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:49.434413    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6f66f7/distinct-2-w4-6f66f7-kserve-6ddf66c755-nfzfh"]
Sep 15 21:38:50.675497 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:50.675469    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8356eab8-04b6-42b2-8a33-f6069e71ac6b" path="/var/lib/kubelet/pods/8356eab8-04b6-42b2-8a33-f6069e71ac6b/volumes"
Sep 15 21:38:50.675901 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:50.675884    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad652e4b-78a3-496e-9cc5-06736f0c6204" path="/var/lib/kubelet/pods/ad652e4b-78a3-496e-9cc5-06736f0c6204/volumes"
Sep 15 21:38:50.676248 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:50.676236    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef0ed4d5-9eca-43a8-b817-18f79a35d213" path="/var/lib/kubelet/pods/ef0ed4d5-9eca-43a8-b817-18f79a35d213/volumes"
Sep 15 21:38:51.132267 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:51.132238    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"]
Sep 15 21:38:51.132479 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:51.132454    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt" podUID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerName="istio-proxy" containerID="cri-o://2e46d419f70b5c2e5584e4843690921117c4d4d40fc45f233da08ba868a56c22" gracePeriod=30
Sep 15 21:38:54.877964 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:54.877930    2584 patch_prober.go:28] interesting pod/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:38:54.878319 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:54.877994    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt" podUID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:38:56.401160 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.401106    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:38:56.418342 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.418319    2584 generic.go:356] "Generic (PLEG): container finished" podID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerID="2e46d419f70b5c2e5584e4843690921117c4d4d40fc45f233da08ba868a56c22" exitCode=0
Sep 15 21:38:56.418467 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.418357    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"
Sep 15 21:38:56.418467 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.418361    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt" event={"ID":"efc6a2e8-4101-46b5-ab18-125c552cd3a1","Type":"ContainerDied","Data":"2e46d419f70b5c2e5584e4843690921117c4d4d40fc45f233da08ba868a56c22"}
Sep 15 21:38:56.418467 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.418432    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt" event={"ID":"efc6a2e8-4101-46b5-ab18-125c552cd3a1","Type":"ContainerDied","Data":"c6cb7ead924de1ec448d98437efef13015da318152cf72001f04df0b42854a5a"}
Sep 15 21:38:56.418467 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.418455    2584 scope.go:117] "RemoveContainer" containerID="2e46d419f70b5c2e5584e4843690921117c4d4d40fc45f233da08ba868a56c22"
Sep 15 21:38:56.427991 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.427969    2584 scope.go:117] "RemoveContainer" containerID="19e481f6870af45588fae621c66bf5cdd14149fdd6916b56e9c770fee5da86d2"
Sep 15 21:38:56.436218 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.436201    2584 scope.go:117] "RemoveContainer" containerID="2e46d419f70b5c2e5584e4843690921117c4d4d40fc45f233da08ba868a56c22"
Sep 15 21:38:56.436534 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:38:56.436515    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e46d419f70b5c2e5584e4843690921117c4d4d40fc45f233da08ba868a56c22\": container with ID starting with 2e46d419f70b5c2e5584e4843690921117c4d4d40fc45f233da08ba868a56c22 not found: ID does not exist" containerID="2e46d419f70b5c2e5584e4843690921117c4d4d40fc45f233da08ba868a56c22"
Sep 15 21:38:56.436601 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.436544    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e46d419f70b5c2e5584e4843690921117c4d4d40fc45f233da08ba868a56c22"} err="failed to get container status \"2e46d419f70b5c2e5584e4843690921117c4d4d40fc45f233da08ba868a56c22\": rpc error: code = NotFound desc = could not find container \"2e46d419f70b5c2e5584e4843690921117c4d4d40fc45f233da08ba868a56c22\": container with ID starting with 2e46d419f70b5c2e5584e4843690921117c4d4d40fc45f233da08ba868a56c22 not found: ID does not exist"
Sep 15 21:38:56.436601 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.436569    2584 scope.go:117] "RemoveContainer" containerID="19e481f6870af45588fae621c66bf5cdd14149fdd6916b56e9c770fee5da86d2"
Sep 15 21:38:56.436798 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:38:56.436782    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e481f6870af45588fae621c66bf5cdd14149fdd6916b56e9c770fee5da86d2\": container with ID starting with 19e481f6870af45588fae621c66bf5cdd14149fdd6916b56e9c770fee5da86d2 not found: ID does not exist" containerID="19e481f6870af45588fae621c66bf5cdd14149fdd6916b56e9c770fee5da86d2"
Sep 15 21:38:56.436840 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.436800    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e481f6870af45588fae621c66bf5cdd14149fdd6916b56e9c770fee5da86d2"} err="failed to get container status \"19e481f6870af45588fae621c66bf5cdd14149fdd6916b56e9c770fee5da86d2\": rpc error: code = NotFound desc = could not find container \"19e481f6870af45588fae621c66bf5cdd14149fdd6916b56e9c770fee5da86d2\": container with ID starting with 19e481f6870af45588fae621c66bf5cdd14149fdd6916b56e9c770fee5da86d2 not found: ID does not exist"
Sep 15 21:38:56.446216 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.446190    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-workload-certs\") pod \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") "
Sep 15 21:38:56.446307 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.446275    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-workload-socket\") pod \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") "
Sep 15 21:38:56.446349 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.446330    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-credential-socket\") pod \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") "
Sep 15 21:38:56.446407 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.446362    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2b89\" (UniqueName: \"kubernetes.io/projected/efc6a2e8-4101-46b5-ab18-125c552cd3a1-kube-api-access-d2b89\") pod \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") "
Sep 15 21:38:56.446474 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.446414    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istiod-ca-cert\") pod \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") "
Sep 15 21:38:56.446474 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.446439    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-podinfo\") pod \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") "
Sep 15 21:38:56.446574 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.446495    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "efc6a2e8-4101-46b5-ab18-125c552cd3a1" (UID: "efc6a2e8-4101-46b5-ab18-125c552cd3a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:56.446574 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.446500    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-envoy\") pod \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") "
Sep 15 21:38:56.446672 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.446586    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-token\") pod \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") "
Sep 15 21:38:56.446782 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.446687    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-data\") pod \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\" (UID: \"efc6a2e8-4101-46b5-ab18-125c552cd3a1\") "
Sep 15 21:38:56.446842 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.446777    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "efc6a2e8-4101-46b5-ab18-125c552cd3a1" (UID: "efc6a2e8-4101-46b5-ab18-125c552cd3a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:56.447178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.446948    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "efc6a2e8-4101-46b5-ab18-125c552cd3a1" (UID: "efc6a2e8-4101-46b5-ab18-125c552cd3a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:56.447178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.447122    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-workload-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:56.447178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.447128    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-data" (OuterVolumeSpecName: "istio-data") pod "efc6a2e8-4101-46b5-ab18-125c552cd3a1" (UID: "efc6a2e8-4101-46b5-ab18-125c552cd3a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:56.447178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.447142    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-credential-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:56.447178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.447158    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-workload-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:56.447178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.447168    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "efc6a2e8-4101-46b5-ab18-125c552cd3a1" (UID: "efc6a2e8-4101-46b5-ab18-125c552cd3a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:38:56.448664 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.448634    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "efc6a2e8-4101-46b5-ab18-125c552cd3a1" (UID: "efc6a2e8-4101-46b5-ab18-125c552cd3a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:38:56.449530 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.449489    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "efc6a2e8-4101-46b5-ab18-125c552cd3a1" (UID: "efc6a2e8-4101-46b5-ab18-125c552cd3a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:56.449729 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.449701    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-token" (OuterVolumeSpecName: "istio-token") pod "efc6a2e8-4101-46b5-ab18-125c552cd3a1" (UID: "efc6a2e8-4101-46b5-ab18-125c552cd3a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:56.449828 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.449810    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc6a2e8-4101-46b5-ab18-125c552cd3a1-kube-api-access-d2b89" (OuterVolumeSpecName: "kube-api-access-d2b89") pod "efc6a2e8-4101-46b5-ab18-125c552cd3a1" (UID: "efc6a2e8-4101-46b5-ab18-125c552cd3a1"). InnerVolumeSpecName "kube-api-access-d2b89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:56.548263 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.548241    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-envoy\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:56.548263 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.548263    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-token\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:56.548443 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.548271    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-data\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:56.548443 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.548279    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2b89\" (UniqueName: \"kubernetes.io/projected/efc6a2e8-4101-46b5-ab18-125c552cd3a1-kube-api-access-d2b89\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:56.548443 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.548288    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istiod-ca-cert\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:56.548443 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.548296    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc6a2e8-4101-46b5-ab18-125c552cd3a1-istio-podinfo\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:38:56.734606 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.734575    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"]
Sep 15 21:38:56.737751 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:56.737727    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6f66f7-openshift-default-655548bdf4-fnbpt"]
Sep 15 21:38:58.673233 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:38:58.673201    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" path="/var/lib/kubelet/pods/efc6a2e8-4101-46b5-ab18-125c552cd3a1/volumes"
Sep 15 21:39:21.113291 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113258    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"]
Sep 15 21:39:21.113871 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113799    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerName="istio-proxy"
Sep 15 21:39:21.113871 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113816    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerName="istio-proxy"
Sep 15 21:39:21.113871 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113833    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87d7733-ca98-4d75-9972-43387262625c" containerName="revoke-keys"
Sep 15 21:39:21.113871 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113838    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87d7733-ca98-4d75-9972-43387262625c" containerName="revoke-keys"
Sep 15 21:39:21.113871 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113846    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerName="istio-proxy"
Sep 15 21:39:21.113871 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113851    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerName="istio-proxy"
Sep 15 21:39:21.113871 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113863    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d1794ad-3eb4-450e-aaa9-424d0fe894eb" containerName="payload-pre-processing"
Sep 15 21:39:21.113871 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113868    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d1794ad-3eb4-450e-aaa9-424d0fe894eb" containerName="payload-pre-processing"
Sep 15 21:39:21.114204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113883    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef0ed4d5-9eca-43a8-b817-18f79a35d213" containerName="main"
Sep 15 21:39:21.114204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113892    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef0ed4d5-9eca-43a8-b817-18f79a35d213" containerName="main"
Sep 15 21:39:21.114204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113904    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8356eab8-04b6-42b2-8a33-f6069e71ac6b" containerName="main"
Sep 15 21:39:21.114204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113909    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8356eab8-04b6-42b2-8a33-f6069e71ac6b" containerName="main"
Sep 15 21:39:21.114204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113917    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad652e4b-78a3-496e-9cc5-06736f0c6204" containerName="main"
Sep 15 21:39:21.114204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113921    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad652e4b-78a3-496e-9cc5-06736f0c6204" containerName="main"
Sep 15 21:39:21.114204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113985    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d1794ad-3eb4-450e-aaa9-424d0fe894eb" containerName="payload-pre-processing"
Sep 15 21:39:21.114204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.113995    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87d7733-ca98-4d75-9972-43387262625c" containerName="revoke-keys"
Sep 15 21:39:21.114204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.114002    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerName="istio-proxy"
Sep 15 21:39:21.114204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.114007    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad652e4b-78a3-496e-9cc5-06736f0c6204" containerName="main"
Sep 15 21:39:21.114204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.114014    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc6a2e8-4101-46b5-ab18-125c552cd3a1" containerName="istio-proxy"
Sep 15 21:39:21.114204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.114021    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8356eab8-04b6-42b2-8a33-f6069e71ac6b" containerName="main"
Sep 15 21:39:21.114204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.114026    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef0ed4d5-9eca-43a8-b817-18f79a35d213" containerName="main"
Sep 15 21:39:21.117211 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.117188    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.119615 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.119592    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f36399\"/\"default-dockercfg-cmswh\""
Sep 15 21:39:21.120000 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.119592    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f36399\"/\"openshift-service-ca.crt\""
Sep 15 21:39:21.120076 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.120059    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f36399\"/\"kube-root-ca.crt\""
Sep 15 21:39:21.120241 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.120220    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f36399\"/\"sub-test-model-f36399-kserve-self-signed-certs\""
Sep 15 21:39:21.124743 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.124719    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"]
Sep 15 21:39:21.272096 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.272068    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-556nl\" (UniqueName: \"kubernetes.io/projected/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-kube-api-access-556nl\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.272289 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.272102    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-dshm\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.272289 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.272234    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-tls-certs\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.272289 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.272278    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-tmp-dir\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.272466 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.272302    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-model-cache\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.272466 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.272372    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-home\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.373749 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.373653    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-tls-certs\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.373749 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.373702    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-tmp-dir\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.373979 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.373754    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-model-cache\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.373979 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.373855    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-home\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.373979 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.373912    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556nl\" (UniqueName: \"kubernetes.io/projected/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-kube-api-access-556nl\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.373979 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.373945    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-dshm\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.374235 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.374191    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-tmp-dir\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.374421 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.374371    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-home\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.374681 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.374661    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-model-cache\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.376746 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.376721    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-tls-certs\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.376861 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.376843    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-dshm\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.382000 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.381980    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-556nl\" (UniqueName: \"kubernetes.io/projected/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-kube-api-access-556nl\") pod \"sub-test-model-f36399-kserve-d94656659-nlwnt\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") " pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.428737 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.428717    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:21.703548 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:21.703522    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"]
Sep 15 21:39:21.705553 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:39:21.705528    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4d0e564_767a_4e8a_b4b7_60f52cab2b85.slice/crio-9f6e3015ce8aeddae17ba7e13ec4278ffe90560933a49d5201eff30b734e14e6 WatchSource:0}: Error finding container 9f6e3015ce8aeddae17ba7e13ec4278ffe90560933a49d5201eff30b734e14e6: Status 404 returned error can't find the container with id 9f6e3015ce8aeddae17ba7e13ec4278ffe90560933a49d5201eff30b734e14e6
Sep 15 21:39:22.572848 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:22.572810    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt" event={"ID":"a4d0e564-767a-4e8a-b4b7-60f52cab2b85","Type":"ContainerStarted","Data":"8e62b2e2b1f33f4626659ed2d01acab614445df7a9daa43b2d86dbbb0e375b28"}
Sep 15 21:39:22.572848 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:22.572854    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt" event={"ID":"a4d0e564-767a-4e8a-b4b7-60f52cab2b85","Type":"ContainerStarted","Data":"9f6e3015ce8aeddae17ba7e13ec4278ffe90560933a49d5201eff30b734e14e6"}
Sep 15 21:39:22.590500 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:22.590457    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt" podStartSLOduration=1.590441634 podStartE2EDuration="1.590441634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:22.588572164 +0000 UTC m=+1730.450505970" watchObservedRunningTime="2026-09-15 21:39:22.590441634 +0000 UTC m=+1730.452375439"
Sep 15 21:39:31.429822 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:31.429785    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:31.430204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:31.429869    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:31.441897 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:31.441876    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:31.634863 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:31.634836    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:39:33.031426 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:33.031370    2584 scope.go:117] "RemoveContainer" containerID="b787c31a5ebf7f0e7fed1750524ba6f42d7a5e02dd285c34adf60dd738765aee"
Sep 15 21:39:33.039978 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:39:33.039952    2584 scope.go:117] "RemoveContainer" containerID="ecd116531c520e075aed87e768646ed7a73c5b51b1d1b278ceab547aab0198d4"
Sep 15 21:40:32.721444 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:32.721412    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:40:32.727450 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:32.727425    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:40:45.760996 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:45.760960    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7854-184ca3"]
Sep 15 21:40:45.764533 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:45.764517    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-184ca3"
Sep 15 21:40:45.766959 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:45.766827    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ncvz5\""
Sep 15 21:40:45.769500 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:45.769477    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7854-184ca3"]
Sep 15 21:40:45.886318 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:45.886290    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xkpz\" (UniqueName: \"kubernetes.io/projected/0a8a5ab3-c326-414e-b30b-0c595db31cd6-kube-api-access-6xkpz\") pod \"test-curl-7854-184ca3\" (UID: \"0a8a5ab3-c326-414e-b30b-0c595db31cd6\") " pod="opendatahub/test-curl-7854-184ca3"
Sep 15 21:40:45.987547 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:45.987515    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xkpz\" (UniqueName: \"kubernetes.io/projected/0a8a5ab3-c326-414e-b30b-0c595db31cd6-kube-api-access-6xkpz\") pod \"test-curl-7854-184ca3\" (UID: \"0a8a5ab3-c326-414e-b30b-0c595db31cd6\") " pod="opendatahub/test-curl-7854-184ca3"
Sep 15 21:40:45.995910 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:45.995889    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xkpz\" (UniqueName: \"kubernetes.io/projected/0a8a5ab3-c326-414e-b30b-0c595db31cd6-kube-api-access-6xkpz\") pod \"test-curl-7854-184ca3\" (UID: \"0a8a5ab3-c326-414e-b30b-0c595db31cd6\") " pod="opendatahub/test-curl-7854-184ca3"
Sep 15 21:40:46.075097 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:46.075073    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-184ca3"
Sep 15 21:40:46.342774 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:46.342748    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7854-184ca3"]
Sep 15 21:40:46.344366 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:40:46.344339    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a8a5ab3_c326_414e_b30b_0c595db31cd6.slice/crio-3e16fcc2a85bef997dee3fcaa244dc8c5f54a4ae48c8d3d409491c56992f6dbd WatchSource:0}: Error finding container 3e16fcc2a85bef997dee3fcaa244dc8c5f54a4ae48c8d3d409491c56992f6dbd: Status 404 returned error can't find the container with id 3e16fcc2a85bef997dee3fcaa244dc8c5f54a4ae48c8d3d409491c56992f6dbd
Sep 15 21:40:46.346437 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:46.346422    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:40:47.028628 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:47.028589    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7854-184ca3" event={"ID":"0a8a5ab3-c326-414e-b30b-0c595db31cd6","Type":"ContainerStarted","Data":"3e16fcc2a85bef997dee3fcaa244dc8c5f54a4ae48c8d3d409491c56992f6dbd"}
Sep 15 21:40:49.040971 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:49.040935    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7854-184ca3" event={"ID":"0a8a5ab3-c326-414e-b30b-0c595db31cd6","Type":"ContainerStarted","Data":"340ceed657c5dc6a036a288aee89ec51c863fb648676a39bbc591b137f77bda5"}
Sep 15 21:40:49.055100 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:49.055050    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7854-184ca3" podStartSLOduration=1.98352423 podStartE2EDuration="4.055029769s" totalImagesPullingTime="2.071505539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:40:49.054794494 +0000 UTC m=+1816.916728301" watchObservedRunningTime="2026-09-15 21:40:49.055029769 +0000 UTC m=+1816.916963575"
Sep 15 21:40:49.634655 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:49.634615    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7854-184ca3"]
Sep 15 21:40:49.638942 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:49.638920    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7854-184ca3"]
Sep 15 21:40:49.749423 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:49.749327    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7854-bc1750"]
Sep 15 21:40:49.749764 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:49.749749    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a8a5ab3-c326-414e-b30b-0c595db31cd6" containerName="test-curl-7854-184ca3"
Sep 15 21:40:49.749764 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:49.749764    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a8a5ab3-c326-414e-b30b-0c595db31cd6" containerName="test-curl-7854-184ca3"
Sep 15 21:40:49.749896 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:49.749846    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a8a5ab3-c326-414e-b30b-0c595db31cd6" containerName="test-curl-7854-184ca3"
Sep 15 21:40:49.753033 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:49.753011    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-bc1750"
Sep 15 21:40:49.756841 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:49.756815    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7854-bc1750"]
Sep 15 21:40:49.926427 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:49.926374    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99b8l\" (UniqueName: \"kubernetes.io/projected/1b2bf98c-61bf-4449-a1df-e9dc7f73f94c-kube-api-access-99b8l\") pod \"test-curl-7854-bc1750\" (UID: \"1b2bf98c-61bf-4449-a1df-e9dc7f73f94c\") " pod="opendatahub/test-curl-7854-bc1750"
Sep 15 21:40:50.027472 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:50.027441    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99b8l\" (UniqueName: \"kubernetes.io/projected/1b2bf98c-61bf-4449-a1df-e9dc7f73f94c-kube-api-access-99b8l\") pod \"test-curl-7854-bc1750\" (UID: \"1b2bf98c-61bf-4449-a1df-e9dc7f73f94c\") " pod="opendatahub/test-curl-7854-bc1750"
Sep 15 21:40:50.035081 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:50.035052    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99b8l\" (UniqueName: \"kubernetes.io/projected/1b2bf98c-61bf-4449-a1df-e9dc7f73f94c-kube-api-access-99b8l\") pod \"test-curl-7854-bc1750\" (UID: \"1b2bf98c-61bf-4449-a1df-e9dc7f73f94c\") " pod="opendatahub/test-curl-7854-bc1750"
Sep 15 21:40:50.047124 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:40:50.047095    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7854-184ca3" container="test-curl-7854-184ca3"
Sep 15 21:40:50.064936 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:50.064918    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-bc1750"
Sep 15 21:40:50.332772 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:50.332746    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7854-bc1750"]
Sep 15 21:40:50.334485 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:40:50.334458    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b2bf98c_61bf_4449_a1df_e9dc7f73f94c.slice/crio-4bda3d21eb834e6ee631bb858da71bf83f940d8daa48ac81cae4b9e381887036 WatchSource:0}: Error finding container 4bda3d21eb834e6ee631bb858da71bf83f940d8daa48ac81cae4b9e381887036: Status 404 returned error can't find the container with id 4bda3d21eb834e6ee631bb858da71bf83f940d8daa48ac81cae4b9e381887036
Sep 15 21:40:51.054071 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:51.054036    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7854-bc1750" event={"ID":"1b2bf98c-61bf-4449-a1df-e9dc7f73f94c","Type":"ContainerStarted","Data":"599db76fbbc6790aaaa3259a18daafc0568909a571b67d38e2ee330c4516732d"}
Sep 15 21:40:51.054071 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:51.054074    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7854-bc1750" event={"ID":"1b2bf98c-61bf-4449-a1df-e9dc7f73f94c","Type":"ContainerStarted","Data":"4bda3d21eb834e6ee631bb858da71bf83f940d8daa48ac81cae4b9e381887036"}
Sep 15 21:40:51.054593 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:51.054150    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7854-184ca3" podUID="0a8a5ab3-c326-414e-b30b-0c595db31cd6" containerName="test-curl-7854-184ca3" containerID="cri-o://340ceed657c5dc6a036a288aee89ec51c863fb648676a39bbc591b137f77bda5" gracePeriod=2
Sep 15 21:40:51.067569 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:51.067493    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7854-bc1750" podStartSLOduration=2.067478201 podStartE2EDuration="2.067478201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:40:51.066865693 +0000 UTC m=+1818.928799499" watchObservedRunningTime="2026-09-15 21:40:51.067478201 +0000 UTC m=+1818.929412006"
Sep 15 21:40:51.629753 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:51.629717    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7854-bc1750"]
Sep 15 21:40:51.632967 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:51.632938    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7854-bc1750"]
Sep 15 21:40:52.059257 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:40:52.059224    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7854-bc1750" container="test-curl-7854-bc1750"
Sep 15 21:40:53.065225 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:53.065168    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7854-bc1750" podUID="1b2bf98c-61bf-4449-a1df-e9dc7f73f94c" containerName="test-curl-7854-bc1750" containerID="cri-o://599db76fbbc6790aaaa3259a18daafc0568909a571b67d38e2ee330c4516732d" gracePeriod=2
Sep 15 21:40:53.244938 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:53.244915    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-184ca3"
Sep 15 21:40:53.257469 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:53.257447    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xkpz\" (UniqueName: \"kubernetes.io/projected/0a8a5ab3-c326-414e-b30b-0c595db31cd6-kube-api-access-6xkpz\") pod \"0a8a5ab3-c326-414e-b30b-0c595db31cd6\" (UID: \"0a8a5ab3-c326-414e-b30b-0c595db31cd6\") "
Sep 15 21:40:53.259521 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:53.259500    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8a5ab3-c326-414e-b30b-0c595db31cd6-kube-api-access-6xkpz" (OuterVolumeSpecName: "kube-api-access-6xkpz") pod "0a8a5ab3-c326-414e-b30b-0c595db31cd6" (UID: "0a8a5ab3-c326-414e-b30b-0c595db31cd6"). InnerVolumeSpecName "kube-api-access-6xkpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:40:53.358450 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:53.358371    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xkpz\" (UniqueName: \"kubernetes.io/projected/0a8a5ab3-c326-414e-b30b-0c595db31cd6-kube-api-access-6xkpz\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:40:53.629424 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:53.629317    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"]
Sep 15 21:40:54.071816 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:54.071782    2584 generic.go:356] "Generic (PLEG): container finished" podID="0a8a5ab3-c326-414e-b30b-0c595db31cd6" containerID="340ceed657c5dc6a036a288aee89ec51c863fb648676a39bbc591b137f77bda5" exitCode=137
Sep 15 21:40:54.072222 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:54.071820    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-184ca3"
Sep 15 21:40:54.072222 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:54.071876    2584 scope.go:117] "RemoveContainer" containerID="340ceed657c5dc6a036a288aee89ec51c863fb648676a39bbc591b137f77bda5"
Sep 15 21:40:54.083580 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:40:54.083553    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7854-184ca3\" is forbidden: User \"system:node:ip-10-0-139-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-184.ec2.internal' and this object" podUID="0a8a5ab3-c326-414e-b30b-0c595db31cd6" pod="opendatahub/test-curl-7854-184ca3"
Sep 15 21:40:54.083656 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:54.083573    2584 scope.go:117] "RemoveContainer" containerID="340ceed657c5dc6a036a288aee89ec51c863fb648676a39bbc591b137f77bda5"
Sep 15 21:40:54.084067 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:40:54.084043    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340ceed657c5dc6a036a288aee89ec51c863fb648676a39bbc591b137f77bda5\": container with ID starting with 340ceed657c5dc6a036a288aee89ec51c863fb648676a39bbc591b137f77bda5 not found: ID does not exist" containerID="340ceed657c5dc6a036a288aee89ec51c863fb648676a39bbc591b137f77bda5"
Sep 15 21:40:54.084133 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:54.084078    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340ceed657c5dc6a036a288aee89ec51c863fb648676a39bbc591b137f77bda5"} err="failed to get container status \"340ceed657c5dc6a036a288aee89ec51c863fb648676a39bbc591b137f77bda5\": rpc error: code = NotFound desc = could not find container \"340ceed657c5dc6a036a288aee89ec51c863fb648676a39bbc591b137f77bda5\": container with ID starting with 340ceed657c5dc6a036a288aee89ec51c863fb648676a39bbc591b137f77bda5 not found: ID does not exist"
Sep 15 21:40:54.687094 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:54.687058    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a8a5ab3-c326-414e-b30b-0c595db31cd6" path="/var/lib/kubelet/pods/0a8a5ab3-c326-414e-b30b-0c595db31cd6/volumes"
Sep 15 21:40:55.246744 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:55.246721    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-bc1750"
Sep 15 21:40:55.279637 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:55.279611    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99b8l\" (UniqueName: \"kubernetes.io/projected/1b2bf98c-61bf-4449-a1df-e9dc7f73f94c-kube-api-access-99b8l\") pod \"1b2bf98c-61bf-4449-a1df-e9dc7f73f94c\" (UID: \"1b2bf98c-61bf-4449-a1df-e9dc7f73f94c\") "
Sep 15 21:40:55.281589 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:55.281561    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2bf98c-61bf-4449-a1df-e9dc7f73f94c-kube-api-access-99b8l" (OuterVolumeSpecName: "kube-api-access-99b8l") pod "1b2bf98c-61bf-4449-a1df-e9dc7f73f94c" (UID: "1b2bf98c-61bf-4449-a1df-e9dc7f73f94c"). InnerVolumeSpecName "kube-api-access-99b8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:40:55.381103 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:55.381036    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99b8l\" (UniqueName: \"kubernetes.io/projected/1b2bf98c-61bf-4449-a1df-e9dc7f73f94c-kube-api-access-99b8l\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:40:56.086119 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:56.086078    2584 generic.go:356] "Generic (PLEG): container finished" podID="1b2bf98c-61bf-4449-a1df-e9dc7f73f94c" containerID="599db76fbbc6790aaaa3259a18daafc0568909a571b67d38e2ee330c4516732d" exitCode=137
Sep 15 21:40:56.086119 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:56.086118    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-bc1750"
Sep 15 21:40:56.086355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:56.086143    2584 scope.go:117] "RemoveContainer" containerID="599db76fbbc6790aaaa3259a18daafc0568909a571b67d38e2ee330c4516732d"
Sep 15 21:40:56.095277 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:56.095261    2584 scope.go:117] "RemoveContainer" containerID="599db76fbbc6790aaaa3259a18daafc0568909a571b67d38e2ee330c4516732d"
Sep 15 21:40:56.095558 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:40:56.095540    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599db76fbbc6790aaaa3259a18daafc0568909a571b67d38e2ee330c4516732d\": container with ID starting with 599db76fbbc6790aaaa3259a18daafc0568909a571b67d38e2ee330c4516732d not found: ID does not exist" containerID="599db76fbbc6790aaaa3259a18daafc0568909a571b67d38e2ee330c4516732d"
Sep 15 21:40:56.095613 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:56.095566    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599db76fbbc6790aaaa3259a18daafc0568909a571b67d38e2ee330c4516732d"} err="failed to get container status \"599db76fbbc6790aaaa3259a18daafc0568909a571b67d38e2ee330c4516732d\": rpc error: code = NotFound desc = could not find container \"599db76fbbc6790aaaa3259a18daafc0568909a571b67d38e2ee330c4516732d\": container with ID starting with 599db76fbbc6790aaaa3259a18daafc0568909a571b67d38e2ee330c4516732d not found: ID does not exist"
Sep 15 21:40:56.096179 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:40:56.096159    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7854-bc1750\" is forbidden: User \"system:node:ip-10-0-139-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-184.ec2.internal' and this object" podUID="1b2bf98c-61bf-4449-a1df-e9dc7f73f94c" pod="opendatahub/test-curl-7854-bc1750"
Sep 15 21:40:56.673782 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:40:56.673748    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b2bf98c-61bf-4449-a1df-e9dc7f73f94c" path="/var/lib/kubelet/pods/1b2bf98c-61bf-4449-a1df-e9dc7f73f94c/volumes"
Sep 15 21:41:08.648542 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:08.648484    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt" podUID="a4d0e564-767a-4e8a-b4b7-60f52cab2b85" containerName="main" containerID="cri-o://8e62b2e2b1f33f4626659ed2d01acab614445df7a9daa43b2d86dbbb0e375b28" gracePeriod=45
Sep 15 21:41:08.908130 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:08.908100    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:41:09.005916 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.005886    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-dshm\") pod \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") "
Sep 15 21:41:09.005916 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.005918    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-556nl\" (UniqueName: \"kubernetes.io/projected/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-kube-api-access-556nl\") pod \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") "
Sep 15 21:41:09.006140 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.005963    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-model-cache\") pod \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") "
Sep 15 21:41:09.006140 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.005984    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-tmp-dir\") pod \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") "
Sep 15 21:41:09.006140 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.006030    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-tls-certs\") pod \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") "
Sep 15 21:41:09.006140 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.006066    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-home\") pod \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\" (UID: \"a4d0e564-767a-4e8a-b4b7-60f52cab2b85\") "
Sep 15 21:41:09.006435 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.006275    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-model-cache" (OuterVolumeSpecName: "model-cache") pod "a4d0e564-767a-4e8a-b4b7-60f52cab2b85" (UID: "a4d0e564-767a-4e8a-b4b7-60f52cab2b85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:41:09.006435 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.006298    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a4d0e564-767a-4e8a-b4b7-60f52cab2b85" (UID: "a4d0e564-767a-4e8a-b4b7-60f52cab2b85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:41:09.006435 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.006339    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-home" (OuterVolumeSpecName: "home") pod "a4d0e564-767a-4e8a-b4b7-60f52cab2b85" (UID: "a4d0e564-767a-4e8a-b4b7-60f52cab2b85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:41:09.007997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.007981    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-dshm" (OuterVolumeSpecName: "dshm") pod "a4d0e564-767a-4e8a-b4b7-60f52cab2b85" (UID: "a4d0e564-767a-4e8a-b4b7-60f52cab2b85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:41:09.008464 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.008434    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4d0e564-767a-4e8a-b4b7-60f52cab2b85" (UID: "a4d0e564-767a-4e8a-b4b7-60f52cab2b85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:41:09.008553 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.008443    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-kube-api-access-556nl" (OuterVolumeSpecName: "kube-api-access-556nl") pod "a4d0e564-767a-4e8a-b4b7-60f52cab2b85" (UID: "a4d0e564-767a-4e8a-b4b7-60f52cab2b85"). InnerVolumeSpecName "kube-api-access-556nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:41:09.107030 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.106996    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-home\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:41:09.107030 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.107029    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-dshm\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:41:09.107030 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.107039    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-556nl\" (UniqueName: \"kubernetes.io/projected/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-kube-api-access-556nl\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:41:09.107207 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.107049    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-model-cache\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:41:09.107207 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.107076    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-tmp-dir\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:41:09.107207 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.107084    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d0e564-767a-4e8a-b4b7-60f52cab2b85-tls-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:41:09.159352 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.159323    2584 generic.go:356] "Generic (PLEG): container finished" podID="a4d0e564-767a-4e8a-b4b7-60f52cab2b85" containerID="8e62b2e2b1f33f4626659ed2d01acab614445df7a9daa43b2d86dbbb0e375b28" exitCode=0
Sep 15 21:41:09.159493 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.159369    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"
Sep 15 21:41:09.159493 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.159371    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt" event={"ID":"a4d0e564-767a-4e8a-b4b7-60f52cab2b85","Type":"ContainerDied","Data":"8e62b2e2b1f33f4626659ed2d01acab614445df7a9daa43b2d86dbbb0e375b28"}
Sep 15 21:41:09.159493 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.159441    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt" event={"ID":"a4d0e564-767a-4e8a-b4b7-60f52cab2b85","Type":"ContainerDied","Data":"9f6e3015ce8aeddae17ba7e13ec4278ffe90560933a49d5201eff30b734e14e6"}
Sep 15 21:41:09.159493 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.159464    2584 scope.go:117] "RemoveContainer" containerID="8e62b2e2b1f33f4626659ed2d01acab614445df7a9daa43b2d86dbbb0e375b28"
Sep 15 21:41:09.169591 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.169574    2584 scope.go:117] "RemoveContainer" containerID="8e62b2e2b1f33f4626659ed2d01acab614445df7a9daa43b2d86dbbb0e375b28"
Sep 15 21:41:09.169851 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:41:09.169832    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e62b2e2b1f33f4626659ed2d01acab614445df7a9daa43b2d86dbbb0e375b28\": container with ID starting with 8e62b2e2b1f33f4626659ed2d01acab614445df7a9daa43b2d86dbbb0e375b28 not found: ID does not exist" containerID="8e62b2e2b1f33f4626659ed2d01acab614445df7a9daa43b2d86dbbb0e375b28"
Sep 15 21:41:09.169914 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.169862    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e62b2e2b1f33f4626659ed2d01acab614445df7a9daa43b2d86dbbb0e375b28"} err="failed to get container status \"8e62b2e2b1f33f4626659ed2d01acab614445df7a9daa43b2d86dbbb0e375b28\": rpc error: code = NotFound desc = could not find container \"8e62b2e2b1f33f4626659ed2d01acab614445df7a9daa43b2d86dbbb0e375b28\": container with ID starting with 8e62b2e2b1f33f4626659ed2d01acab614445df7a9daa43b2d86dbbb0e375b28 not found: ID does not exist"
Sep 15 21:41:09.183186 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.183163    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"]
Sep 15 21:41:09.187036 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:09.187015    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f36399/sub-test-model-f36399-kserve-d94656659-nlwnt"]
Sep 15 21:41:10.672834 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:10.672801    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4d0e564-767a-4e8a-b4b7-60f52cab2b85" path="/var/lib/kubelet/pods/a4d0e564-767a-4e8a-b4b7-60f52cab2b85/volumes"
Sep 15 21:41:28.908282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:28.908245    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"]
Sep 15 21:41:28.909491 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:28.909463    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b2bf98c-61bf-4449-a1df-e9dc7f73f94c" containerName="test-curl-7854-bc1750"
Sep 15 21:41:28.909491 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:28.909494    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b2bf98c-61bf-4449-a1df-e9dc7f73f94c" containerName="test-curl-7854-bc1750"
Sep 15 21:41:28.909644 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:28.909545    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4d0e564-767a-4e8a-b4b7-60f52cab2b85" containerName="main"
Sep 15 21:41:28.909644 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:28.909554    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4d0e564-767a-4e8a-b4b7-60f52cab2b85" containerName="main"
Sep 15 21:41:28.909713 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:28.909657    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4d0e564-767a-4e8a-b4b7-60f52cab2b85" containerName="main"
Sep 15 21:41:28.909713 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:28.909671    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b2bf98c-61bf-4449-a1df-e9dc7f73f94c" containerName="test-curl-7854-bc1750"
Sep 15 21:41:28.918310 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:28.918286    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:28.919338 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:28.919313    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"]
Sep 15 21:41:28.921449 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:28.921282    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-745dff\"/\"openshift-service-ca.crt\""
Sep 15 21:41:28.921449 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:28.921411    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-745dff\"/\"default-dockercfg-clz6r\""
Sep 15 21:41:28.921636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:28.921563    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-745dff\"/\"kube-root-ca.crt\""
Sep 15 21:41:28.921636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:28.921287    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-745dff\"/\"rate-test-model-745dff-kserve-self-signed-certs\""
Sep 15 21:41:29.080265 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.080231    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b482bd0-d481-4d69-8626-cd64fb43580f-tls-certs\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.080265 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.080270    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-tmp-dir\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.080503 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.080292    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5zm5\" (UniqueName: \"kubernetes.io/projected/6b482bd0-d481-4d69-8626-cd64fb43580f-kube-api-access-r5zm5\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.080503 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.080351    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-model-cache\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.080503 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.080470    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-home\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.080503 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.080497    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-dshm\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.181087 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.181016    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-model-cache\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.181087 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.181078    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-home\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.181284 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.181097    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-dshm\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.181284 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.181130    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b482bd0-d481-4d69-8626-cd64fb43580f-tls-certs\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.181284 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.181152    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-tmp-dir\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.181284 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.181173    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zm5\" (UniqueName: \"kubernetes.io/projected/6b482bd0-d481-4d69-8626-cd64fb43580f-kube-api-access-r5zm5\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.181542 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.181449    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-model-cache\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.181542 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.181519    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-home\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.181623 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.181545    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-tmp-dir\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.183493 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.183473    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-dshm\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.183680 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.183665    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b482bd0-d481-4d69-8626-cd64fb43580f-tls-certs\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.188505 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.188482    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zm5\" (UniqueName: \"kubernetes.io/projected/6b482bd0-d481-4d69-8626-cd64fb43580f-kube-api-access-r5zm5\") pod \"rate-test-model-745dff-kserve-8496f87755-28l87\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") " pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.229247 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.229222    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:29.503711 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:29.503684    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"]
Sep 15 21:41:29.505335 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:41:29.505304    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b482bd0_d481_4d69_8626_cd64fb43580f.slice/crio-3e91e911c1763a997e341e3af8f611262ad497c826a88ffeef327688af2a8702 WatchSource:0}: Error finding container 3e91e911c1763a997e341e3af8f611262ad497c826a88ffeef327688af2a8702: Status 404 returned error can't find the container with id 3e91e911c1763a997e341e3af8f611262ad497c826a88ffeef327688af2a8702
Sep 15 21:41:30.275396 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:30.275351    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87" event={"ID":"6b482bd0-d481-4d69-8626-cd64fb43580f","Type":"ContainerStarted","Data":"b76e8973f5418db5126819872fdab57158cf86f304109e0894056e3c1563e478"}
Sep 15 21:41:30.275834 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:30.275419    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87" event={"ID":"6b482bd0-d481-4d69-8626-cd64fb43580f","Type":"ContainerStarted","Data":"3e91e911c1763a997e341e3af8f611262ad497c826a88ffeef327688af2a8702"}
Sep 15 21:41:39.229990 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:39.229961    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:39.230447 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:39.230007    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:39.242177 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:39.242154    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:41:39.258103 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:39.258054    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87" podStartSLOduration=11.258042004 podStartE2EDuration="11.258042004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:30.290602034 +0000 UTC m=+1858.152535839" watchObservedRunningTime="2026-09-15 21:41:39.258042004 +0000 UTC m=+1867.119975806"
Sep 15 21:41:39.336801 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:41:39.336781    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:42:02.291341 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:02.291263    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"]
Sep 15 21:42:03.041630 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.041594    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"]
Sep 15 21:42:03.045470 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.045437    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.047730 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.047710    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6d9d39-openshift-default-dockercfg-fd4fm\""
Sep 15 21:42:03.056223 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.056199    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"]
Sep 15 21:42:03.187009 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.186977    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istiod-ca-cert\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.187178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.187031    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-data\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.187178 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.187148    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-token\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.187258 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.187197    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-workload-socket\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.187258 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.187234    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-credential-socket\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.187350 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.187266    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-podinfo\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.187350 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.187297    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-workload-certs\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.187350 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.187320    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqjjh\" (UniqueName: \"kubernetes.io/projected/192e4d7e-dfc0-46bb-a356-4b5f479113a4-kube-api-access-hqjjh\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.187497 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.187353    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-envoy\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.288676 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.288638    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-token\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.288838 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.288686    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-workload-socket\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.288838 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.288714    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-credential-socket\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.288838 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.288744    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-podinfo\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.288838 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.288800    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-workload-certs\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.288838 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.288826    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqjjh\" (UniqueName: \"kubernetes.io/projected/192e4d7e-dfc0-46bb-a356-4b5f479113a4-kube-api-access-hqjjh\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.289115 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.288860    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-envoy\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.289115 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.288938    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istiod-ca-cert\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.289115 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.288994    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-data\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.289115 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.289094    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-workload-socket\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.289311 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.289242    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-credential-socket\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.289438 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.289419    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-workload-certs\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.289575 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.289558    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-data\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.289715 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.289692    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istiod-ca-cert\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.291227 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.291197    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-envoy\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.291317 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.291277    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-podinfo\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.297221 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.297157    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqjjh\" (UniqueName: \"kubernetes.io/projected/192e4d7e-dfc0-46bb-a356-4b5f479113a4-kube-api-access-hqjjh\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.297571 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.297432    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-token\") pod \"e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") " pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.357229 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.357206    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:03.637808 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.637784    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"]
Sep 15 21:42:03.639530 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:42:03.639505    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192e4d7e_dfc0_46bb_a356_4b5f479113a4.slice/crio-601d356140a7ac478f481aba305c8a5c66fba9e5bc40d3da66ca504c812bdf3c WatchSource:0}: Error finding container 601d356140a7ac478f481aba305c8a5c66fba9e5bc40d3da66ca504c812bdf3c: Status 404 returned error can't find the container with id 601d356140a7ac478f481aba305c8a5c66fba9e5bc40d3da66ca504c812bdf3c
Sep 15 21:42:03.641767 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.641739    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:42:03.641835 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.641801    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:42:03.641835 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:03.641827    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:42:04.466021 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:04.465985    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt" event={"ID":"192e4d7e-dfc0-46bb-a356-4b5f479113a4","Type":"ContainerStarted","Data":"95f851b92ccad588b784ffe6b0e3351b786e27b89b2b874bc94c0fd55002d8eb"}
Sep 15 21:42:04.466481 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:04.466048    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt" event={"ID":"192e4d7e-dfc0-46bb-a356-4b5f479113a4","Type":"ContainerStarted","Data":"601d356140a7ac478f481aba305c8a5c66fba9e5bc40d3da66ca504c812bdf3c"}
Sep 15 21:42:04.485530 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:04.485489    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt" podStartSLOduration=1.485474177 podStartE2EDuration="1.485474177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:42:04.483976376 +0000 UTC m=+1892.345910182" watchObservedRunningTime="2026-09-15 21:42:04.485474177 +0000 UTC m=+1892.347408034"
Sep 15 21:42:05.357834 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:05.357799    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:05.362863 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:05.362834    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:05.471849 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:05.471821    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:05.472762 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:05.472744    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:08.839875 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:08.839845    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"]
Sep 15 21:42:08.843490 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:08.843473    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"
Sep 15 21:42:08.845586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:08.845563    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6d9d39\""
Sep 15 21:42:08.845707 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:08.845564    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6d9d39-dockercfg-fjrkv\""
Sep 15 21:42:08.854441 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:08.854412    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"]
Sep 15 21:42:08.859490 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:08.859465    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"]
Sep 15 21:42:08.863690 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:08.863672    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"
Sep 15 21:42:08.869637 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:08.869575    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"]
Sep 15 21:42:08.949906 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:08.949862    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6t8j\" (UniqueName: \"kubernetes.io/projected/a5c7a87d-8597-43c8-a33d-eda0f5ffd474-kube-api-access-c6t8j\") pod \"payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn\" (UID: \"a5c7a87d-8597-43c8-a33d-eda0f5ffd474\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"
Sep 15 21:42:08.950238 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:08.950217    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq\" (UID: \"6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"
Sep 15 21:42:08.950373 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:08.950359    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5c7a87d-8597-43c8-a33d-eda0f5ffd474-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn\" (UID: \"a5c7a87d-8597-43c8-a33d-eda0f5ffd474\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"
Sep 15 21:42:08.950613 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:08.950593    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmrh4\" (UniqueName: \"kubernetes.io/projected/6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe-kube-api-access-zmrh4\") pod \"payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq\" (UID: \"6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"
Sep 15 21:42:09.052203 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.052169    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmrh4\" (UniqueName: \"kubernetes.io/projected/6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe-kube-api-access-zmrh4\") pod \"payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq\" (UID: \"6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"
Sep 15 21:42:09.052203 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.052211    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6t8j\" (UniqueName: \"kubernetes.io/projected/a5c7a87d-8597-43c8-a33d-eda0f5ffd474-kube-api-access-c6t8j\") pod \"payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn\" (UID: \"a5c7a87d-8597-43c8-a33d-eda0f5ffd474\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"
Sep 15 21:42:09.052473 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.052274    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq\" (UID: \"6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"
Sep 15 21:42:09.052473 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.052296    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5c7a87d-8597-43c8-a33d-eda0f5ffd474-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn\" (UID: \"a5c7a87d-8597-43c8-a33d-eda0f5ffd474\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"
Sep 15 21:42:09.053239 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.053210    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5c7a87d-8597-43c8-a33d-eda0f5ffd474-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn\" (UID: \"a5c7a87d-8597-43c8-a33d-eda0f5ffd474\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"
Sep 15 21:42:09.053355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.053210    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq\" (UID: \"6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"
Sep 15 21:42:09.061971 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.061944    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmrh4\" (UniqueName: \"kubernetes.io/projected/6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe-kube-api-access-zmrh4\") pod \"payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq\" (UID: \"6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"
Sep 15 21:42:09.062107 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.061944    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6t8j\" (UniqueName: \"kubernetes.io/projected/a5c7a87d-8597-43c8-a33d-eda0f5ffd474-kube-api-access-c6t8j\") pod \"payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn\" (UID: \"a5c7a87d-8597-43c8-a33d-eda0f5ffd474\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"
Sep 15 21:42:09.156452 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.156368    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"
Sep 15 21:42:09.178048 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.177991    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"
Sep 15 21:42:09.432015 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.431991    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"]
Sep 15 21:42:09.433714 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:42:09.433689    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c7a87d_8597_43c8_a33d_eda0f5ffd474.slice/crio-2ac17efe3831f08d754d5abfe88c6487eab08c5f93e86f0428ad8b6deb89e400 WatchSource:0}: Error finding container 2ac17efe3831f08d754d5abfe88c6487eab08c5f93e86f0428ad8b6deb89e400: Status 404 returned error can't find the container with id 2ac17efe3831f08d754d5abfe88c6487eab08c5f93e86f0428ad8b6deb89e400
Sep 15 21:42:09.459626 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.459602    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"]
Sep 15 21:42:09.464464 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:42:09.464441    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0bb3e7_77e9_415f_82e2_4c6d7def9bfe.slice/crio-1879b6fe40a2d18e19096c980b020b87bdebd64d2e8991c2e268e7e151afe6de WatchSource:0}: Error finding container 1879b6fe40a2d18e19096c980b020b87bdebd64d2e8991c2e268e7e151afe6de: Status 404 returned error can't find the container with id 1879b6fe40a2d18e19096c980b020b87bdebd64d2e8991c2e268e7e151afe6de
Sep 15 21:42:09.498663 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.498639    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq" event={"ID":"6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe","Type":"ContainerStarted","Data":"1879b6fe40a2d18e19096c980b020b87bdebd64d2e8991c2e268e7e151afe6de"}
Sep 15 21:42:09.500269 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.500236    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn" event={"ID":"a5c7a87d-8597-43c8-a33d-eda0f5ffd474","Type":"ContainerStarted","Data":"8a0cae5ef4770a230d8a8c5f6012f599071e9248366e994c42c4ea19f32471da"}
Sep 15 21:42:09.500418 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.500290    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn" event={"ID":"a5c7a87d-8597-43c8-a33d-eda0f5ffd474","Type":"ContainerStarted","Data":"2ac17efe3831f08d754d5abfe88c6487eab08c5f93e86f0428ad8b6deb89e400"}
Sep 15 21:42:09.500418 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.500312    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"
Sep 15 21:42:09.515945 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:09.515899    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn" podStartSLOduration=1.5158816800000001 podStartE2EDuration="1.51588168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:42:09.515013732 +0000 UTC m=+1897.376947536" watchObservedRunningTime="2026-09-15 21:42:09.51588168 +0000 UTC m=+1897.377815497"
Sep 15 21:42:10.507809 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:10.507775    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq" event={"ID":"6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe","Type":"ContainerStarted","Data":"17c2cfa2c61c947e1d5421d724e7427cebe4087000b588d1ac2b768686c9ef09"}
Sep 15 21:42:10.508293 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:10.508005    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"
Sep 15 21:42:10.526204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:10.526154    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq" podStartSLOduration=2.526136879 podStartE2EDuration="2.526136879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:42:10.52305825 +0000 UTC m=+1898.384992053" watchObservedRunningTime="2026-09-15 21:42:10.526136879 +0000 UTC m=+1898.388070688"
Sep 15 21:42:17.309955 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.309915    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87" podUID="6b482bd0-d481-4d69-8626-cd64fb43580f" containerName="main" containerID="cri-o://b76e8973f5418db5126819872fdab57158cf86f304109e0894056e3c1563e478" gracePeriod=45
Sep 15 21:42:17.564452 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.561869    2584 generic.go:356] "Generic (PLEG): container finished" podID="6b482bd0-d481-4d69-8626-cd64fb43580f" containerID="b76e8973f5418db5126819872fdab57158cf86f304109e0894056e3c1563e478" exitCode=0
Sep 15 21:42:17.564452 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.562016    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87" event={"ID":"6b482bd0-d481-4d69-8626-cd64fb43580f","Type":"ContainerDied","Data":"b76e8973f5418db5126819872fdab57158cf86f304109e0894056e3c1563e478"}
Sep 15 21:42:17.609928 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.609809    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:42:17.632281 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.632252    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b482bd0-d481-4d69-8626-cd64fb43580f-tls-certs\") pod \"6b482bd0-d481-4d69-8626-cd64fb43580f\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") "
Sep 15 21:42:17.632397 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.632309    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-home\") pod \"6b482bd0-d481-4d69-8626-cd64fb43580f\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") "
Sep 15 21:42:17.632397 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.632367    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-model-cache\") pod \"6b482bd0-d481-4d69-8626-cd64fb43580f\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") "
Sep 15 21:42:17.632491 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.632458    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-dshm\") pod \"6b482bd0-d481-4d69-8626-cd64fb43580f\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") "
Sep 15 21:42:17.632529 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.632513    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5zm5\" (UniqueName: \"kubernetes.io/projected/6b482bd0-d481-4d69-8626-cd64fb43580f-kube-api-access-r5zm5\") pod \"6b482bd0-d481-4d69-8626-cd64fb43580f\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") "
Sep 15 21:42:17.632656 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.632642    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-tmp-dir\") pod \"6b482bd0-d481-4d69-8626-cd64fb43580f\" (UID: \"6b482bd0-d481-4d69-8626-cd64fb43580f\") "
Sep 15 21:42:17.632888 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.632869    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b482bd0-d481-4d69-8626-cd64fb43580f" (UID: "6b482bd0-d481-4d69-8626-cd64fb43580f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:42:17.633339 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.633194    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-model-cache\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:42:17.633575 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.633454    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-home" (OuterVolumeSpecName: "home") pod "6b482bd0-d481-4d69-8626-cd64fb43580f" (UID: "6b482bd0-d481-4d69-8626-cd64fb43580f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:42:17.633681 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.633659    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b482bd0-d481-4d69-8626-cd64fb43580f" (UID: "6b482bd0-d481-4d69-8626-cd64fb43580f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:42:17.635570 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.635545    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b482bd0-d481-4d69-8626-cd64fb43580f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b482bd0-d481-4d69-8626-cd64fb43580f" (UID: "6b482bd0-d481-4d69-8626-cd64fb43580f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:42:17.635714 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.635695    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b482bd0-d481-4d69-8626-cd64fb43580f-kube-api-access-r5zm5" (OuterVolumeSpecName: "kube-api-access-r5zm5") pod "6b482bd0-d481-4d69-8626-cd64fb43580f" (UID: "6b482bd0-d481-4d69-8626-cd64fb43580f"). InnerVolumeSpecName "kube-api-access-r5zm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:42:17.635824 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.635807    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-dshm" (OuterVolumeSpecName: "dshm") pod "6b482bd0-d481-4d69-8626-cd64fb43580f" (UID: "6b482bd0-d481-4d69-8626-cd64fb43580f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:42:17.733887 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.733853    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-tmp-dir\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:42:17.733887 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.733881    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b482bd0-d481-4d69-8626-cd64fb43580f-tls-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:42:17.733887 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.733890    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-home\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:42:17.733887 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.733897    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b482bd0-d481-4d69-8626-cd64fb43580f-dshm\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:42:17.734145 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:17.733905    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5zm5\" (UniqueName: \"kubernetes.io/projected/6b482bd0-d481-4d69-8626-cd64fb43580f-kube-api-access-r5zm5\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:42:18.573017 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:18.572971    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87" event={"ID":"6b482bd0-d481-4d69-8626-cd64fb43580f","Type":"ContainerDied","Data":"3e91e911c1763a997e341e3af8f611262ad497c826a88ffeef327688af2a8702"}
Sep 15 21:42:18.573533 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:18.573041    2584 scope.go:117] "RemoveContainer" containerID="b76e8973f5418db5126819872fdab57158cf86f304109e0894056e3c1563e478"
Sep 15 21:42:18.573533 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:18.573217    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"
Sep 15 21:42:18.596910 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:18.596878    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"]
Sep 15 21:42:18.602880 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:18.602859    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-745dff/rate-test-model-745dff-kserve-8496f87755-28l87"]
Sep 15 21:42:18.677521 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:18.677479    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b482bd0-d481-4d69-8626-cd64fb43580f" path="/var/lib/kubelet/pods/6b482bd0-d481-4d69-8626-cd64fb43580f/volumes"
Sep 15 21:42:20.510172 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:20.510139    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"
Sep 15 21:42:21.512522 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:21.512487    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 21:42:21.512885 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:21.512560    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt" podUID="192e4d7e-dfc0-46bb-a356-4b5f479113a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 21:42:21.514897 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:21.514876    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"
Sep 15 21:42:22.597696 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:22.597669    2584 generic.go:356] "Generic (PLEG): container finished" podID="192e4d7e-dfc0-46bb-a356-4b5f479113a4" containerID="95f851b92ccad588b784ffe6b0e3351b786e27b89b2b874bc94c0fd55002d8eb" exitCode=0
Sep 15 21:42:22.598103 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:22.597745    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt" event={"ID":"192e4d7e-dfc0-46bb-a356-4b5f479113a4","Type":"ContainerDied","Data":"95f851b92ccad588b784ffe6b0e3351b786e27b89b2b874bc94c0fd55002d8eb"}
Sep 15 21:42:22.598151 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:22.598127    2584 scope.go:117] "RemoveContainer" containerID="95f851b92ccad588b784ffe6b0e3351b786e27b89b2b874bc94c0fd55002d8eb"
Sep 15 21:42:22.599167 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:22.599138    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:42:22.599260 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:22.599200    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:42:22.599260 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:22.599233    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:42:23.604907 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:23.604868    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt" event={"ID":"192e4d7e-dfc0-46bb-a356-4b5f479113a4","Type":"ContainerStarted","Data":"e6020d82905a0233e77999cfeb17d1fc2dea779d0798fc4a13c2672b52d4735a"}
Sep 15 21:42:24.358084 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:24.358038    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:24.358287 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:24.358153    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Sep 15 21:42:24.358287 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:24.358196    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt" podUID="192e4d7e-dfc0-46bb-a356-4b5f479113a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Sep 15 21:42:25.358289 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:25.358253    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Sep 15 21:42:25.358888 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:25.358856    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt" podUID="192e4d7e-dfc0-46bb-a356-4b5f479113a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Sep 15 21:42:26.358647 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:26.358611    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Sep 15 21:42:26.359199 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:26.359165    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt" podUID="192e4d7e-dfc0-46bb-a356-4b5f479113a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Sep 15 21:42:27.358489 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:27.358448    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Sep 15 21:42:27.358698 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:27.358525    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt" podUID="192e4d7e-dfc0-46bb-a356-4b5f479113a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Sep 15 21:42:28.358638 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:28.358592    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Sep 15 21:42:28.358829 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:28.358667    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt" podUID="192e4d7e-dfc0-46bb-a356-4b5f479113a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Sep 15 21:42:29.361115 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:29.361086    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:29.361497 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:29.361366    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:29.361973 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:29.361954    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:42:37.339012 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.338982    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"]
Sep 15 21:42:37.339476 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.339462    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b482bd0-d481-4d69-8626-cd64fb43580f" containerName="main"
Sep 15 21:42:37.339531 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.339480    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b482bd0-d481-4d69-8626-cd64fb43580f" containerName="main"
Sep 15 21:42:37.339590 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.339580    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b482bd0-d481-4d69-8626-cd64fb43580f" containerName="main"
Sep 15 21:42:37.341740 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.341722    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.344555 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:42:37.344524    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" is forbidden: User \"system:node:ip-10-0-139-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-184.ec2.internal' and this object" podUID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.344555 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:42:37.344534    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-ipp-b-0da705-openshift-default-dockercfg-rml2b\" is forbidden: User \"system:node:ip-10-0-139-184.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-184.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0da705-openshift-default-dockercfg-rml2b\"" type="*v1.Secret"
Sep 15 21:42:37.364337 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.364315    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"]
Sep 15 21:42:37.426514 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.426477    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-podinfo\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.426705 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.426527    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-credential-socket\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.426705 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.426578    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-data\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.426705 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.426608    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-workload-socket\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.426705 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.426673    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shprz\" (UniqueName: \"kubernetes.io/projected/59ce2e8f-0198-4399-b52f-a965a5b2c12d-kube-api-access-shprz\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.426705 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.426702    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-envoy\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.427017 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.426733    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-token\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.427017 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.426776    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-workload-certs\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.427017 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.426824    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istiod-ca-cert\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.527720 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.527684    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shprz\" (UniqueName: \"kubernetes.io/projected/59ce2e8f-0198-4399-b52f-a965a5b2c12d-kube-api-access-shprz\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.527908 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.527738    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-envoy\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.527908 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.527773    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-token\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.527908 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.527810    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-workload-certs\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.527908 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.527838    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istiod-ca-cert\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.528153 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.527918    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-podinfo\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.528153 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.527952    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-credential-socket\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.528153 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.528012    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-data\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.528153 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.528048    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-workload-socket\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.528363 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.528228    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-workload-certs\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.528455 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.528404    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-credential-socket\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.528536 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.528515    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-workload-socket\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.528598 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.528546    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-data\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.528654 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.528595    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istiod-ca-cert\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.530049 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.530031    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-envoy\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.530598 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.530581    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-podinfo\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.535857 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.535833    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-token\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:37.536217 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:37.536195    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shprz\" (UniqueName: \"kubernetes.io/projected/59ce2e8f-0198-4399-b52f-a965a5b2c12d-kube-api-access-shprz\") pod \"e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") " pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:38.504110 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:38.504086    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0da705-openshift-default-dockercfg-rml2b\""
Sep 15 21:42:38.504891 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:38.504875    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:38.781140 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:38.781112    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"]
Sep 15 21:42:38.783074 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:42:38.783047    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ce2e8f_0198_4399_b52f_a965a5b2c12d.slice/crio-b970b808b14d1ae8d58e4ba46e2a9fd7cd8d777d64ff0aa50b0f8e82b8b9e7c0 WatchSource:0}: Error finding container b970b808b14d1ae8d58e4ba46e2a9fd7cd8d777d64ff0aa50b0f8e82b8b9e7c0: Status 404 returned error can't find the container with id b970b808b14d1ae8d58e4ba46e2a9fd7cd8d777d64ff0aa50b0f8e82b8b9e7c0
Sep 15 21:42:38.785735 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:38.785697    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:42:38.785821 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:38.785784    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:42:38.785865 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:38.785831    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:42:39.701478 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:39.701444    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x" event={"ID":"59ce2e8f-0198-4399-b52f-a965a5b2c12d","Type":"ContainerStarted","Data":"4bf743b051c27a7148d9a609aa4aa06734bb2fba33bb8ddf13bfcb0fdfc39983"}
Sep 15 21:42:39.701840 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:39.701501    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x" event={"ID":"59ce2e8f-0198-4399-b52f-a965a5b2c12d","Type":"ContainerStarted","Data":"b970b808b14d1ae8d58e4ba46e2a9fd7cd8d777d64ff0aa50b0f8e82b8b9e7c0"}
Sep 15 21:42:39.718644 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:39.718595    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x" podStartSLOduration=2.7185797860000003 podStartE2EDuration="2.718579786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:42:39.717250062 +0000 UTC m=+1927.579183865" watchObservedRunningTime="2026-09-15 21:42:39.718579786 +0000 UTC m=+1927.580513590"
Sep 15 21:42:40.505432 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:40.505399    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:40.510502 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:40.510482    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:40.707604 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:40.707577    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:40.708652 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:40.708634    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:42:56.749536 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:56.749500    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 21:42:56.749955 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:56.749571    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x" podUID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 21:42:58.820991 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:58.820959    2584 generic.go:356] "Generic (PLEG): container finished" podID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" containerID="4bf743b051c27a7148d9a609aa4aa06734bb2fba33bb8ddf13bfcb0fdfc39983" exitCode=0
Sep 15 21:42:58.820991 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:58.820995    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x" event={"ID":"59ce2e8f-0198-4399-b52f-a965a5b2c12d","Type":"ContainerDied","Data":"4bf743b051c27a7148d9a609aa4aa06734bb2fba33bb8ddf13bfcb0fdfc39983"}
Sep 15 21:42:58.821447 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:58.821342    2584 scope.go:117] "RemoveContainer" containerID="4bf743b051c27a7148d9a609aa4aa06734bb2fba33bb8ddf13bfcb0fdfc39983"
Sep 15 21:42:58.822404 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:58.822358    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:42:58.822511 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:58.822465    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:42:58.822574 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:58.822521    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:42:59.828737 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:42:59.828696    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x" event={"ID":"59ce2e8f-0198-4399-b52f-a965a5b2c12d","Type":"ContainerStarted","Data":"6dc3f7db7ba423ffb63176aaf0fc9113fe8805bb1dccd978f594440d2b311ecf"}
Sep 15 21:43:00.506132 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:00.506092    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:43:00.506439 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:00.506406    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Sep 15 21:43:00.506566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:00.506470    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x" podUID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Sep 15 21:43:01.505558 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:01.505522    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Sep 15 21:43:01.505932 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:01.505575    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x" podUID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Sep 15 21:43:02.505598 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:02.505563    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Sep 15 21:43:02.505972 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:02.505615    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x" podUID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Sep 15 21:43:03.506117 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:03.506081    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Sep 15 21:43:03.506522 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:03.506141    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x" podUID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Sep 15 21:43:04.505361 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:04.505324    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Sep 15 21:43:04.505540 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:04.505434    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x" podUID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Sep 15 21:43:05.508767 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:05.508738    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:43:05.509239 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:05.509075    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:43:05.509796 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:05.509774    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:43:33.134814 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:33.134739    2584 scope.go:117] "RemoveContainer" containerID="91e7316b32110cf4e89fe011a79fdfc586c8ecf4560670097580a8de8ccb0ee4"
Sep 15 21:43:33.144214 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:33.144196    2584 scope.go:117] "RemoveContainer" containerID="65d15d45e3def5c795d134859addffaaf53708e05d044ea5f05ccd57ccd1a3b7"
Sep 15 21:43:33.153143 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:33.153121    2584 scope.go:117] "RemoveContainer" containerID="5ba0d9d833650b99987b38823dca6b4182d39eb287665c21939a5fe56f2c8b95"
Sep 15 21:43:37.391037 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:37.390982    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq_6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe/payload-processing/0.log"
Sep 15 21:43:38.203513 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.203479    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"]
Sep 15 21:43:38.207353 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.207332    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.209791 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.209770    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-a36624-openshift-default-dockercfg-zvbmj\""
Sep 15 21:43:38.219701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.219676    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"]
Sep 15 21:43:38.325473 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.325437    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-token\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.325473 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.325474    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-podinfo\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.325740 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.325503    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct6zq\" (UniqueName: \"kubernetes.io/projected/36c0e3bf-baf4-47f2-8c7f-812c6459084f-kube-api-access-ct6zq\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.325740 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.325542    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-envoy\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.325740 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.325587    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-credential-socket\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.325740 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.325605    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-data\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.325740 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.325640    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.325740 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.325717    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-workload-certs\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.326060 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.325749    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-workload-socket\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.427204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.427167    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-token\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.427636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.427212    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-podinfo\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.427636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.427247    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct6zq\" (UniqueName: \"kubernetes.io/projected/36c0e3bf-baf4-47f2-8c7f-812c6459084f-kube-api-access-ct6zq\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.427636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.427278    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-envoy\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.427636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.427317    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-credential-socket\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.427636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.427452    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-data\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.427636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.427526    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.427636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.427566    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-workload-certs\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.427636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.427591    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-workload-socket\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.428079 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.427817    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-credential-socket\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.428079 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.427933    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-data\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.428191 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.428139    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.428191 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.428179    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-workload-certs\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.428268 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.428197    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-workload-socket\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.430112 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.430093    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-envoy\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.430431 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.430411    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-podinfo\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.435016 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.434985    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct6zq\" (UniqueName: \"kubernetes.io/projected/36c0e3bf-baf4-47f2-8c7f-812c6459084f-kube-api-access-ct6zq\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.435120 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.435065    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-token\") pod \"e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") " pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.520294 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.520229    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:38.797080 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.797052    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"]
Sep 15 21:43:38.799965 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:43:38.799935    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c0e3bf_baf4_47f2_8c7f_812c6459084f.slice/crio-208f4ff8557bbed6729b7383d24fbbe10b7e26df8353e96542614f98c547ae07 WatchSource:0}: Error finding container 208f4ff8557bbed6729b7383d24fbbe10b7e26df8353e96542614f98c547ae07: Status 404 returned error can't find the container with id 208f4ff8557bbed6729b7383d24fbbe10b7e26df8353e96542614f98c547ae07
Sep 15 21:43:38.802291 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.802256    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:43:38.802419 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.802334    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:43:38.802419 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:38.802374    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:43:39.051670 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:39.051574    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr" event={"ID":"36c0e3bf-baf4-47f2-8c7f-812c6459084f","Type":"ContainerStarted","Data":"3855b1813350f376da55f68eb4ed8d7c0cf63a4bf114af96b5970c72be1361ba"}
Sep 15 21:43:39.051670 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:39.051633    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr" event={"ID":"36c0e3bf-baf4-47f2-8c7f-812c6459084f","Type":"ContainerStarted","Data":"208f4ff8557bbed6729b7383d24fbbe10b7e26df8353e96542614f98c547ae07"}
Sep 15 21:43:39.071138 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:39.071092    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr" podStartSLOduration=1.071076034 podStartE2EDuration="1.071076034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:43:39.068869983 +0000 UTC m=+1986.930803792" watchObservedRunningTime="2026-09-15 21:43:39.071076034 +0000 UTC m=+1986.933009843"
Sep 15 21:43:39.521297 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:39.521226    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:39.526638 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:39.526613    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:40.057710 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:40.057675    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:40.058739 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:40.058715    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:43:44.675716 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:44.675680    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"]
Sep 15 21:43:44.679575 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:44.679557    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"
Sep 15 21:43:44.681699 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:44.681663    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-a36624\""
Sep 15 21:43:44.681841 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:44.681815    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-a36624-dockercfg-chtd9\""
Sep 15 21:43:44.688496 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:44.688470    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"]
Sep 15 21:43:44.787857 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:44.787819    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a00d29c7-3c3d-4379-a661-e01c3041d048-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz\" (UID: \"a00d29c7-3c3d-4379-a661-e01c3041d048\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"
Sep 15 21:43:44.788047 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:44.788000    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltg4v\" (UniqueName: \"kubernetes.io/projected/a00d29c7-3c3d-4379-a661-e01c3041d048-kube-api-access-ltg4v\") pod \"payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz\" (UID: \"a00d29c7-3c3d-4379-a661-e01c3041d048\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"
Sep 15 21:43:44.889063 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:44.889029    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltg4v\" (UniqueName: \"kubernetes.io/projected/a00d29c7-3c3d-4379-a661-e01c3041d048-kube-api-access-ltg4v\") pod \"payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz\" (UID: \"a00d29c7-3c3d-4379-a661-e01c3041d048\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"
Sep 15 21:43:44.889202 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:44.889179    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a00d29c7-3c3d-4379-a661-e01c3041d048-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz\" (UID: \"a00d29c7-3c3d-4379-a661-e01c3041d048\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"
Sep 15 21:43:44.889863 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:44.889845    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a00d29c7-3c3d-4379-a661-e01c3041d048-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz\" (UID: \"a00d29c7-3c3d-4379-a661-e01c3041d048\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"
Sep 15 21:43:44.897728 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:44.897702    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltg4v\" (UniqueName: \"kubernetes.io/projected/a00d29c7-3c3d-4379-a661-e01c3041d048-kube-api-access-ltg4v\") pod \"payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz\" (UID: \"a00d29c7-3c3d-4379-a661-e01c3041d048\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"
Sep 15 21:43:44.992839 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:44.992772    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"
Sep 15 21:43:45.264057 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:45.263958    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"]
Sep 15 21:43:45.267396 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:43:45.266539    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00d29c7_3c3d_4379_a661_e01c3041d048.slice/crio-ffe7d57394b3f4d0d589e8b7503fc1b31e5940fe442cb8859a0511912b0ab595 WatchSource:0}: Error finding container ffe7d57394b3f4d0d589e8b7503fc1b31e5940fe442cb8859a0511912b0ab595: Status 404 returned error can't find the container with id ffe7d57394b3f4d0d589e8b7503fc1b31e5940fe442cb8859a0511912b0ab595
Sep 15 21:43:46.099199 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:46.099166    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz" event={"ID":"a00d29c7-3c3d-4379-a661-e01c3041d048","Type":"ContainerStarted","Data":"d6c1fcff609c17f76227d2c3f34ea275ea531484dfd47fa167ba9749b9e12e26"}
Sep 15 21:43:46.099596 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:46.099246    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz" event={"ID":"a00d29c7-3c3d-4379-a661-e01c3041d048","Type":"ContainerStarted","Data":"ffe7d57394b3f4d0d589e8b7503fc1b31e5940fe442cb8859a0511912b0ab595"}
Sep 15 21:43:46.099596 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:46.099275    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"
Sep 15 21:43:46.128212 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:46.128167    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz" podStartSLOduration=2.128152028 podStartE2EDuration="2.128152028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:43:46.126276335 +0000 UTC m=+1993.988210141" watchObservedRunningTime="2026-09-15 21:43:46.128152028 +0000 UTC m=+1993.990085832"
Sep 15 21:43:57.106333 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:57.106297    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"
Sep 15 21:43:59.179987 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:59.179957    2584 generic.go:356] "Generic (PLEG): container finished" podID="36c0e3bf-baf4-47f2-8c7f-812c6459084f" containerID="3855b1813350f376da55f68eb4ed8d7c0cf63a4bf114af96b5970c72be1361ba" exitCode=0
Sep 15 21:43:59.180317 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:59.179991    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr" event={"ID":"36c0e3bf-baf4-47f2-8c7f-812c6459084f","Type":"ContainerDied","Data":"3855b1813350f376da55f68eb4ed8d7c0cf63a4bf114af96b5970c72be1361ba"}
Sep 15 21:43:59.180543 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:59.180528    2584 scope.go:117] "RemoveContainer" containerID="3855b1813350f376da55f68eb4ed8d7c0cf63a4bf114af96b5970c72be1361ba"
Sep 15 21:43:59.181672 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:59.181647    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:43:59.181740 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:59.181727    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:43:59.181781 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:43:59.181762    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 21:44:00.187348 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:00.187315    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr" event={"ID":"36c0e3bf-baf4-47f2-8c7f-812c6459084f","Type":"ContainerStarted","Data":"c88c3a65b90af7861c3ab721ff64ca5fddfb36a1402730579a1871e5919a2394"}
Sep 15 21:44:00.520406 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:00.520312    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:44:00.520591 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:00.520569    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Sep 15 21:44:00.520652 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:00.520622    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr" podUID="36c0e3bf-baf4-47f2-8c7f-812c6459084f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Sep 15 21:44:01.520567 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:01.520535    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Sep 15 21:44:01.520954 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:01.520586    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr" podUID="36c0e3bf-baf4-47f2-8c7f-812c6459084f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Sep 15 21:44:02.521480 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:02.521447    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Sep 15 21:44:02.521851 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:02.521497    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr" podUID="36c0e3bf-baf4-47f2-8c7f-812c6459084f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Sep 15 21:44:03.521369 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:03.521334    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Sep 15 21:44:03.521552 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:03.521415    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr" podUID="36c0e3bf-baf4-47f2-8c7f-812c6459084f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Sep 15 21:44:04.520858 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:04.520824    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Sep 15 21:44:04.521015 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:04.520880    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr" podUID="36c0e3bf-baf4-47f2-8c7f-812c6459084f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Sep 15 21:44:05.521029 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:05.520991    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Sep 15 21:44:05.521535 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:05.521052    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr" podUID="36c0e3bf-baf4-47f2-8c7f-812c6459084f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Sep 15 21:44:06.524639 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:06.524605    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:44:06.525006 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:06.524886    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:44:06.525748 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:06.525727    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:44:11.771910 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:11.771877    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb"]
Sep 15 21:44:11.775611 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:11.775595    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb"
Sep 15 21:44:11.777769 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:11.777746    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:44:11.777902 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:11.777831    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:44:11.781940 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:11.781919    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb"]
Sep 15 21:44:11.871059 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:11.871030    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13580655-7f0a-48fc-b767-1d4589b47b06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb\" (UID: \"13580655-7f0a-48fc-b767-1d4589b47b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb"
Sep 15 21:44:11.972015 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:11.971985    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13580655-7f0a-48fc-b767-1d4589b47b06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb\" (UID: \"13580655-7f0a-48fc-b767-1d4589b47b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb"
Sep 15 21:44:11.972615 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:11.972598    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13580655-7f0a-48fc-b767-1d4589b47b06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb\" (UID: \"13580655-7f0a-48fc-b767-1d4589b47b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb"
Sep 15 21:44:12.087906 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:12.087883    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb"
Sep 15 21:44:12.356568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:12.356546    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb"]
Sep 15 21:44:12.358362 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:44:12.358330    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13580655_7f0a_48fc_b767_1d4589b47b06.slice/crio-b1494cdb8b7da2297b7b5a7a1d4cd80df1e75a2dd5868191f1ff4456584311b3 WatchSource:0}: Error finding container b1494cdb8b7da2297b7b5a7a1d4cd80df1e75a2dd5868191f1ff4456584311b3: Status 404 returned error can't find the container with id b1494cdb8b7da2297b7b5a7a1d4cd80df1e75a2dd5868191f1ff4456584311b3
Sep 15 21:44:13.264154 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:13.264065    2584 generic.go:356] "Generic (PLEG): container finished" podID="13580655-7f0a-48fc-b767-1d4589b47b06" containerID="683053e5d2fc843b5f5a283c5d455610a6f31711e7224682a8fcaad6eaf9d9f9" exitCode=0
Sep 15 21:44:13.264154 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:13.264143    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb" event={"ID":"13580655-7f0a-48fc-b767-1d4589b47b06","Type":"ContainerDied","Data":"683053e5d2fc843b5f5a283c5d455610a6f31711e7224682a8fcaad6eaf9d9f9"}
Sep 15 21:44:13.264658 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:13.264203    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb" event={"ID":"13580655-7f0a-48fc-b767-1d4589b47b06","Type":"ContainerStarted","Data":"b1494cdb8b7da2297b7b5a7a1d4cd80df1e75a2dd5868191f1ff4456584311b3"}
Sep 15 21:44:14.401325 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:14.401301    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb"
Sep 15 21:44:14.498885 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:14.498859    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13580655-7f0a-48fc-b767-1d4589b47b06-maas-api-service-ca\") pod \"13580655-7f0a-48fc-b767-1d4589b47b06\" (UID: \"13580655-7f0a-48fc-b767-1d4589b47b06\") "
Sep 15 21:44:14.499196 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:14.499170    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13580655-7f0a-48fc-b767-1d4589b47b06-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "13580655-7f0a-48fc-b767-1d4589b47b06" (UID: "13580655-7f0a-48fc-b767-1d4589b47b06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:44:14.600406 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:14.600358    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13580655-7f0a-48fc-b767-1d4589b47b06-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:15.276634 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:15.276593    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb" event={"ID":"13580655-7f0a-48fc-b767-1d4589b47b06","Type":"ContainerDied","Data":"b1494cdb8b7da2297b7b5a7a1d4cd80df1e75a2dd5868191f1ff4456584311b3"}
Sep 15 21:44:15.276802 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:15.276635    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb"
Sep 15 21:44:15.276802 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:15.276675    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1494cdb8b7da2297b7b5a7a1d4cd80df1e75a2dd5868191f1ff4456584311b3"
Sep 15 21:44:22.678687 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:22.678323    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"]
Sep 15 21:44:22.678687 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:22.678597    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz" podUID="a00d29c7-3c3d-4379-a661-e01c3041d048" containerName="payload-processing" containerID="cri-o://d6c1fcff609c17f76227d2c3f34ea275ea531484dfd47fa167ba9749b9e12e26" gracePeriod=30
Sep 15 21:44:22.834452 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:22.834365    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb"]
Sep 15 21:44:22.837003 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:22.836973    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a36624-7g9bb"]
Sep 15 21:44:22.938259 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:22.938205    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"
Sep 15 21:44:23.075239 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:23.075207    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a00d29c7-3c3d-4379-a661-e01c3041d048-plugins-config-volume\") pod \"a00d29c7-3c3d-4379-a661-e01c3041d048\" (UID: \"a00d29c7-3c3d-4379-a661-e01c3041d048\") "
Sep 15 21:44:23.075445 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:23.075285    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltg4v\" (UniqueName: \"kubernetes.io/projected/a00d29c7-3c3d-4379-a661-e01c3041d048-kube-api-access-ltg4v\") pod \"a00d29c7-3c3d-4379-a661-e01c3041d048\" (UID: \"a00d29c7-3c3d-4379-a661-e01c3041d048\") "
Sep 15 21:44:23.075601 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:23.075577    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00d29c7-3c3d-4379-a661-e01c3041d048-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a00d29c7-3c3d-4379-a661-e01c3041d048" (UID: "a00d29c7-3c3d-4379-a661-e01c3041d048"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:44:23.075675 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:23.075654    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a00d29c7-3c3d-4379-a661-e01c3041d048-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:23.077509 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:23.077487    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00d29c7-3c3d-4379-a661-e01c3041d048-kube-api-access-ltg4v" (OuterVolumeSpecName: "kube-api-access-ltg4v") pod "a00d29c7-3c3d-4379-a661-e01c3041d048" (UID: "a00d29c7-3c3d-4379-a661-e01c3041d048"). InnerVolumeSpecName "kube-api-access-ltg4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:23.176949 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:23.176927    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltg4v\" (UniqueName: \"kubernetes.io/projected/a00d29c7-3c3d-4379-a661-e01c3041d048-kube-api-access-ltg4v\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:23.326254 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:23.326224    2584 generic.go:356] "Generic (PLEG): container finished" podID="a00d29c7-3c3d-4379-a661-e01c3041d048" containerID="d6c1fcff609c17f76227d2c3f34ea275ea531484dfd47fa167ba9749b9e12e26" exitCode=0
Sep 15 21:44:23.326467 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:23.326275    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"
Sep 15 21:44:23.326467 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:23.326297    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz" event={"ID":"a00d29c7-3c3d-4379-a661-e01c3041d048","Type":"ContainerDied","Data":"d6c1fcff609c17f76227d2c3f34ea275ea531484dfd47fa167ba9749b9e12e26"}
Sep 15 21:44:23.326467 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:23.326348    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz" event={"ID":"a00d29c7-3c3d-4379-a661-e01c3041d048","Type":"ContainerDied","Data":"ffe7d57394b3f4d0d589e8b7503fc1b31e5940fe442cb8859a0511912b0ab595"}
Sep 15 21:44:23.326467 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:23.326370    2584 scope.go:117] "RemoveContainer" containerID="d6c1fcff609c17f76227d2c3f34ea275ea531484dfd47fa167ba9749b9e12e26"
Sep 15 21:44:23.335396 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:23.335358    2584 scope.go:117] "RemoveContainer" containerID="d6c1fcff609c17f76227d2c3f34ea275ea531484dfd47fa167ba9749b9e12e26"
Sep 15 21:44:23.335645 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:44:23.335624    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c1fcff609c17f76227d2c3f34ea275ea531484dfd47fa167ba9749b9e12e26\": container with ID starting with d6c1fcff609c17f76227d2c3f34ea275ea531484dfd47fa167ba9749b9e12e26 not found: ID does not exist" containerID="d6c1fcff609c17f76227d2c3f34ea275ea531484dfd47fa167ba9749b9e12e26"
Sep 15 21:44:23.335704 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:23.335652    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c1fcff609c17f76227d2c3f34ea275ea531484dfd47fa167ba9749b9e12e26"} err="failed to get container status \"d6c1fcff609c17f76227d2c3f34ea275ea531484dfd47fa167ba9749b9e12e26\": rpc error: code = NotFound desc = could not find container \"d6c1fcff609c17f76227d2c3f34ea275ea531484dfd47fa167ba9749b9e12e26\": container with ID starting with d6c1fcff609c17f76227d2c3f34ea275ea531484dfd47fa167ba9749b9e12e26 not found: ID does not exist"
Sep 15 21:44:23.350026 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:23.350003    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"]
Sep 15 21:44:23.352290 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:23.352268    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a36624-66879cd68c-zsmcz"]
Sep 15 21:44:24.673840 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:24.673810    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13580655-7f0a-48fc-b767-1d4589b47b06" path="/var/lib/kubelet/pods/13580655-7f0a-48fc-b767-1d4589b47b06/volumes"
Sep 15 21:44:24.674197 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:24.674169    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a00d29c7-3c3d-4379-a661-e01c3041d048" path="/var/lib/kubelet/pods/a00d29c7-3c3d-4379-a661-e01c3041d048/volumes"
Sep 15 21:44:29.573143 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:29.573109    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"]
Sep 15 21:44:29.573624 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:29.573340    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr" podUID="36c0e3bf-baf4-47f2-8c7f-812c6459084f" containerName="istio-proxy" containerID="cri-o://c88c3a65b90af7861c3ab721ff64ca5fddfb36a1402730579a1871e5919a2394" gracePeriod=30
Sep 15 21:44:33.184079 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.184048    2584 scope.go:117] "RemoveContainer" containerID="c75eff9916ff0a5b69e13e6bc9ec5616eb53d79dfeccd8e4dfa18c6814a80b35"
Sep 15 21:44:33.194850 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.194825    2584 scope.go:117] "RemoveContainer" containerID="ad9e411953681fcf872fa3157c42782ddb14d9936ee11c0b6290f4c72574b65e"
Sep 15 21:44:33.203247 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.203226    2584 scope.go:117] "RemoveContainer" containerID="796742c7b626bc144c68e2a7f3f2e0054507be6a61812f6929c78c6f72c87052"
Sep 15 21:44:33.213102 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.213083    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6"]
Sep 15 21:44:33.213553 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.213539    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a00d29c7-3c3d-4379-a661-e01c3041d048" containerName="payload-processing"
Sep 15 21:44:33.213553 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.213554    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a00d29c7-3c3d-4379-a661-e01c3041d048" containerName="payload-processing"
Sep 15 21:44:33.213675 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.213565    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13580655-7f0a-48fc-b767-1d4589b47b06" containerName="revoke-keys"
Sep 15 21:44:33.213675 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.213571    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="13580655-7f0a-48fc-b767-1d4589b47b06" containerName="revoke-keys"
Sep 15 21:44:33.213675 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.213634    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="13580655-7f0a-48fc-b767-1d4589b47b06" containerName="revoke-keys"
Sep 15 21:44:33.213675 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.213645    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a00d29c7-3c3d-4379-a661-e01c3041d048" containerName="payload-processing"
Sep 15 21:44:33.218492 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.218473    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6"
Sep 15 21:44:33.221005 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.220981    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:44:33.221253 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.221234    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:44:33.223764 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.223744    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6"]
Sep 15 21:44:33.384568 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.384540    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/442ee1bf-cbf1-4e22-99f1-0293cc3a6611-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6\" (UID: \"442ee1bf-cbf1-4e22-99f1-0293cc3a6611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6"
Sep 15 21:44:33.485315 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.485252    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/442ee1bf-cbf1-4e22-99f1-0293cc3a6611-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6\" (UID: \"442ee1bf-cbf1-4e22-99f1-0293cc3a6611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6"
Sep 15 21:44:33.485853 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.485830    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/442ee1bf-cbf1-4e22-99f1-0293cc3a6611-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6\" (UID: \"442ee1bf-cbf1-4e22-99f1-0293cc3a6611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6"
Sep 15 21:44:33.532068 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.532046    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6"
Sep 15 21:44:33.809076 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:33.809047    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6"]
Sep 15 21:44:33.810591 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:44:33.810562    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442ee1bf_cbf1_4e22_99f1_0293cc3a6611.slice/crio-93c1735a0a871e524813d3fba953555ece96c50c8ec9da3fc5cb34f50cc93260 WatchSource:0}: Error finding container 93c1735a0a871e524813d3fba953555ece96c50c8ec9da3fc5cb34f50cc93260: Status 404 returned error can't find the container with id 93c1735a0a871e524813d3fba953555ece96c50c8ec9da3fc5cb34f50cc93260
Sep 15 21:44:34.392162 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.392081    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6" event={"ID":"442ee1bf-cbf1-4e22-99f1-0293cc3a6611","Type":"ContainerStarted","Data":"80a0d47cbbf42d02d68873874c9b8309371e20ccf784afe40f7b3e3d5815528b"}
Sep 15 21:44:34.392162 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.392150    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6" event={"ID":"442ee1bf-cbf1-4e22-99f1-0293cc3a6611","Type":"ContainerStarted","Data":"93c1735a0a871e524813d3fba953555ece96c50c8ec9da3fc5cb34f50cc93260"}
Sep 15 21:44:34.408499 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.408453    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6" podStartSLOduration=1.408437757 podStartE2EDuration="1.408437757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:44:34.406062246 +0000 UTC m=+2042.267996052" watchObservedRunningTime="2026-09-15 21:44:34.408437757 +0000 UTC m=+2042.270371560"
Sep 15 21:44:34.823352 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.823329    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:44:34.898608 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.898580    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istiod-ca-cert\") pod \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") "
Sep 15 21:44:34.898757 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.898624    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-workload-certs\") pod \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") "
Sep 15 21:44:34.898757 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.898668    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct6zq\" (UniqueName: \"kubernetes.io/projected/36c0e3bf-baf4-47f2-8c7f-812c6459084f-kube-api-access-ct6zq\") pod \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") "
Sep 15 21:44:34.898757 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.898735    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-workload-socket\") pod \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") "
Sep 15 21:44:34.898937 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.898765    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-token\") pod \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") "
Sep 15 21:44:34.898937 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.898800    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-credential-socket\") pod \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") "
Sep 15 21:44:34.898937 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.898826    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-podinfo\") pod \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") "
Sep 15 21:44:34.898937 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.898920    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-data\") pod \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") "
Sep 15 21:44:34.899133 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.898950    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-envoy\") pod \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\" (UID: \"36c0e3bf-baf4-47f2-8c7f-812c6459084f\") "
Sep 15 21:44:34.899133 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.898949    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "36c0e3bf-baf4-47f2-8c7f-812c6459084f" (UID: "36c0e3bf-baf4-47f2-8c7f-812c6459084f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:44:34.899133 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.898974    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "36c0e3bf-baf4-47f2-8c7f-812c6459084f" (UID: "36c0e3bf-baf4-47f2-8c7f-812c6459084f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:34.899133 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.899058    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "36c0e3bf-baf4-47f2-8c7f-812c6459084f" (UID: "36c0e3bf-baf4-47f2-8c7f-812c6459084f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:34.899336 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.899185    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-data" (OuterVolumeSpecName: "istio-data") pod "36c0e3bf-baf4-47f2-8c7f-812c6459084f" (UID: "36c0e3bf-baf4-47f2-8c7f-812c6459084f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:34.899336 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.899243    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "36c0e3bf-baf4-47f2-8c7f-812c6459084f" (UID: "36c0e3bf-baf4-47f2-8c7f-812c6459084f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:34.899484 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.899409    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istiod-ca-cert\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:34.899484 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.899429    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-workload-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:34.899484 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.899441    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-workload-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:34.899484 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.899453    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-credential-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:34.899484 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.899461    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-data\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:34.901138 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.901104    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-token" (OuterVolumeSpecName: "istio-token") pod "36c0e3bf-baf4-47f2-8c7f-812c6459084f" (UID: "36c0e3bf-baf4-47f2-8c7f-812c6459084f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:34.901242 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.901119    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "36c0e3bf-baf4-47f2-8c7f-812c6459084f" (UID: "36c0e3bf-baf4-47f2-8c7f-812c6459084f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:34.901325 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.901308    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c0e3bf-baf4-47f2-8c7f-812c6459084f-kube-api-access-ct6zq" (OuterVolumeSpecName: "kube-api-access-ct6zq") pod "36c0e3bf-baf4-47f2-8c7f-812c6459084f" (UID: "36c0e3bf-baf4-47f2-8c7f-812c6459084f"). InnerVolumeSpecName "kube-api-access-ct6zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:34.901402 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:34.901304    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "36c0e3bf-baf4-47f2-8c7f-812c6459084f" (UID: "36c0e3bf-baf4-47f2-8c7f-812c6459084f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:44:35.000291 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.000234    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-token\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:35.000291 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.000256    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-podinfo\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:35.000291 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.000268    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36c0e3bf-baf4-47f2-8c7f-812c6459084f-istio-envoy\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:35.000291 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.000277    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct6zq\" (UniqueName: \"kubernetes.io/projected/36c0e3bf-baf4-47f2-8c7f-812c6459084f-kube-api-access-ct6zq\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:35.399601 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.399567    2584 generic.go:356] "Generic (PLEG): container finished" podID="36c0e3bf-baf4-47f2-8c7f-812c6459084f" containerID="c88c3a65b90af7861c3ab721ff64ca5fddfb36a1402730579a1871e5919a2394" exitCode=0
Sep 15 21:44:35.399990 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.399612    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"
Sep 15 21:44:35.399990 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.399656    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr" event={"ID":"36c0e3bf-baf4-47f2-8c7f-812c6459084f","Type":"ContainerDied","Data":"c88c3a65b90af7861c3ab721ff64ca5fddfb36a1402730579a1871e5919a2394"}
Sep 15 21:44:35.399990 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.399704    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr" event={"ID":"36c0e3bf-baf4-47f2-8c7f-812c6459084f","Type":"ContainerDied","Data":"208f4ff8557bbed6729b7383d24fbbe10b7e26df8353e96542614f98c547ae07"}
Sep 15 21:44:35.399990 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.399732    2584 scope.go:117] "RemoveContainer" containerID="c88c3a65b90af7861c3ab721ff64ca5fddfb36a1402730579a1871e5919a2394"
Sep 15 21:44:35.401291 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.401272    2584 generic.go:356] "Generic (PLEG): container finished" podID="442ee1bf-cbf1-4e22-99f1-0293cc3a6611" containerID="80a0d47cbbf42d02d68873874c9b8309371e20ccf784afe40f7b3e3d5815528b" exitCode=0
Sep 15 21:44:35.401428 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.401366    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6" event={"ID":"442ee1bf-cbf1-4e22-99f1-0293cc3a6611","Type":"ContainerDied","Data":"80a0d47cbbf42d02d68873874c9b8309371e20ccf784afe40f7b3e3d5815528b"}
Sep 15 21:44:35.411708 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.410251    2584 scope.go:117] "RemoveContainer" containerID="3855b1813350f376da55f68eb4ed8d7c0cf63a4bf114af96b5970c72be1361ba"
Sep 15 21:44:35.423303 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.423278    2584 scope.go:117] "RemoveContainer" containerID="c88c3a65b90af7861c3ab721ff64ca5fddfb36a1402730579a1871e5919a2394"
Sep 15 21:44:35.423654 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:44:35.423622    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c88c3a65b90af7861c3ab721ff64ca5fddfb36a1402730579a1871e5919a2394\": container with ID starting with c88c3a65b90af7861c3ab721ff64ca5fddfb36a1402730579a1871e5919a2394 not found: ID does not exist" containerID="c88c3a65b90af7861c3ab721ff64ca5fddfb36a1402730579a1871e5919a2394"
Sep 15 21:44:35.423762 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.423676    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c88c3a65b90af7861c3ab721ff64ca5fddfb36a1402730579a1871e5919a2394"} err="failed to get container status \"c88c3a65b90af7861c3ab721ff64ca5fddfb36a1402730579a1871e5919a2394\": rpc error: code = NotFound desc = could not find container \"c88c3a65b90af7861c3ab721ff64ca5fddfb36a1402730579a1871e5919a2394\": container with ID starting with c88c3a65b90af7861c3ab721ff64ca5fddfb36a1402730579a1871e5919a2394 not found: ID does not exist"
Sep 15 21:44:35.423762 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.423711    2584 scope.go:117] "RemoveContainer" containerID="3855b1813350f376da55f68eb4ed8d7c0cf63a4bf114af96b5970c72be1361ba"
Sep 15 21:44:35.424243 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:44:35.424220    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3855b1813350f376da55f68eb4ed8d7c0cf63a4bf114af96b5970c72be1361ba\": container with ID starting with 3855b1813350f376da55f68eb4ed8d7c0cf63a4bf114af96b5970c72be1361ba not found: ID does not exist" containerID="3855b1813350f376da55f68eb4ed8d7c0cf63a4bf114af96b5970c72be1361ba"
Sep 15 21:44:35.424338 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.424249    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3855b1813350f376da55f68eb4ed8d7c0cf63a4bf114af96b5970c72be1361ba"} err="failed to get container status \"3855b1813350f376da55f68eb4ed8d7c0cf63a4bf114af96b5970c72be1361ba\": rpc error: code = NotFound desc = could not find container \"3855b1813350f376da55f68eb4ed8d7c0cf63a4bf114af96b5970c72be1361ba\": container with ID starting with 3855b1813350f376da55f68eb4ed8d7c0cf63a4bf114af96b5970c72be1361ba not found: ID does not exist"
Sep 15 21:44:35.439207 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.439187    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"]
Sep 15 21:44:35.441355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:35.441334    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a36624-openshift-default-7d99486c84-6x7hr"]
Sep 15 21:44:36.538971 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:36.538950    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6"
Sep 15 21:44:36.617788 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:36.617758    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/442ee1bf-cbf1-4e22-99f1-0293cc3a6611-maas-api-service-ca\") pod \"442ee1bf-cbf1-4e22-99f1-0293cc3a6611\" (UID: \"442ee1bf-cbf1-4e22-99f1-0293cc3a6611\") "
Sep 15 21:44:36.618076 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:36.618051    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442ee1bf-cbf1-4e22-99f1-0293cc3a6611-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "442ee1bf-cbf1-4e22-99f1-0293cc3a6611" (UID: "442ee1bf-cbf1-4e22-99f1-0293cc3a6611"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:44:36.674286 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:36.674229    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36c0e3bf-baf4-47f2-8c7f-812c6459084f" path="/var/lib/kubelet/pods/36c0e3bf-baf4-47f2-8c7f-812c6459084f/volumes"
Sep 15 21:44:36.718861 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:36.718839    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/442ee1bf-cbf1-4e22-99f1-0293cc3a6611-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:37.416022 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:37.415988    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6"
Sep 15 21:44:37.416182 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:37.416022    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6" event={"ID":"442ee1bf-cbf1-4e22-99f1-0293cc3a6611","Type":"ContainerDied","Data":"93c1735a0a871e524813d3fba953555ece96c50c8ec9da3fc5cb34f50cc93260"}
Sep 15 21:44:37.416182 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:37.416070    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93c1735a0a871e524813d3fba953555ece96c50c8ec9da3fc5cb34f50cc93260"
Sep 15 21:44:48.808681 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:48.808638    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"]
Sep 15 21:44:48.809254 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:48.808923    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq" podUID="6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe" containerName="payload-processing" containerID="cri-o://17c2cfa2c61c947e1d5421d724e7427cebe4087000b588d1ac2b768686c9ef09" gracePeriod=30
Sep 15 21:44:48.816362 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:48.816324    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"]
Sep 15 21:44:48.816591 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:48.816564    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn" podUID="a5c7a87d-8597-43c8-a33d-eda0f5ffd474" containerName="payload-pre-processing" containerID="cri-o://8a0cae5ef4770a230d8a8c5f6012f599071e9248366e994c42c4ea19f32471da" gracePeriod=30
Sep 15 21:44:48.989787 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:48.989723    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6"]
Sep 15 21:44:48.992700 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:48.992640    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6d9d39-6qpn6"]
Sep 15 21:44:49.085672 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.085649    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"
Sep 15 21:44:49.113361 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.113336    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"
Sep 15 21:44:49.133896 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.133870    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmrh4\" (UniqueName: \"kubernetes.io/projected/6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe-kube-api-access-zmrh4\") pod \"6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe\" (UID: \"6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe\") "
Sep 15 21:44:49.134004 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.133988    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe-plugins-config-volume\") pod \"6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe\" (UID: \"6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe\") "
Sep 15 21:44:49.134350 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.134329    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe" (UID: "6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:44:49.135829 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.135801    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe-kube-api-access-zmrh4" (OuterVolumeSpecName: "kube-api-access-zmrh4") pod "6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe" (UID: "6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe"). InnerVolumeSpecName "kube-api-access-zmrh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:49.234866 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.234835    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5c7a87d-8597-43c8-a33d-eda0f5ffd474-plugins-config-volume\") pod \"a5c7a87d-8597-43c8-a33d-eda0f5ffd474\" (UID: \"a5c7a87d-8597-43c8-a33d-eda0f5ffd474\") "
Sep 15 21:44:49.235058 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.234930    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6t8j\" (UniqueName: \"kubernetes.io/projected/a5c7a87d-8597-43c8-a33d-eda0f5ffd474-kube-api-access-c6t8j\") pod \"a5c7a87d-8597-43c8-a33d-eda0f5ffd474\" (UID: \"a5c7a87d-8597-43c8-a33d-eda0f5ffd474\") "
Sep 15 21:44:49.235174 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.235151    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c7a87d-8597-43c8-a33d-eda0f5ffd474-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a5c7a87d-8597-43c8-a33d-eda0f5ffd474" (UID: "a5c7a87d-8597-43c8-a33d-eda0f5ffd474"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:44:49.235325 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.235309    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:49.235410 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.235332    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmrh4\" (UniqueName: \"kubernetes.io/projected/6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe-kube-api-access-zmrh4\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:49.235410 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.235347    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5c7a87d-8597-43c8-a33d-eda0f5ffd474-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:49.236975 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.236955    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c7a87d-8597-43c8-a33d-eda0f5ffd474-kube-api-access-c6t8j" (OuterVolumeSpecName: "kube-api-access-c6t8j") pod "a5c7a87d-8597-43c8-a33d-eda0f5ffd474" (UID: "a5c7a87d-8597-43c8-a33d-eda0f5ffd474"). InnerVolumeSpecName "kube-api-access-c6t8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:49.336047 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.335972    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6t8j\" (UniqueName: \"kubernetes.io/projected/a5c7a87d-8597-43c8-a33d-eda0f5ffd474-kube-api-access-c6t8j\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:44:49.491737 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.491704    2584 generic.go:356] "Generic (PLEG): container finished" podID="6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe" containerID="17c2cfa2c61c947e1d5421d724e7427cebe4087000b588d1ac2b768686c9ef09" exitCode=0
Sep 15 21:44:49.491893 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.491747    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"
Sep 15 21:44:49.491893 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.491787    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq" event={"ID":"6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe","Type":"ContainerDied","Data":"17c2cfa2c61c947e1d5421d724e7427cebe4087000b588d1ac2b768686c9ef09"}
Sep 15 21:44:49.491893 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.491839    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq" event={"ID":"6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe","Type":"ContainerDied","Data":"1879b6fe40a2d18e19096c980b020b87bdebd64d2e8991c2e268e7e151afe6de"}
Sep 15 21:44:49.491893 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.491869    2584 scope.go:117] "RemoveContainer" containerID="17c2cfa2c61c947e1d5421d724e7427cebe4087000b588d1ac2b768686c9ef09"
Sep 15 21:44:49.493185 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.493163    2584 generic.go:356] "Generic (PLEG): container finished" podID="a5c7a87d-8597-43c8-a33d-eda0f5ffd474" containerID="8a0cae5ef4770a230d8a8c5f6012f599071e9248366e994c42c4ea19f32471da" exitCode=0
Sep 15 21:44:49.493291 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.493202    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"
Sep 15 21:44:49.493291 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.493224    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn" event={"ID":"a5c7a87d-8597-43c8-a33d-eda0f5ffd474","Type":"ContainerDied","Data":"8a0cae5ef4770a230d8a8c5f6012f599071e9248366e994c42c4ea19f32471da"}
Sep 15 21:44:49.493291 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.493258    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn" event={"ID":"a5c7a87d-8597-43c8-a33d-eda0f5ffd474","Type":"ContainerDied","Data":"2ac17efe3831f08d754d5abfe88c6487eab08c5f93e86f0428ad8b6deb89e400"}
Sep 15 21:44:49.501829 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.501704    2584 scope.go:117] "RemoveContainer" containerID="17c2cfa2c61c947e1d5421d724e7427cebe4087000b588d1ac2b768686c9ef09"
Sep 15 21:44:49.501981 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:44:49.501950    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17c2cfa2c61c947e1d5421d724e7427cebe4087000b588d1ac2b768686c9ef09\": container with ID starting with 17c2cfa2c61c947e1d5421d724e7427cebe4087000b588d1ac2b768686c9ef09 not found: ID does not exist" containerID="17c2cfa2c61c947e1d5421d724e7427cebe4087000b588d1ac2b768686c9ef09"
Sep 15 21:44:49.502093 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.501981    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17c2cfa2c61c947e1d5421d724e7427cebe4087000b588d1ac2b768686c9ef09"} err="failed to get container status \"17c2cfa2c61c947e1d5421d724e7427cebe4087000b588d1ac2b768686c9ef09\": rpc error: code = NotFound desc = could not find container \"17c2cfa2c61c947e1d5421d724e7427cebe4087000b588d1ac2b768686c9ef09\": container with ID starting with 17c2cfa2c61c947e1d5421d724e7427cebe4087000b588d1ac2b768686c9ef09 not found: ID does not exist"
Sep 15 21:44:49.502093 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.502005    2584 scope.go:117] "RemoveContainer" containerID="8a0cae5ef4770a230d8a8c5f6012f599071e9248366e994c42c4ea19f32471da"
Sep 15 21:44:49.509771 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.509756    2584 scope.go:117] "RemoveContainer" containerID="8a0cae5ef4770a230d8a8c5f6012f599071e9248366e994c42c4ea19f32471da"
Sep 15 21:44:49.510005 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:44:49.509988    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a0cae5ef4770a230d8a8c5f6012f599071e9248366e994c42c4ea19f32471da\": container with ID starting with 8a0cae5ef4770a230d8a8c5f6012f599071e9248366e994c42c4ea19f32471da not found: ID does not exist" containerID="8a0cae5ef4770a230d8a8c5f6012f599071e9248366e994c42c4ea19f32471da"
Sep 15 21:44:49.510050 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.510012    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a0cae5ef4770a230d8a8c5f6012f599071e9248366e994c42c4ea19f32471da"} err="failed to get container status \"8a0cae5ef4770a230d8a8c5f6012f599071e9248366e994c42c4ea19f32471da\": rpc error: code = NotFound desc = could not find container \"8a0cae5ef4770a230d8a8c5f6012f599071e9248366e994c42c4ea19f32471da\": container with ID starting with 8a0cae5ef4770a230d8a8c5f6012f599071e9248366e994c42c4ea19f32471da not found: ID does not exist"
Sep 15 21:44:49.517598 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.517576    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"]
Sep 15 21:44:49.521732 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.521714    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6d9d39-68645c855f-stgnq"]
Sep 15 21:44:49.531388 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.531356    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"]
Sep 15 21:44:49.535634 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:49.535615    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6d9d39-6774c9d7d-nh7cn"]
Sep 15 21:44:50.674368 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:50.674332    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="442ee1bf-cbf1-4e22-99f1-0293cc3a6611" path="/var/lib/kubelet/pods/442ee1bf-cbf1-4e22-99f1-0293cc3a6611/volumes"
Sep 15 21:44:50.675466 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:50.675443    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe" path="/var/lib/kubelet/pods/6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe/volumes"
Sep 15 21:44:50.676136 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:44:50.676114    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5c7a87d-8597-43c8-a33d-eda0f5ffd474" path="/var/lib/kubelet/pods/a5c7a87d-8597-43c8-a33d-eda0f5ffd474/volumes"
Sep 15 21:45:00.137655 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.137582    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-745dff-29825145-gx5pk"]
Sep 15 21:45:00.138021 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.138008    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5c7a87d-8597-43c8-a33d-eda0f5ffd474" containerName="payload-pre-processing"
Sep 15 21:45:00.138067 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.138022    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5c7a87d-8597-43c8-a33d-eda0f5ffd474" containerName="payload-pre-processing"
Sep 15 21:45:00.138067 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.138031    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36c0e3bf-baf4-47f2-8c7f-812c6459084f" containerName="istio-proxy"
Sep 15 21:45:00.138067 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.138036    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="36c0e3bf-baf4-47f2-8c7f-812c6459084f" containerName="istio-proxy"
Sep 15 21:45:00.138067 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.138042    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="442ee1bf-cbf1-4e22-99f1-0293cc3a6611" containerName="revoke-keys"
Sep 15 21:45:00.138067 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.138048    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="442ee1bf-cbf1-4e22-99f1-0293cc3a6611" containerName="revoke-keys"
Sep 15 21:45:00.138067 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.138054    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36c0e3bf-baf4-47f2-8c7f-812c6459084f" containerName="istio-proxy"
Sep 15 21:45:00.138067 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.138059    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="36c0e3bf-baf4-47f2-8c7f-812c6459084f" containerName="istio-proxy"
Sep 15 21:45:00.138282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.138085    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe" containerName="payload-processing"
Sep 15 21:45:00.138282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.138091    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe" containerName="payload-processing"
Sep 15 21:45:00.138282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.138149    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5c7a87d-8597-43c8-a33d-eda0f5ffd474" containerName="payload-pre-processing"
Sep 15 21:45:00.138282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.138157    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f0bb3e7-77e9-415f-82e2-4c6d7def9bfe" containerName="payload-processing"
Sep 15 21:45:00.138282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.138164    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="36c0e3bf-baf4-47f2-8c7f-812c6459084f" containerName="istio-proxy"
Sep 15 21:45:00.138282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.138171    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="442ee1bf-cbf1-4e22-99f1-0293cc3a6611" containerName="revoke-keys"
Sep 15 21:45:00.138282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.138176    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="36c0e3bf-baf4-47f2-8c7f-812c6459084f" containerName="istio-proxy"
Sep 15 21:45:00.143248 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.143227    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0da705-29825145-5c2th"]
Sep 15 21:45:00.143373 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.143336    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-745dff-29825145-gx5pk"
Sep 15 21:45:00.145421 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.145402    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:45:00.146593 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.146570    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825145-8dchd"]
Sep 15 21:45:00.146700 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.146682    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0da705-29825145-5c2th"
Sep 15 21:45:00.151684 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.151051    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-745dff-29825145-gx5pk"]
Sep 15 21:45:00.151684 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.151131    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-8dchd"
Sep 15 21:45:00.155363 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.155343    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0da705-29825145-5c2th"]
Sep 15 21:45:00.157770 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.157750    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-745dff-29825145-gx5pk"
Sep 15 21:45:00.158446 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.158394    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825145-8dchd"]
Sep 15 21:45:00.167433 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.167415    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0da705-29825145-5c2th"
Sep 15 21:45:00.208016 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.207992    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-8dchd"
Sep 15 21:45:00.443822 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.443756    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-745dff-29825145-gx5pk"]
Sep 15 21:45:00.450327 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:45:00.450286    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8540b799_1704_41c1_a277_67bd80e333d0.slice/crio-73f689764d412d486216cd85d0af2430a5bfa91e7c72a1b80da7a17e6c325a60 WatchSource:0}: Error finding container 73f689764d412d486216cd85d0af2430a5bfa91e7c72a1b80da7a17e6c325a60: Status 404 returned error can't find the container with id 73f689764d412d486216cd85d0af2430a5bfa91e7c72a1b80da7a17e6c325a60
Sep 15 21:45:00.453410 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.453361    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0da705-29825145-5c2th"]
Sep 15 21:45:00.455804 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:45:00.455781    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8960d672_6e6e_4e6e_a038_e54e58d743c9.slice/crio-6dd96f98347296c1d800e6c3947e68d9e4ea4a76d1dd05cc0b0e6723bfe6cf42 WatchSource:0}: Error finding container 6dd96f98347296c1d800e6c3947e68d9e4ea4a76d1dd05cc0b0e6723bfe6cf42: Status 404 returned error can't find the container with id 6dd96f98347296c1d800e6c3947e68d9e4ea4a76d1dd05cc0b0e6723bfe6cf42
Sep 15 21:45:00.499613 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.499590    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825145-8dchd"]
Sep 15 21:45:00.502328 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:45:00.502307    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87b7732_e9a7_4653_a499_38caaac5ae6e.slice/crio-7611a73b2fca02b91dffd2234f3a2373a0a965fc9c1676d034040d12ce826321 WatchSource:0}: Error finding container 7611a73b2fca02b91dffd2234f3a2373a0a965fc9c1676d034040d12ce826321: Status 404 returned error can't find the container with id 7611a73b2fca02b91dffd2234f3a2373a0a965fc9c1676d034040d12ce826321
Sep 15 21:45:00.555114 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.555080    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-8dchd" event={"ID":"c87b7732-e9a7-4653-a499-38caaac5ae6e","Type":"ContainerStarted","Data":"7611a73b2fca02b91dffd2234f3a2373a0a965fc9c1676d034040d12ce826321"}
Sep 15 21:45:00.556321 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.556298    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-745dff-29825145-gx5pk" event={"ID":"8540b799-1704-41c1-a277-67bd80e333d0","Type":"ContainerStarted","Data":"73f689764d412d486216cd85d0af2430a5bfa91e7c72a1b80da7a17e6c325a60"}
Sep 15 21:45:00.557487 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:00.557467    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0da705-29825145-5c2th" event={"ID":"8960d672-6e6e-4e6e-a038-e54e58d743c9","Type":"ContainerStarted","Data":"6dd96f98347296c1d800e6c3947e68d9e4ea4a76d1dd05cc0b0e6723bfe6cf42"}
Sep 15 21:45:01.564780 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:01.564752    2584 generic.go:356] "Generic (PLEG): container finished" podID="c87b7732-e9a7-4653-a499-38caaac5ae6e" containerID="2cc385361c21ef9e21f3596770ef4e41ab05293b54c55642bfc2c39195daf5e3" exitCode=0
Sep 15 21:45:01.565104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:01.564823    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-8dchd" event={"ID":"c87b7732-e9a7-4653-a499-38caaac5ae6e","Type":"ContainerDied","Data":"2cc385361c21ef9e21f3596770ef4e41ab05293b54c55642bfc2c39195daf5e3"}
Sep 15 21:45:01.566316 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:01.566291    2584 generic.go:356] "Generic (PLEG): container finished" podID="8540b799-1704-41c1-a277-67bd80e333d0" containerID="558ddc904931201016b9e438f9a59993a30809afc571f93fc84d31cd1b0fd5f9" exitCode=0
Sep 15 21:45:01.566455 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:01.566366    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-745dff-29825145-gx5pk" event={"ID":"8540b799-1704-41c1-a277-67bd80e333d0","Type":"ContainerDied","Data":"558ddc904931201016b9e438f9a59993a30809afc571f93fc84d31cd1b0fd5f9"}
Sep 15 21:45:01.567792 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:01.567772    2584 generic.go:356] "Generic (PLEG): container finished" podID="8960d672-6e6e-4e6e-a038-e54e58d743c9" containerID="bda957c619952fd54c051bb144bd17dbb9e9c3b38ba4a6f2f33ba0e01828cd61" exitCode=0
Sep 15 21:45:01.567880 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:01.567834    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0da705-29825145-5c2th" event={"ID":"8960d672-6e6e-4e6e-a038-e54e58d743c9","Type":"ContainerDied","Data":"bda957c619952fd54c051bb144bd17dbb9e9c3b38ba4a6f2f33ba0e01828cd61"}
Sep 15 21:45:02.722215 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:02.722187    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0da705-29825145-5c2th"
Sep 15 21:45:02.770740 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:02.770713    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-745dff-29825145-gx5pk"
Sep 15 21:45:02.774011 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:02.773986    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-8dchd"
Sep 15 21:45:03.580822 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:03.580791    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-8dchd"
Sep 15 21:45:03.581012 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:03.580787    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-8dchd" event={"ID":"c87b7732-e9a7-4653-a499-38caaac5ae6e","Type":"ContainerDied","Data":"7611a73b2fca02b91dffd2234f3a2373a0a965fc9c1676d034040d12ce826321"}
Sep 15 21:45:03.581012 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:03.580932    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7611a73b2fca02b91dffd2234f3a2373a0a965fc9c1676d034040d12ce826321"
Sep 15 21:45:03.582118 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:03.582097    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-745dff-29825145-gx5pk"
Sep 15 21:45:03.582118 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:03.582111    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-745dff-29825145-gx5pk" event={"ID":"8540b799-1704-41c1-a277-67bd80e333d0","Type":"ContainerDied","Data":"73f689764d412d486216cd85d0af2430a5bfa91e7c72a1b80da7a17e6c325a60"}
Sep 15 21:45:03.582284 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:03.582146    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73f689764d412d486216cd85d0af2430a5bfa91e7c72a1b80da7a17e6c325a60"
Sep 15 21:45:03.583475 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:03.583456    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0da705-29825145-5c2th"
Sep 15 21:45:03.583587 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:03.583474    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0da705-29825145-5c2th" event={"ID":"8960d672-6e6e-4e6e-a038-e54e58d743c9","Type":"ContainerDied","Data":"6dd96f98347296c1d800e6c3947e68d9e4ea4a76d1dd05cc0b0e6723bfe6cf42"}
Sep 15 21:45:03.583587 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:03.583508    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dd96f98347296c1d800e6c3947e68d9e4ea4a76d1dd05cc0b0e6723bfe6cf42"
Sep 15 21:45:32.760222 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:32.760191    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:45:32.771605 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:32.771580    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:45:37.427027 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.426993    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"]
Sep 15 21:45:37.427446 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.427263    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt" podUID="192e4d7e-dfc0-46bb-a356-4b5f479113a4" containerName="istio-proxy" containerID="cri-o://e6020d82905a0233e77999cfeb17d1fc2dea779d0798fc4a13c2672b52d4735a" gracePeriod=30
Sep 15 21:45:37.825509 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.825479    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw"]
Sep 15 21:45:37.825968 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.825951    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8960d672-6e6e-4e6e-a038-e54e58d743c9" containerName="cleanup"
Sep 15 21:45:37.826061 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.825970    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8960d672-6e6e-4e6e-a038-e54e58d743c9" containerName="cleanup"
Sep 15 21:45:37.826061 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.825987    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8540b799-1704-41c1-a277-67bd80e333d0" containerName="cleanup"
Sep 15 21:45:37.826061 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.825995    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8540b799-1704-41c1-a277-67bd80e333d0" containerName="cleanup"
Sep 15 21:45:37.826061 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.826030    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c87b7732-e9a7-4653-a499-38caaac5ae6e" containerName="cleanup"
Sep 15 21:45:37.826061 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.826041    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c87b7732-e9a7-4653-a499-38caaac5ae6e" containerName="cleanup"
Sep 15 21:45:37.826301 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.826140    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c87b7732-e9a7-4653-a499-38caaac5ae6e" containerName="cleanup"
Sep 15 21:45:37.826301 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.826154    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8540b799-1704-41c1-a277-67bd80e333d0" containerName="cleanup"
Sep 15 21:45:37.826301 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.826168    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8960d672-6e6e-4e6e-a038-e54e58d743c9" containerName="cleanup"
Sep 15 21:45:37.835585 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.835565    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw"
Sep 15 21:45:37.836184 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.836158    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw"]
Sep 15 21:45:37.837713 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.837694    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:45:37.837819 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.837696    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:45:37.865521 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.865500    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45e63e7-83a5-4748-9f4e-b56f6055f3af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw\" (UID: \"e45e63e7-83a5-4748-9f4e-b56f6055f3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw"
Sep 15 21:45:37.966188 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.966153    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45e63e7-83a5-4748-9f4e-b56f6055f3af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw\" (UID: \"e45e63e7-83a5-4748-9f4e-b56f6055f3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw"
Sep 15 21:45:37.966735 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:37.966717    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45e63e7-83a5-4748-9f4e-b56f6055f3af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw\" (UID: \"e45e63e7-83a5-4748-9f4e-b56f6055f3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw"
Sep 15 21:45:38.148087 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:38.148000    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw"
Sep 15 21:45:38.420155 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:38.419102    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw"]
Sep 15 21:45:38.771224 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:38.771200    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw" event={"ID":"e45e63e7-83a5-4748-9f4e-b56f6055f3af","Type":"ContainerStarted","Data":"40ff0c17794a3a70e923a5b8577d2b0a7369f6dfcdc61afaaf477d98b2e7cfe0"}
Sep 15 21:45:38.771548 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:38.771248    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw" event={"ID":"e45e63e7-83a5-4748-9f4e-b56f6055f3af","Type":"ContainerStarted","Data":"f07a954b8aa1ae7005db0c1a43efdd5754d11619f861a6e535ba980140a0e5d3"}
Sep 15 21:45:38.786058 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:38.786017    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw" podStartSLOduration=1.7860021879999999 podStartE2EDuration="1.786002188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:45:38.784130257 +0000 UTC m=+2106.646064060" watchObservedRunningTime="2026-09-15 21:45:38.786002188 +0000 UTC m=+2106.647935992"
Sep 15 21:45:39.777663 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:39.777635    2584 generic.go:356] "Generic (PLEG): container finished" podID="e45e63e7-83a5-4748-9f4e-b56f6055f3af" containerID="40ff0c17794a3a70e923a5b8577d2b0a7369f6dfcdc61afaaf477d98b2e7cfe0" exitCode=0
Sep 15 21:45:39.778041 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:39.777715    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw" event={"ID":"e45e63e7-83a5-4748-9f4e-b56f6055f3af","Type":"ContainerDied","Data":"40ff0c17794a3a70e923a5b8577d2b0a7369f6dfcdc61afaaf477d98b2e7cfe0"}
Sep 15 21:45:40.920662 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:40.920641    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw"
Sep 15 21:45:40.996424 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:40.996369    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45e63e7-83a5-4748-9f4e-b56f6055f3af-maas-api-service-ca\") pod \"e45e63e7-83a5-4748-9f4e-b56f6055f3af\" (UID: \"e45e63e7-83a5-4748-9f4e-b56f6055f3af\") "
Sep 15 21:45:40.996747 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:40.996722    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45e63e7-83a5-4748-9f4e-b56f6055f3af-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e45e63e7-83a5-4748-9f4e-b56f6055f3af" (UID: "e45e63e7-83a5-4748-9f4e-b56f6055f3af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:45:41.097777 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:41.097715    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45e63e7-83a5-4748-9f4e-b56f6055f3af-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:45:41.789957 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:41.789922    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw" event={"ID":"e45e63e7-83a5-4748-9f4e-b56f6055f3af","Type":"ContainerDied","Data":"f07a954b8aa1ae7005db0c1a43efdd5754d11619f861a6e535ba980140a0e5d3"}
Sep 15 21:45:41.789957 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:41.789947    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw"
Sep 15 21:45:41.790158 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:41.789974    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f07a954b8aa1ae7005db0c1a43efdd5754d11619f861a6e535ba980140a0e5d3"
Sep 15 21:45:42.674815 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.674790    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:45:42.796668 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.796637    2584 generic.go:356] "Generic (PLEG): container finished" podID="192e4d7e-dfc0-46bb-a356-4b5f479113a4" containerID="e6020d82905a0233e77999cfeb17d1fc2dea779d0798fc4a13c2672b52d4735a" exitCode=0
Sep 15 21:45:42.796864 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.796676    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt" event={"ID":"192e4d7e-dfc0-46bb-a356-4b5f479113a4","Type":"ContainerDied","Data":"e6020d82905a0233e77999cfeb17d1fc2dea779d0798fc4a13c2672b52d4735a"}
Sep 15 21:45:42.796864 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.796687    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"
Sep 15 21:45:42.796864 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.796758    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt" event={"ID":"192e4d7e-dfc0-46bb-a356-4b5f479113a4","Type":"ContainerDied","Data":"601d356140a7ac478f481aba305c8a5c66fba9e5bc40d3da66ca504c812bdf3c"}
Sep 15 21:45:42.796864 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.796790    2584 scope.go:117] "RemoveContainer" containerID="e6020d82905a0233e77999cfeb17d1fc2dea779d0798fc4a13c2672b52d4735a"
Sep 15 21:45:42.805308 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.805281    2584 scope.go:117] "RemoveContainer" containerID="95f851b92ccad588b784ffe6b0e3351b786e27b89b2b874bc94c0fd55002d8eb"
Sep 15 21:45:42.813312 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.813296    2584 scope.go:117] "RemoveContainer" containerID="e6020d82905a0233e77999cfeb17d1fc2dea779d0798fc4a13c2672b52d4735a"
Sep 15 21:45:42.813421 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.813404    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqjjh\" (UniqueName: \"kubernetes.io/projected/192e4d7e-dfc0-46bb-a356-4b5f479113a4-kube-api-access-hqjjh\") pod \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") "
Sep 15 21:45:42.813492 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.813448    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istiod-ca-cert\") pod \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") "
Sep 15 21:45:42.813492 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.813476    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-credential-socket\") pod \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") "
Sep 15 21:45:42.813593 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.813507    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-data\") pod \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") "
Sep 15 21:45:42.813646 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.813605    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-workload-certs\") pod \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") "
Sep 15 21:45:42.813646 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:45:42.813604    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6020d82905a0233e77999cfeb17d1fc2dea779d0798fc4a13c2672b52d4735a\": container with ID starting with e6020d82905a0233e77999cfeb17d1fc2dea779d0798fc4a13c2672b52d4735a not found: ID does not exist" containerID="e6020d82905a0233e77999cfeb17d1fc2dea779d0798fc4a13c2672b52d4735a"
Sep 15 21:45:42.813646 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.813630    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6020d82905a0233e77999cfeb17d1fc2dea779d0798fc4a13c2672b52d4735a"} err="failed to get container status \"e6020d82905a0233e77999cfeb17d1fc2dea779d0798fc4a13c2672b52d4735a\": rpc error: code = NotFound desc = could not find container \"e6020d82905a0233e77999cfeb17d1fc2dea779d0798fc4a13c2672b52d4735a\": container with ID starting with e6020d82905a0233e77999cfeb17d1fc2dea779d0798fc4a13c2672b52d4735a not found: ID does not exist"
Sep 15 21:45:42.813646 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.813642    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-envoy\") pod \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") "
Sep 15 21:45:42.813857 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.813653    2584 scope.go:117] "RemoveContainer" containerID="95f851b92ccad588b784ffe6b0e3351b786e27b89b2b874bc94c0fd55002d8eb"
Sep 15 21:45:42.813857 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.813718    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-podinfo\") pod \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") "
Sep 15 21:45:42.813857 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.813762    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "192e4d7e-dfc0-46bb-a356-4b5f479113a4" (UID: "192e4d7e-dfc0-46bb-a356-4b5f479113a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:42.813857 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.813775    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-token\") pod \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") "
Sep 15 21:45:42.813857 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.813802    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-workload-socket\") pod \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\" (UID: \"192e4d7e-dfc0-46bb-a356-4b5f479113a4\") "
Sep 15 21:45:42.814094 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.813860    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "192e4d7e-dfc0-46bb-a356-4b5f479113a4" (UID: "192e4d7e-dfc0-46bb-a356-4b5f479113a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:45:42.814094 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.813890    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "192e4d7e-dfc0-46bb-a356-4b5f479113a4" (UID: "192e4d7e-dfc0-46bb-a356-4b5f479113a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:42.814094 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:45:42.814004    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95f851b92ccad588b784ffe6b0e3351b786e27b89b2b874bc94c0fd55002d8eb\": container with ID starting with 95f851b92ccad588b784ffe6b0e3351b786e27b89b2b874bc94c0fd55002d8eb not found: ID does not exist" containerID="95f851b92ccad588b784ffe6b0e3351b786e27b89b2b874bc94c0fd55002d8eb"
Sep 15 21:45:42.814094 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.814037    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f851b92ccad588b784ffe6b0e3351b786e27b89b2b874bc94c0fd55002d8eb"} err="failed to get container status \"95f851b92ccad588b784ffe6b0e3351b786e27b89b2b874bc94c0fd55002d8eb\": rpc error: code = NotFound desc = could not find container \"95f851b92ccad588b784ffe6b0e3351b786e27b89b2b874bc94c0fd55002d8eb\": container with ID starting with 95f851b92ccad588b784ffe6b0e3351b786e27b89b2b874bc94c0fd55002d8eb not found: ID does not exist"
Sep 15 21:45:42.814293 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.814175    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "192e4d7e-dfc0-46bb-a356-4b5f479113a4" (UID: "192e4d7e-dfc0-46bb-a356-4b5f479113a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:42.814293 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.814181    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istiod-ca-cert\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:45:42.814293 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.814216    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-credential-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:45:42.814293 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.814232    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-workload-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:45:42.814517 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.814417    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "192e4d7e-dfc0-46bb-a356-4b5f479113a4" (UID: "192e4d7e-dfc0-46bb-a356-4b5f479113a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:42.815695 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.815670    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192e4d7e-dfc0-46bb-a356-4b5f479113a4-kube-api-access-hqjjh" (OuterVolumeSpecName: "kube-api-access-hqjjh") pod "192e4d7e-dfc0-46bb-a356-4b5f479113a4" (UID: "192e4d7e-dfc0-46bb-a356-4b5f479113a4"). InnerVolumeSpecName "kube-api-access-hqjjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:45:42.816025 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.816004    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "192e4d7e-dfc0-46bb-a356-4b5f479113a4" (UID: "192e4d7e-dfc0-46bb-a356-4b5f479113a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:45:42.816025 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.816021    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "192e4d7e-dfc0-46bb-a356-4b5f479113a4" (UID: "192e4d7e-dfc0-46bb-a356-4b5f479113a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:42.816152 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.816074    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "192e4d7e-dfc0-46bb-a356-4b5f479113a4" (UID: "192e4d7e-dfc0-46bb-a356-4b5f479113a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:45:42.914955 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.914933    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-token\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:45:42.915058 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.914967    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-workload-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:45:42.915058 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.914978    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqjjh\" (UniqueName: \"kubernetes.io/projected/192e4d7e-dfc0-46bb-a356-4b5f479113a4-kube-api-access-hqjjh\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:45:42.915058 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.914988    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-data\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:45:42.915058 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.914995    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-envoy\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:45:42.915058 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:42.915002    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192e4d7e-dfc0-46bb-a356-4b5f479113a4-istio-podinfo\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:45:43.126607 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:43.126579    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"]
Sep 15 21:45:43.143978 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:43.143951    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6d9d39-openshift-default-6f54878ffb-v5ggt"]
Sep 15 21:45:44.674401 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:44.674353    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="192e4d7e-dfc0-46bb-a356-4b5f479113a4" path="/var/lib/kubelet/pods/192e4d7e-dfc0-46bb-a356-4b5f479113a4/volumes"
Sep 15 21:45:48.402577 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:48.402544    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0da705-29825145-5c2th"]
Sep 15 21:45:48.405613 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:48.405584    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0da705-29825145-5c2th"]
Sep 15 21:45:48.593064 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:48.593015    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw"]
Sep 15 21:45:48.594995 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:48.594973    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0da705-dtthw"]
Sep 15 21:45:48.674761 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:48.674667    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8960d672-6e6e-4e6e-a038-e54e58d743c9" path="/var/lib/kubelet/pods/8960d672-6e6e-4e6e-a038-e54e58d743c9/volumes"
Sep 15 21:45:48.675131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:48.675110    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e45e63e7-83a5-4748-9f4e-b56f6055f3af" path="/var/lib/kubelet/pods/e45e63e7-83a5-4748-9f4e-b56f6055f3af/volumes"
Sep 15 21:45:55.130228 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.130198    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"]
Sep 15 21:45:55.130625 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.130456    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x" podUID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" containerName="istio-proxy" containerID="cri-o://6dc3f7db7ba423ffb63176aaf0fc9113fe8805bb1dccd978f594440d2b311ecf" gracePeriod=30
Sep 15 21:45:55.508282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.508209    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85"]
Sep 15 21:45:55.508667 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.508653    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="192e4d7e-dfc0-46bb-a356-4b5f479113a4" containerName="istio-proxy"
Sep 15 21:45:55.508667 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.508668    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="192e4d7e-dfc0-46bb-a356-4b5f479113a4" containerName="istio-proxy"
Sep 15 21:45:55.508757 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.508681    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e45e63e7-83a5-4748-9f4e-b56f6055f3af" containerName="revoke-keys"
Sep 15 21:45:55.508757 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.508687    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e45e63e7-83a5-4748-9f4e-b56f6055f3af" containerName="revoke-keys"
Sep 15 21:45:55.508757 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.508694    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="192e4d7e-dfc0-46bb-a356-4b5f479113a4" containerName="istio-proxy"
Sep 15 21:45:55.508757 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.508700    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="192e4d7e-dfc0-46bb-a356-4b5f479113a4" containerName="istio-proxy"
Sep 15 21:45:55.508757 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.508757    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="192e4d7e-dfc0-46bb-a356-4b5f479113a4" containerName="istio-proxy"
Sep 15 21:45:55.508904 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.508769    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="192e4d7e-dfc0-46bb-a356-4b5f479113a4" containerName="istio-proxy"
Sep 15 21:45:55.508904 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.508776    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e45e63e7-83a5-4748-9f4e-b56f6055f3af" containerName="revoke-keys"
Sep 15 21:45:55.513280 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.513264    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85"
Sep 15 21:45:55.515699 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.515675    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:45:55.515868 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.515679    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:45:55.517924 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.517904    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85"]
Sep 15 21:45:55.626260 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.626237    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd43459-b2ac-4308-94fc-41de911ac43f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85\" (UID: \"0bd43459-b2ac-4308-94fc-41de911ac43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85"
Sep 15 21:45:55.727648 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.727623    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd43459-b2ac-4308-94fc-41de911ac43f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85\" (UID: \"0bd43459-b2ac-4308-94fc-41de911ac43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85"
Sep 15 21:45:55.728197 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.728180    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd43459-b2ac-4308-94fc-41de911ac43f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85\" (UID: \"0bd43459-b2ac-4308-94fc-41de911ac43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85"
Sep 15 21:45:55.824842 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:55.824819    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85"
Sep 15 21:45:56.098109 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:56.098083    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85"]
Sep 15 21:45:56.100841 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:45:56.100815    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd43459_b2ac_4308_94fc_41de911ac43f.slice/crio-99d42871d2f339a72cb4f80edc2025885fe05c03edb6a4ab60a9244a994596bc WatchSource:0}: Error finding container 99d42871d2f339a72cb4f80edc2025885fe05c03edb6a4ab60a9244a994596bc: Status 404 returned error can't find the container with id 99d42871d2f339a72cb4f80edc2025885fe05c03edb6a4ab60a9244a994596bc
Sep 15 21:45:56.876533 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:56.876503    2584 generic.go:356] "Generic (PLEG): container finished" podID="0bd43459-b2ac-4308-94fc-41de911ac43f" containerID="f6f07b0563d1b4216e6eefaa8a339f01ce710c7466ec16d5beb2ce900e7ad954" exitCode=0
Sep 15 21:45:56.876533 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:56.876542    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85" event={"ID":"0bd43459-b2ac-4308-94fc-41de911ac43f","Type":"ContainerDied","Data":"f6f07b0563d1b4216e6eefaa8a339f01ce710c7466ec16d5beb2ce900e7ad954"}
Sep 15 21:45:56.876967 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:56.876594    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85" event={"ID":"0bd43459-b2ac-4308-94fc-41de911ac43f","Type":"ContainerStarted","Data":"99d42871d2f339a72cb4f80edc2025885fe05c03edb6a4ab60a9244a994596bc"}
Sep 15 21:45:58.015165 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:58.015144    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85"
Sep 15 21:45:58.153821 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:58.153746    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd43459-b2ac-4308-94fc-41de911ac43f-maas-api-service-ca\") pod \"0bd43459-b2ac-4308-94fc-41de911ac43f\" (UID: \"0bd43459-b2ac-4308-94fc-41de911ac43f\") "
Sep 15 21:45:58.154179 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:58.154152    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd43459-b2ac-4308-94fc-41de911ac43f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0bd43459-b2ac-4308-94fc-41de911ac43f" (UID: "0bd43459-b2ac-4308-94fc-41de911ac43f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:45:58.254759 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:58.254726    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd43459-b2ac-4308-94fc-41de911ac43f-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:45:58.888802 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:58.888775    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85"
Sep 15 21:45:58.888954 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:58.888801    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85" event={"ID":"0bd43459-b2ac-4308-94fc-41de911ac43f","Type":"ContainerDied","Data":"99d42871d2f339a72cb4f80edc2025885fe05c03edb6a4ab60a9244a994596bc"}
Sep 15 21:45:58.888954 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:45:58.888845    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99d42871d2f339a72cb4f80edc2025885fe05c03edb6a4ab60a9244a994596bc"
Sep 15 21:46:00.390356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.390332    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:46:00.475492 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.475417    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-workload-socket\") pod \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") "
Sep 15 21:46:00.475492 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.475457    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-credential-socket\") pod \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") "
Sep 15 21:46:00.475492 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.475488    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istiod-ca-cert\") pod \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") "
Sep 15 21:46:00.475758 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.475548    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shprz\" (UniqueName: \"kubernetes.io/projected/59ce2e8f-0198-4399-b52f-a965a5b2c12d-kube-api-access-shprz\") pod \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") "
Sep 15 21:46:00.475758 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.475604    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-token\") pod \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") "
Sep 15 21:46:00.475758 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.475630    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-envoy\") pod \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") "
Sep 15 21:46:00.475758 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.475661    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-data\") pod \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") "
Sep 15 21:46:00.475758 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.475684    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-podinfo\") pod \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") "
Sep 15 21:46:00.475758 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.475721    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-workload-certs\") pod \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\" (UID: \"59ce2e8f-0198-4399-b52f-a965a5b2c12d\") "
Sep 15 21:46:00.476104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.475754    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "59ce2e8f-0198-4399-b52f-a965a5b2c12d" (UID: "59ce2e8f-0198-4399-b52f-a965a5b2c12d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:00.476104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.475770    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "59ce2e8f-0198-4399-b52f-a965a5b2c12d" (UID: "59ce2e8f-0198-4399-b52f-a965a5b2c12d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:00.476104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.475976    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-data" (OuterVolumeSpecName: "istio-data") pod "59ce2e8f-0198-4399-b52f-a965a5b2c12d" (UID: "59ce2e8f-0198-4399-b52f-a965a5b2c12d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:00.476104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.476002    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "59ce2e8f-0198-4399-b52f-a965a5b2c12d" (UID: "59ce2e8f-0198-4399-b52f-a965a5b2c12d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:46:00.476325 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.476144    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-data\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:00.476325 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.476162    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-workload-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:00.476325 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.476175    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-credential-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:00.476325 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.476188    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istiod-ca-cert\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:00.476325 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.476217    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "59ce2e8f-0198-4399-b52f-a965a5b2c12d" (UID: "59ce2e8f-0198-4399-b52f-a965a5b2c12d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:00.477905 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.477878    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ce2e8f-0198-4399-b52f-a965a5b2c12d-kube-api-access-shprz" (OuterVolumeSpecName: "kube-api-access-shprz") pod "59ce2e8f-0198-4399-b52f-a965a5b2c12d" (UID: "59ce2e8f-0198-4399-b52f-a965a5b2c12d"). InnerVolumeSpecName "kube-api-access-shprz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:00.478003 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.477907    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "59ce2e8f-0198-4399-b52f-a965a5b2c12d" (UID: "59ce2e8f-0198-4399-b52f-a965a5b2c12d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:00.478063 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.478050    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "59ce2e8f-0198-4399-b52f-a965a5b2c12d" (UID: "59ce2e8f-0198-4399-b52f-a965a5b2c12d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:46:00.478188 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.478173    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-token" (OuterVolumeSpecName: "istio-token") pod "59ce2e8f-0198-4399-b52f-a965a5b2c12d" (UID: "59ce2e8f-0198-4399-b52f-a965a5b2c12d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:00.577266 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.577241    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shprz\" (UniqueName: \"kubernetes.io/projected/59ce2e8f-0198-4399-b52f-a965a5b2c12d-kube-api-access-shprz\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:00.577266 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.577264    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-token\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:00.577440 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.577275    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-envoy\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:00.577440 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.577283    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ce2e8f-0198-4399-b52f-a965a5b2c12d-istio-podinfo\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:00.577440 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.577290    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ce2e8f-0198-4399-b52f-a965a5b2c12d-workload-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:00.900795 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.900767    2584 generic.go:356] "Generic (PLEG): container finished" podID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" containerID="6dc3f7db7ba423ffb63176aaf0fc9113fe8805bb1dccd978f594440d2b311ecf" exitCode=0
Sep 15 21:46:00.900943 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.900811    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"
Sep 15 21:46:00.900943 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.900860    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x" event={"ID":"59ce2e8f-0198-4399-b52f-a965a5b2c12d","Type":"ContainerDied","Data":"6dc3f7db7ba423ffb63176aaf0fc9113fe8805bb1dccd978f594440d2b311ecf"}
Sep 15 21:46:00.900943 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.900909    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x" event={"ID":"59ce2e8f-0198-4399-b52f-a965a5b2c12d","Type":"ContainerDied","Data":"b970b808b14d1ae8d58e4ba46e2a9fd7cd8d777d64ff0aa50b0f8e82b8b9e7c0"}
Sep 15 21:46:00.900943 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.900930    2584 scope.go:117] "RemoveContainer" containerID="6dc3f7db7ba423ffb63176aaf0fc9113fe8805bb1dccd978f594440d2b311ecf"
Sep 15 21:46:00.909707 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.909691    2584 scope.go:117] "RemoveContainer" containerID="4bf743b051c27a7148d9a609aa4aa06734bb2fba33bb8ddf13bfcb0fdfc39983"
Sep 15 21:46:00.917602 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.917578    2584 scope.go:117] "RemoveContainer" containerID="6dc3f7db7ba423ffb63176aaf0fc9113fe8805bb1dccd978f594440d2b311ecf"
Sep 15 21:46:00.917870 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:46:00.917821    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc3f7db7ba423ffb63176aaf0fc9113fe8805bb1dccd978f594440d2b311ecf\": container with ID starting with 6dc3f7db7ba423ffb63176aaf0fc9113fe8805bb1dccd978f594440d2b311ecf not found: ID does not exist" containerID="6dc3f7db7ba423ffb63176aaf0fc9113fe8805bb1dccd978f594440d2b311ecf"
Sep 15 21:46:00.917870 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.917845    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc3f7db7ba423ffb63176aaf0fc9113fe8805bb1dccd978f594440d2b311ecf"} err="failed to get container status \"6dc3f7db7ba423ffb63176aaf0fc9113fe8805bb1dccd978f594440d2b311ecf\": rpc error: code = NotFound desc = could not find container \"6dc3f7db7ba423ffb63176aaf0fc9113fe8805bb1dccd978f594440d2b311ecf\": container with ID starting with 6dc3f7db7ba423ffb63176aaf0fc9113fe8805bb1dccd978f594440d2b311ecf not found: ID does not exist"
Sep 15 21:46:00.917870 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.917860    2584 scope.go:117] "RemoveContainer" containerID="4bf743b051c27a7148d9a609aa4aa06734bb2fba33bb8ddf13bfcb0fdfc39983"
Sep 15 21:46:00.918099 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:46:00.918080    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bf743b051c27a7148d9a609aa4aa06734bb2fba33bb8ddf13bfcb0fdfc39983\": container with ID starting with 4bf743b051c27a7148d9a609aa4aa06734bb2fba33bb8ddf13bfcb0fdfc39983 not found: ID does not exist" containerID="4bf743b051c27a7148d9a609aa4aa06734bb2fba33bb8ddf13bfcb0fdfc39983"
Sep 15 21:46:00.918156 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.918104    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf743b051c27a7148d9a609aa4aa06734bb2fba33bb8ddf13bfcb0fdfc39983"} err="failed to get container status \"4bf743b051c27a7148d9a609aa4aa06734bb2fba33bb8ddf13bfcb0fdfc39983\": rpc error: code = NotFound desc = could not find container \"4bf743b051c27a7148d9a609aa4aa06734bb2fba33bb8ddf13bfcb0fdfc39983\": container with ID starting with 4bf743b051c27a7148d9a609aa4aa06734bb2fba33bb8ddf13bfcb0fdfc39983 not found: ID does not exist"
Sep 15 21:46:00.919068 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.919047    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"]
Sep 15 21:46:00.922469 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:00.922449    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0da705-openshift-default-79459d4dfd-hzn4x"]
Sep 15 21:46:02.673764 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:02.673733    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" path="/var/lib/kubelet/pods/59ce2e8f-0198-4399-b52f-a965a5b2c12d/volumes"
Sep 15 21:46:06.397014 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.396982    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"]
Sep 15 21:46:06.397647 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.397214    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5" podUID="e5e75854-bfed-4c73-99c6-486d34b60b57" containerName="payload-pre-processing" containerID="cri-o://38f2b9f303d979e3567f8c10eba88bc0e832a209f056973f8746b58046996b7b" gracePeriod=30
Sep 15 21:46:06.555570 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.555540    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85"]
Sep 15 21:46:06.563694 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.563664    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f36399-dps85"]
Sep 15 21:46:06.659456 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.659404    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"
Sep 15 21:46:06.672581 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.672557    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bd43459-b2ac-4308-94fc-41de911ac43f" path="/var/lib/kubelet/pods/0bd43459-b2ac-4308-94fc-41de911ac43f/volumes"
Sep 15 21:46:06.832794 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.832767    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e75854-bfed-4c73-99c6-486d34b60b57-plugins-config-volume\") pod \"e5e75854-bfed-4c73-99c6-486d34b60b57\" (UID: \"e5e75854-bfed-4c73-99c6-486d34b60b57\") "
Sep 15 21:46:06.832967 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.832812    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62zr5\" (UniqueName: \"kubernetes.io/projected/e5e75854-bfed-4c73-99c6-486d34b60b57-kube-api-access-62zr5\") pod \"e5e75854-bfed-4c73-99c6-486d34b60b57\" (UID: \"e5e75854-bfed-4c73-99c6-486d34b60b57\") "
Sep 15 21:46:06.833161 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.833136    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e75854-bfed-4c73-99c6-486d34b60b57-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e5e75854-bfed-4c73-99c6-486d34b60b57" (UID: "e5e75854-bfed-4c73-99c6-486d34b60b57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:46:06.834784 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.834762    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e75854-bfed-4c73-99c6-486d34b60b57-kube-api-access-62zr5" (OuterVolumeSpecName: "kube-api-access-62zr5") pod "e5e75854-bfed-4c73-99c6-486d34b60b57" (UID: "e5e75854-bfed-4c73-99c6-486d34b60b57"). InnerVolumeSpecName "kube-api-access-62zr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:06.933558 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.933493    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e75854-bfed-4c73-99c6-486d34b60b57-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:06.933558 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.933514    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62zr5\" (UniqueName: \"kubernetes.io/projected/e5e75854-bfed-4c73-99c6-486d34b60b57-kube-api-access-62zr5\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:06.934504 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.934483    2584 generic.go:356] "Generic (PLEG): container finished" podID="e5e75854-bfed-4c73-99c6-486d34b60b57" containerID="38f2b9f303d979e3567f8c10eba88bc0e832a209f056973f8746b58046996b7b" exitCode=0
Sep 15 21:46:06.934587 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.934534    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"
Sep 15 21:46:06.934587 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.934573    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5" event={"ID":"e5e75854-bfed-4c73-99c6-486d34b60b57","Type":"ContainerDied","Data":"38f2b9f303d979e3567f8c10eba88bc0e832a209f056973f8746b58046996b7b"}
Sep 15 21:46:06.934666 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.934620    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5" event={"ID":"e5e75854-bfed-4c73-99c6-486d34b60b57","Type":"ContainerDied","Data":"f865931c571c1597eaf5ff817f1c60431213ad72718f3f30b425ffc51cb37e6c"}
Sep 15 21:46:06.934666 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.934641    2584 scope.go:117] "RemoveContainer" containerID="38f2b9f303d979e3567f8c10eba88bc0e832a209f056973f8746b58046996b7b"
Sep 15 21:46:06.943186 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.943170    2584 scope.go:117] "RemoveContainer" containerID="38f2b9f303d979e3567f8c10eba88bc0e832a209f056973f8746b58046996b7b"
Sep 15 21:46:06.943454 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:46:06.943428    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f2b9f303d979e3567f8c10eba88bc0e832a209f056973f8746b58046996b7b\": container with ID starting with 38f2b9f303d979e3567f8c10eba88bc0e832a209f056973f8746b58046996b7b not found: ID does not exist" containerID="38f2b9f303d979e3567f8c10eba88bc0e832a209f056973f8746b58046996b7b"
Sep 15 21:46:06.943508 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.943467    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f2b9f303d979e3567f8c10eba88bc0e832a209f056973f8746b58046996b7b"} err="failed to get container status \"38f2b9f303d979e3567f8c10eba88bc0e832a209f056973f8746b58046996b7b\": rpc error: code = NotFound desc = could not find container \"38f2b9f303d979e3567f8c10eba88bc0e832a209f056973f8746b58046996b7b\": container with ID starting with 38f2b9f303d979e3567f8c10eba88bc0e832a209f056973f8746b58046996b7b not found: ID does not exist"
Sep 15 21:46:06.957011 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.956987    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"]
Sep 15 21:46:06.962519 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:06.962499    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f36399-84b477f78b-789f5"]
Sep 15 21:46:08.672817 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:08.672739    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5e75854-bfed-4c73-99c6-486d34b60b57" path="/var/lib/kubelet/pods/e5e75854-bfed-4c73-99c6-486d34b60b57/volumes"
Sep 15 21:46:13.472514 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.472482    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn"]
Sep 15 21:46:13.472903 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.472890    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" containerName="istio-proxy"
Sep 15 21:46:13.472946 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.472906    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" containerName="istio-proxy"
Sep 15 21:46:13.472946 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.472919    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" containerName="istio-proxy"
Sep 15 21:46:13.472946 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.472925    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" containerName="istio-proxy"
Sep 15 21:46:13.472946 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.472934    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd43459-b2ac-4308-94fc-41de911ac43f" containerName="revoke-keys"
Sep 15 21:46:13.472946 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.472939    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd43459-b2ac-4308-94fc-41de911ac43f" containerName="revoke-keys"
Sep 15 21:46:13.472946 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.472947    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e75854-bfed-4c73-99c6-486d34b60b57" containerName="payload-pre-processing"
Sep 15 21:46:13.473125 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.472953    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e75854-bfed-4c73-99c6-486d34b60b57" containerName="payload-pre-processing"
Sep 15 21:46:13.473125 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.473010    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e75854-bfed-4c73-99c6-486d34b60b57" containerName="payload-pre-processing"
Sep 15 21:46:13.473125 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.473016    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bd43459-b2ac-4308-94fc-41de911ac43f" containerName="revoke-keys"
Sep 15 21:46:13.473125 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.473025    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" containerName="istio-proxy"
Sep 15 21:46:13.476334 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.476318    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn"
Sep 15 21:46:13.478356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.478338    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:46:13.478593 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.478576    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-82cf7\""
Sep 15 21:46:13.482028 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.482007    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn"]
Sep 15 21:46:13.483417 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.483395    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38af9c5e-bb64-45f1-9d41-12a766cef2a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn\" (UID: \"38af9c5e-bb64-45f1-9d41-12a766cef2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn"
Sep 15 21:46:13.584725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.584699    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38af9c5e-bb64-45f1-9d41-12a766cef2a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn\" (UID: \"38af9c5e-bb64-45f1-9d41-12a766cef2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn"
Sep 15 21:46:13.585274 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.585255    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38af9c5e-bb64-45f1-9d41-12a766cef2a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn\" (UID: \"38af9c5e-bb64-45f1-9d41-12a766cef2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn"
Sep 15 21:46:13.787191 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:13.787163    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn"
Sep 15 21:46:14.056181 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:14.056157    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn"]
Sep 15 21:46:14.057653 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:46:14.057631    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38af9c5e_bb64_45f1_9d41_12a766cef2a3.slice/crio-7467bd03b444c8b071ab7ea1fea1fbac17b2193e65fd8896ce07d5aeea6f603c WatchSource:0}: Error finding container 7467bd03b444c8b071ab7ea1fea1fbac17b2193e65fd8896ce07d5aeea6f603c: Status 404 returned error can't find the container with id 7467bd03b444c8b071ab7ea1fea1fbac17b2193e65fd8896ce07d5aeea6f603c
Sep 15 21:46:14.981115 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:14.981086    2584 generic.go:356] "Generic (PLEG): container finished" podID="38af9c5e-bb64-45f1-9d41-12a766cef2a3" containerID="13cf2cb22cf31e6429a760d84667c5027ce2f61010efaef226731d5e2a1af42c" exitCode=0
Sep 15 21:46:14.981466 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:14.981135    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn" event={"ID":"38af9c5e-bb64-45f1-9d41-12a766cef2a3","Type":"ContainerDied","Data":"13cf2cb22cf31e6429a760d84667c5027ce2f61010efaef226731d5e2a1af42c"}
Sep 15 21:46:14.981466 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:14.981174    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn" event={"ID":"38af9c5e-bb64-45f1-9d41-12a766cef2a3","Type":"ContainerStarted","Data":"7467bd03b444c8b071ab7ea1fea1fbac17b2193e65fd8896ce07d5aeea6f603c"}
Sep 15 21:46:16.118799 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:16.118773    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn"
Sep 15 21:46:16.210615 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:16.210593    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38af9c5e-bb64-45f1-9d41-12a766cef2a3-maas-api-service-ca\") pod \"38af9c5e-bb64-45f1-9d41-12a766cef2a3\" (UID: \"38af9c5e-bb64-45f1-9d41-12a766cef2a3\") "
Sep 15 21:46:16.210984 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:16.210958    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38af9c5e-bb64-45f1-9d41-12a766cef2a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38af9c5e-bb64-45f1-9d41-12a766cef2a3" (UID: "38af9c5e-bb64-45f1-9d41-12a766cef2a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:46:16.311779 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:16.311755    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38af9c5e-bb64-45f1-9d41-12a766cef2a3-maas-api-service-ca\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:16.993886 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:16.993811    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn" event={"ID":"38af9c5e-bb64-45f1-9d41-12a766cef2a3","Type":"ContainerDied","Data":"7467bd03b444c8b071ab7ea1fea1fbac17b2193e65fd8896ce07d5aeea6f603c"}
Sep 15 21:46:16.993886 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:16.993864    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7467bd03b444c8b071ab7ea1fea1fbac17b2193e65fd8896ce07d5aeea6f603c"
Sep 15 21:46:16.993886 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:16.993817    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn"
Sep 15 21:46:24.138306 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:24.138269    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-745dff-29825145-gx5pk"]
Sep 15 21:46:24.141550 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:24.141526    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-745dff-29825145-gx5pk"]
Sep 15 21:46:24.168257 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:24.168226    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"]
Sep 15 21:46:24.168525 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:24.168484    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn" podUID="4b33ab9d-8cf7-4c02-b455-f6f604998bef" containerName="payload-processing" containerID="cri-o://43a83e5fed2d3510a9222cca3d8a52621c13c2450b544c8fe1544ebd9fe9f26d" gracePeriod=30
Sep 15 21:46:24.400083 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:24.400055    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn"]
Sep 15 21:46:24.408047 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:24.408026    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-745dff-wscrn"]
Sep 15 21:46:24.420359 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:24.420340    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"
Sep 15 21:46:24.479210 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:24.479185    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b33ab9d-8cf7-4c02-b455-f6f604998bef-plugins-config-volume\") pod \"4b33ab9d-8cf7-4c02-b455-f6f604998bef\" (UID: \"4b33ab9d-8cf7-4c02-b455-f6f604998bef\") "
Sep 15 21:46:24.479402 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:24.479281    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g4gv\" (UniqueName: \"kubernetes.io/projected/4b33ab9d-8cf7-4c02-b455-f6f604998bef-kube-api-access-9g4gv\") pod \"4b33ab9d-8cf7-4c02-b455-f6f604998bef\" (UID: \"4b33ab9d-8cf7-4c02-b455-f6f604998bef\") "
Sep 15 21:46:24.479576 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:24.479555    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b33ab9d-8cf7-4c02-b455-f6f604998bef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b33ab9d-8cf7-4c02-b455-f6f604998bef" (UID: "4b33ab9d-8cf7-4c02-b455-f6f604998bef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:46:24.481205 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:24.481183    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b33ab9d-8cf7-4c02-b455-f6f604998bef-kube-api-access-9g4gv" (OuterVolumeSpecName: "kube-api-access-9g4gv") pod "4b33ab9d-8cf7-4c02-b455-f6f604998bef" (UID: "4b33ab9d-8cf7-4c02-b455-f6f604998bef"). InnerVolumeSpecName "kube-api-access-9g4gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:24.580131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:24.580100    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b33ab9d-8cf7-4c02-b455-f6f604998bef-plugins-config-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:24.580131 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:24.580126    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g4gv\" (UniqueName: \"kubernetes.io/projected/4b33ab9d-8cf7-4c02-b455-f6f604998bef-kube-api-access-9g4gv\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:24.674262 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:24.674198    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38af9c5e-bb64-45f1-9d41-12a766cef2a3" path="/var/lib/kubelet/pods/38af9c5e-bb64-45f1-9d41-12a766cef2a3/volumes"
Sep 15 21:46:24.674548 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:24.674531    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8540b799-1704-41c1-a277-67bd80e333d0" path="/var/lib/kubelet/pods/8540b799-1704-41c1-a277-67bd80e333d0/volumes"
Sep 15 21:46:25.038095 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:25.038065    2584 generic.go:356] "Generic (PLEG): container finished" podID="4b33ab9d-8cf7-4c02-b455-f6f604998bef" containerID="43a83e5fed2d3510a9222cca3d8a52621c13c2450b544c8fe1544ebd9fe9f26d" exitCode=0
Sep 15 21:46:25.038268 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:25.038108    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"
Sep 15 21:46:25.038268 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:25.038155    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn" event={"ID":"4b33ab9d-8cf7-4c02-b455-f6f604998bef","Type":"ContainerDied","Data":"43a83e5fed2d3510a9222cca3d8a52621c13c2450b544c8fe1544ebd9fe9f26d"}
Sep 15 21:46:25.038268 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:25.038215    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn" event={"ID":"4b33ab9d-8cf7-4c02-b455-f6f604998bef","Type":"ContainerDied","Data":"0e6a2f2aeb4e6d02ea24e37c55bb40b89b53b38687a54908dbff0c4f4f87e2b6"}
Sep 15 21:46:25.038268 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:25.038243    2584 scope.go:117] "RemoveContainer" containerID="43a83e5fed2d3510a9222cca3d8a52621c13c2450b544c8fe1544ebd9fe9f26d"
Sep 15 21:46:25.048886 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:25.048857    2584 scope.go:117] "RemoveContainer" containerID="43a83e5fed2d3510a9222cca3d8a52621c13c2450b544c8fe1544ebd9fe9f26d"
Sep 15 21:46:25.049231 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:46:25.049208    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a83e5fed2d3510a9222cca3d8a52621c13c2450b544c8fe1544ebd9fe9f26d\": container with ID starting with 43a83e5fed2d3510a9222cca3d8a52621c13c2450b544c8fe1544ebd9fe9f26d not found: ID does not exist" containerID="43a83e5fed2d3510a9222cca3d8a52621c13c2450b544c8fe1544ebd9fe9f26d"
Sep 15 21:46:25.049338 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:25.049239    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a83e5fed2d3510a9222cca3d8a52621c13c2450b544c8fe1544ebd9fe9f26d"} err="failed to get container status \"43a83e5fed2d3510a9222cca3d8a52621c13c2450b544c8fe1544ebd9fe9f26d\": rpc error: code = NotFound desc = could not find container \"43a83e5fed2d3510a9222cca3d8a52621c13c2450b544c8fe1544ebd9fe9f26d\": container with ID starting with 43a83e5fed2d3510a9222cca3d8a52621c13c2450b544c8fe1544ebd9fe9f26d not found: ID does not exist"
Sep 15 21:46:25.055711 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:25.055688    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"]
Sep 15 21:46:25.057571 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:25.057548    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-745dff-69fd997ddf-9cfzn"]
Sep 15 21:46:26.672589 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:26.672551    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b33ab9d-8cf7-4c02-b455-f6f604998bef" path="/var/lib/kubelet/pods/4b33ab9d-8cf7-4c02-b455-f6f604998bef/volumes"
Sep 15 21:46:31.171521 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:31.171492    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"]
Sep 15 21:46:31.171961 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:31.171736    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p" podUID="4437923b-e05b-4d87-896e-7cc6f9b61b7b" containerName="istio-proxy" containerID="cri-o://d8ff5fac1c20f6c01377f14fa47337ab6a8e47bbd65061464fd7ba2c833671f1" gracePeriod=30
Sep 15 21:46:36.436882 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.436858    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:46:36.490443 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.490350    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-workload-certs\") pod \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") "
Sep 15 21:46:36.490645 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.490604    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4437923b-e05b-4d87-896e-7cc6f9b61b7b" (UID: "4437923b-e05b-4d87-896e-7cc6f9b61b7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:36.490823 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.490797    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-token\") pod \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") "
Sep 15 21:46:36.490985 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.490957    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-envoy\") pod \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") "
Sep 15 21:46:36.491126 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.491110    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-workload-socket\") pod \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") "
Sep 15 21:46:36.491246 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.491233    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-credential-socket\") pod \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") "
Sep 15 21:46:36.491359 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.491348    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-data\") pod \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") "
Sep 15 21:46:36.491801 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.491785    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-podinfo\") pod \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") "
Sep 15 21:46:36.492236 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.492221    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkgxc\" (UniqueName: \"kubernetes.io/projected/4437923b-e05b-4d87-896e-7cc6f9b61b7b-kube-api-access-rkgxc\") pod \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") "
Sep 15 21:46:36.492419 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.492361    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istiod-ca-cert\") pod \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\" (UID: \"4437923b-e05b-4d87-896e-7cc6f9b61b7b\") "
Sep 15 21:46:36.492753 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.491612    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4437923b-e05b-4d87-896e-7cc6f9b61b7b" (UID: "4437923b-e05b-4d87-896e-7cc6f9b61b7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:36.492753 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.491684    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-data" (OuterVolumeSpecName: "istio-data") pod "4437923b-e05b-4d87-896e-7cc6f9b61b7b" (UID: "4437923b-e05b-4d87-896e-7cc6f9b61b7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:36.492753 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.491747    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4437923b-e05b-4d87-896e-7cc6f9b61b7b" (UID: "4437923b-e05b-4d87-896e-7cc6f9b61b7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:36.492964 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.492796    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4437923b-e05b-4d87-896e-7cc6f9b61b7b" (UID: "4437923b-e05b-4d87-896e-7cc6f9b61b7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:46:36.493370 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.493352    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-workload-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:36.493520 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.493504    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-workload-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:36.495962 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.494190    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-credential-socket\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:36.495962 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.494209    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-data\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:36.495962 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.494242    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istiod-ca-cert\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:36.495962 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.494371    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4437923b-e05b-4d87-896e-7cc6f9b61b7b" (UID: "4437923b-e05b-4d87-896e-7cc6f9b61b7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:36.495962 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.494514    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-token" (OuterVolumeSpecName: "istio-token") pod "4437923b-e05b-4d87-896e-7cc6f9b61b7b" (UID: "4437923b-e05b-4d87-896e-7cc6f9b61b7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:36.496755 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.496707    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4437923b-e05b-4d87-896e-7cc6f9b61b7b-kube-api-access-rkgxc" (OuterVolumeSpecName: "kube-api-access-rkgxc") pod "4437923b-e05b-4d87-896e-7cc6f9b61b7b" (UID: "4437923b-e05b-4d87-896e-7cc6f9b61b7b"). InnerVolumeSpecName "kube-api-access-rkgxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:36.497094 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.497075    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4437923b-e05b-4d87-896e-7cc6f9b61b7b" (UID: "4437923b-e05b-4d87-896e-7cc6f9b61b7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:46:36.594895 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.594874    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-envoy\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:36.594895 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.594897    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-podinfo\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:36.595035 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.594910    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkgxc\" (UniqueName: \"kubernetes.io/projected/4437923b-e05b-4d87-896e-7cc6f9b61b7b-kube-api-access-rkgxc\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:36.595035 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:36.594921    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4437923b-e05b-4d87-896e-7cc6f9b61b7b-istio-token\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:46:37.106180 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:37.106145    2584 generic.go:356] "Generic (PLEG): container finished" podID="4437923b-e05b-4d87-896e-7cc6f9b61b7b" containerID="d8ff5fac1c20f6c01377f14fa47337ab6a8e47bbd65061464fd7ba2c833671f1" exitCode=0
Sep 15 21:46:37.106401 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:37.106206    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"
Sep 15 21:46:37.106401 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:37.106215    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p" event={"ID":"4437923b-e05b-4d87-896e-7cc6f9b61b7b","Type":"ContainerDied","Data":"d8ff5fac1c20f6c01377f14fa47337ab6a8e47bbd65061464fd7ba2c833671f1"}
Sep 15 21:46:37.106401 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:37.106255    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p" event={"ID":"4437923b-e05b-4d87-896e-7cc6f9b61b7b","Type":"ContainerDied","Data":"f9a15d04c43519e9893d88ba56b34e769900a737d0deefc6807f86310495d0f0"}
Sep 15 21:46:37.106401 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:37.106275    2584 scope.go:117] "RemoveContainer" containerID="d8ff5fac1c20f6c01377f14fa47337ab6a8e47bbd65061464fd7ba2c833671f1"
Sep 15 21:46:37.115588 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:37.115577    2584 scope.go:117] "RemoveContainer" containerID="cc7bbf23d9adf946df284a427adcb78b49a7d9ee700f320ef59f0ceed11fc9b9"
Sep 15 21:46:37.123547 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:37.123525    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"]
Sep 15 21:46:37.124215 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:37.124196    2584 scope.go:117] "RemoveContainer" containerID="d8ff5fac1c20f6c01377f14fa47337ab6a8e47bbd65061464fd7ba2c833671f1"
Sep 15 21:46:37.124499 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:46:37.124482    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ff5fac1c20f6c01377f14fa47337ab6a8e47bbd65061464fd7ba2c833671f1\": container with ID starting with d8ff5fac1c20f6c01377f14fa47337ab6a8e47bbd65061464fd7ba2c833671f1 not found: ID does not exist" containerID="d8ff5fac1c20f6c01377f14fa47337ab6a8e47bbd65061464fd7ba2c833671f1"
Sep 15 21:46:37.124588 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:37.124509    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ff5fac1c20f6c01377f14fa47337ab6a8e47bbd65061464fd7ba2c833671f1"} err="failed to get container status \"d8ff5fac1c20f6c01377f14fa47337ab6a8e47bbd65061464fd7ba2c833671f1\": rpc error: code = NotFound desc = could not find container \"d8ff5fac1c20f6c01377f14fa47337ab6a8e47bbd65061464fd7ba2c833671f1\": container with ID starting with d8ff5fac1c20f6c01377f14fa47337ab6a8e47bbd65061464fd7ba2c833671f1 not found: ID does not exist"
Sep 15 21:46:37.124588 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:37.124533    2584 scope.go:117] "RemoveContainer" containerID="cc7bbf23d9adf946df284a427adcb78b49a7d9ee700f320ef59f0ceed11fc9b9"
Sep 15 21:46:37.124859 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:46:37.124834    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7bbf23d9adf946df284a427adcb78b49a7d9ee700f320ef59f0ceed11fc9b9\": container with ID starting with cc7bbf23d9adf946df284a427adcb78b49a7d9ee700f320ef59f0ceed11fc9b9 not found: ID does not exist" containerID="cc7bbf23d9adf946df284a427adcb78b49a7d9ee700f320ef59f0ceed11fc9b9"
Sep 15 21:46:37.124958 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:37.124872    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7bbf23d9adf946df284a427adcb78b49a7d9ee700f320ef59f0ceed11fc9b9"} err="failed to get container status \"cc7bbf23d9adf946df284a427adcb78b49a7d9ee700f320ef59f0ceed11fc9b9\": rpc error: code = NotFound desc = could not find container \"cc7bbf23d9adf946df284a427adcb78b49a7d9ee700f320ef59f0ceed11fc9b9\": container with ID starting with cc7bbf23d9adf946df284a427adcb78b49a7d9ee700f320ef59f0ceed11fc9b9 not found: ID does not exist"
Sep 15 21:46:37.127370 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:37.127349    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-745dff-openshift-default-6c5d68bb6-gxm2p"]
Sep 15 21:46:38.672364 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:46:38.672330    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4437923b-e05b-4d87-896e-7cc6f9b61b7b" path="/var/lib/kubelet/pods/4437923b-e05b-4d87-896e-7cc6f9b61b7b/volumes"
Sep 15 21:47:35.000041 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.000009    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74f4d48d8b-ct979"]
Sep 15 21:47:35.000593 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.000573    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38af9c5e-bb64-45f1-9d41-12a766cef2a3" containerName="revoke-keys"
Sep 15 21:47:35.000663 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.000595    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="38af9c5e-bb64-45f1-9d41-12a766cef2a3" containerName="revoke-keys"
Sep 15 21:47:35.000663 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.000627    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4437923b-e05b-4d87-896e-7cc6f9b61b7b" containerName="istio-proxy"
Sep 15 21:47:35.000663 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.000632    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4437923b-e05b-4d87-896e-7cc6f9b61b7b" containerName="istio-proxy"
Sep 15 21:47:35.000663 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.000642    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4437923b-e05b-4d87-896e-7cc6f9b61b7b" containerName="istio-proxy"
Sep 15 21:47:35.000663 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.000648    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4437923b-e05b-4d87-896e-7cc6f9b61b7b" containerName="istio-proxy"
Sep 15 21:47:35.000663 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.000656    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b33ab9d-8cf7-4c02-b455-f6f604998bef" containerName="payload-processing"
Sep 15 21:47:35.000663 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.000661    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b33ab9d-8cf7-4c02-b455-f6f604998bef" containerName="payload-processing"
Sep 15 21:47:35.000886 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.000716    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4437923b-e05b-4d87-896e-7cc6f9b61b7b" containerName="istio-proxy"
Sep 15 21:47:35.000886 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.000724    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="59ce2e8f-0198-4399-b52f-a965a5b2c12d" containerName="istio-proxy"
Sep 15 21:47:35.000886 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.000731    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="38af9c5e-bb64-45f1-9d41-12a766cef2a3" containerName="revoke-keys"
Sep 15 21:47:35.000886 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.000741    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b33ab9d-8cf7-4c02-b455-f6f604998bef" containerName="payload-processing"
Sep 15 21:47:35.004870 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.004854    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74f4d48d8b-ct979"
Sep 15 21:47:35.007923 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.007901    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-96fm8\""
Sep 15 21:47:35.008033 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.007927    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 21:47:35.008033 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.007902    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 21:47:35.008033 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.007902    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 21:47:35.013597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.013575    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74f4d48d8b-ct979"]
Sep 15 21:47:35.094337 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.094314    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmxbp\" (UniqueName: \"kubernetes.io/projected/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-kube-api-access-nmxbp\") pod \"maas-controller-74f4d48d8b-ct979\" (UID: \"543f6213-a0a9-49c6-97ed-d9ec0b48a54f\") " pod="opendatahub/maas-controller-74f4d48d8b-ct979"
Sep 15 21:47:35.094470 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.094348    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-metrics-certs\") pod \"maas-controller-74f4d48d8b-ct979\" (UID: \"543f6213-a0a9-49c6-97ed-d9ec0b48a54f\") " pod="opendatahub/maas-controller-74f4d48d8b-ct979"
Sep 15 21:47:35.094470 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.094437    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-webhook-cert\") pod \"maas-controller-74f4d48d8b-ct979\" (UID: \"543f6213-a0a9-49c6-97ed-d9ec0b48a54f\") " pod="opendatahub/maas-controller-74f4d48d8b-ct979"
Sep 15 21:47:35.194997 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.194973    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmxbp\" (UniqueName: \"kubernetes.io/projected/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-kube-api-access-nmxbp\") pod \"maas-controller-74f4d48d8b-ct979\" (UID: \"543f6213-a0a9-49c6-97ed-d9ec0b48a54f\") " pod="opendatahub/maas-controller-74f4d48d8b-ct979"
Sep 15 21:47:35.195114 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.195005    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-metrics-certs\") pod \"maas-controller-74f4d48d8b-ct979\" (UID: \"543f6213-a0a9-49c6-97ed-d9ec0b48a54f\") " pod="opendatahub/maas-controller-74f4d48d8b-ct979"
Sep 15 21:47:35.195114 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.195056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-webhook-cert\") pod \"maas-controller-74f4d48d8b-ct979\" (UID: \"543f6213-a0a9-49c6-97ed-d9ec0b48a54f\") " pod="opendatahub/maas-controller-74f4d48d8b-ct979"
Sep 15 21:47:35.197507 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.197485    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-webhook-cert\") pod \"maas-controller-74f4d48d8b-ct979\" (UID: \"543f6213-a0a9-49c6-97ed-d9ec0b48a54f\") " pod="opendatahub/maas-controller-74f4d48d8b-ct979"
Sep 15 21:47:35.197636 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.197617    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-metrics-certs\") pod \"maas-controller-74f4d48d8b-ct979\" (UID: \"543f6213-a0a9-49c6-97ed-d9ec0b48a54f\") " pod="opendatahub/maas-controller-74f4d48d8b-ct979"
Sep 15 21:47:35.202491 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.202473    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmxbp\" (UniqueName: \"kubernetes.io/projected/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-kube-api-access-nmxbp\") pod \"maas-controller-74f4d48d8b-ct979\" (UID: \"543f6213-a0a9-49c6-97ed-d9ec0b48a54f\") " pod="opendatahub/maas-controller-74f4d48d8b-ct979"
Sep 15 21:47:35.316424 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.316396    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74f4d48d8b-ct979"
Sep 15 21:47:35.587096 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.587072    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74f4d48d8b-ct979"]
Sep 15 21:47:35.589128 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:47:35.589093    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543f6213_a0a9_49c6_97ed_d9ec0b48a54f.slice/crio-1fb1539f1566da8108048735fa49f816c671fa486c14a4e1f6e054b52df9b5ad WatchSource:0}: Error finding container 1fb1539f1566da8108048735fa49f816c671fa486c14a4e1f6e054b52df9b5ad: Status 404 returned error can't find the container with id 1fb1539f1566da8108048735fa49f816c671fa486c14a4e1f6e054b52df9b5ad
Sep 15 21:47:35.590533 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:35.590512    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:47:36.422788 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:36.422755    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74f4d48d8b-ct979" event={"ID":"543f6213-a0a9-49c6-97ed-d9ec0b48a54f","Type":"ContainerStarted","Data":"1fb1539f1566da8108048735fa49f816c671fa486c14a4e1f6e054b52df9b5ad"}
Sep 15 21:47:38.437216 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:38.437172    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74f4d48d8b-ct979" event={"ID":"543f6213-a0a9-49c6-97ed-d9ec0b48a54f","Type":"ContainerStarted","Data":"bc0760808ed5e743e266d896c23fea13dc7ca1ca81c75dc6e4e2c9e87f083f1b"}
Sep 15 21:47:38.437609 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:38.437265    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74f4d48d8b-ct979"
Sep 15 21:47:38.453796 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:38.453716    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-74f4d48d8b-ct979" podStartSLOduration=1.842259611 podStartE2EDuration="4.453702397s" totalImagesPullingTime="2.611442786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:47:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:47:38.452794741 +0000 UTC m=+2226.314728543" watchObservedRunningTime="2026-09-15 21:47:38.453702397 +0000 UTC m=+2226.315636200"
Sep 15 21:47:49.444509 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:47:49.444477    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74f4d48d8b-ct979"
Sep 15 21:48:00.855580 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:00.855362    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74f4d48d8b-ct979"]
Sep 15 21:48:00.856241 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:00.856199    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-74f4d48d8b-ct979" podUID="543f6213-a0a9-49c6-97ed-d9ec0b48a54f" containerName="manager" containerID="cri-o://bc0760808ed5e743e266d896c23fea13dc7ca1ca81c75dc6e4e2c9e87f083f1b" gracePeriod=30
Sep 15 21:48:01.129935 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.129914    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74f4d48d8b-ct979"
Sep 15 21:48:01.218311 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.218278    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmxbp\" (UniqueName: \"kubernetes.io/projected/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-kube-api-access-nmxbp\") pod \"543f6213-a0a9-49c6-97ed-d9ec0b48a54f\" (UID: \"543f6213-a0a9-49c6-97ed-d9ec0b48a54f\") "
Sep 15 21:48:01.218526 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.218346    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-metrics-certs\") pod \"543f6213-a0a9-49c6-97ed-d9ec0b48a54f\" (UID: \"543f6213-a0a9-49c6-97ed-d9ec0b48a54f\") "
Sep 15 21:48:01.218526 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.218391    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-webhook-cert\") pod \"543f6213-a0a9-49c6-97ed-d9ec0b48a54f\" (UID: \"543f6213-a0a9-49c6-97ed-d9ec0b48a54f\") "
Sep 15 21:48:01.220860 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.220834    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-kube-api-access-nmxbp" (OuterVolumeSpecName: "kube-api-access-nmxbp") pod "543f6213-a0a9-49c6-97ed-d9ec0b48a54f" (UID: "543f6213-a0a9-49c6-97ed-d9ec0b48a54f"). InnerVolumeSpecName "kube-api-access-nmxbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:48:01.220975 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.220887    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "543f6213-a0a9-49c6-97ed-d9ec0b48a54f" (UID: "543f6213-a0a9-49c6-97ed-d9ec0b48a54f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:48:01.220975 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.220914    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "543f6213-a0a9-49c6-97ed-d9ec0b48a54f" (UID: "543f6213-a0a9-49c6-97ed-d9ec0b48a54f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:48:01.319237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.319207    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmxbp\" (UniqueName: \"kubernetes.io/projected/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-kube-api-access-nmxbp\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:48:01.319237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.319233    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-metrics-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:48:01.319237 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.319243    2584 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/543f6213-a0a9-49c6-97ed-d9ec0b48a54f-webhook-cert\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:48:01.564280 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.564247    2584 generic.go:356] "Generic (PLEG): container finished" podID="543f6213-a0a9-49c6-97ed-d9ec0b48a54f" containerID="bc0760808ed5e743e266d896c23fea13dc7ca1ca81c75dc6e4e2c9e87f083f1b" exitCode=0
Sep 15 21:48:01.564501 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.564304    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74f4d48d8b-ct979" event={"ID":"543f6213-a0a9-49c6-97ed-d9ec0b48a54f","Type":"ContainerDied","Data":"bc0760808ed5e743e266d896c23fea13dc7ca1ca81c75dc6e4e2c9e87f083f1b"}
Sep 15 21:48:01.564501 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.564364    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74f4d48d8b-ct979" event={"ID":"543f6213-a0a9-49c6-97ed-d9ec0b48a54f","Type":"ContainerDied","Data":"1fb1539f1566da8108048735fa49f816c671fa486c14a4e1f6e054b52df9b5ad"}
Sep 15 21:48:01.564501 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.564304    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74f4d48d8b-ct979"
Sep 15 21:48:01.564501 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.564409    2584 scope.go:117] "RemoveContainer" containerID="bc0760808ed5e743e266d896c23fea13dc7ca1ca81c75dc6e4e2c9e87f083f1b"
Sep 15 21:48:01.576061 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.576038    2584 scope.go:117] "RemoveContainer" containerID="bc0760808ed5e743e266d896c23fea13dc7ca1ca81c75dc6e4e2c9e87f083f1b"
Sep 15 21:48:01.576519 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:48:01.576496    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0760808ed5e743e266d896c23fea13dc7ca1ca81c75dc6e4e2c9e87f083f1b\": container with ID starting with bc0760808ed5e743e266d896c23fea13dc7ca1ca81c75dc6e4e2c9e87f083f1b not found: ID does not exist" containerID="bc0760808ed5e743e266d896c23fea13dc7ca1ca81c75dc6e4e2c9e87f083f1b"
Sep 15 21:48:01.576637 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.576525    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0760808ed5e743e266d896c23fea13dc7ca1ca81c75dc6e4e2c9e87f083f1b"} err="failed to get container status \"bc0760808ed5e743e266d896c23fea13dc7ca1ca81c75dc6e4e2c9e87f083f1b\": rpc error: code = NotFound desc = could not find container \"bc0760808ed5e743e266d896c23fea13dc7ca1ca81c75dc6e4e2c9e87f083f1b\": container with ID starting with bc0760808ed5e743e266d896c23fea13dc7ca1ca81c75dc6e4e2c9e87f083f1b not found: ID does not exist"
Sep 15 21:48:01.586659 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.586639    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74f4d48d8b-ct979"]
Sep 15 21:48:01.590485 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:01.590466    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-74f4d48d8b-ct979"]
Sep 15 21:48:02.418675 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.418644    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74f4d48d8b-qfxl5"]
Sep 15 21:48:02.419080 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.419048    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="543f6213-a0a9-49c6-97ed-d9ec0b48a54f" containerName="manager"
Sep 15 21:48:02.419080 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.419061    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="543f6213-a0a9-49c6-97ed-d9ec0b48a54f" containerName="manager"
Sep 15 21:48:02.419215 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.419118    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4437923b-e05b-4d87-896e-7cc6f9b61b7b" containerName="istio-proxy"
Sep 15 21:48:02.419215 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.419127    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="543f6213-a0a9-49c6-97ed-d9ec0b48a54f" containerName="manager"
Sep 15 21:48:02.423574 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.423556    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74f4d48d8b-qfxl5"
Sep 15 21:48:02.426481 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.426457    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 21:48:02.426583 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.426485    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 21:48:02.426583 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.426532    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-96fm8\""
Sep 15 21:48:02.426583 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.426561    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 21:48:02.432444 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.432423    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74f4d48d8b-qfxl5"]
Sep 15 21:48:02.527905 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.527877    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a55a8124-0a35-4e43-baec-4f953c2f5cf6-webhook-cert\") pod \"maas-controller-74f4d48d8b-qfxl5\" (UID: \"a55a8124-0a35-4e43-baec-4f953c2f5cf6\") " pod="opendatahub/maas-controller-74f4d48d8b-qfxl5"
Sep 15 21:48:02.527905 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.527904    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc2vl\" (UniqueName: \"kubernetes.io/projected/a55a8124-0a35-4e43-baec-4f953c2f5cf6-kube-api-access-tc2vl\") pod \"maas-controller-74f4d48d8b-qfxl5\" (UID: \"a55a8124-0a35-4e43-baec-4f953c2f5cf6\") " pod="opendatahub/maas-controller-74f4d48d8b-qfxl5"
Sep 15 21:48:02.528111 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.527974    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a55a8124-0a35-4e43-baec-4f953c2f5cf6-metrics-certs\") pod \"maas-controller-74f4d48d8b-qfxl5\" (UID: \"a55a8124-0a35-4e43-baec-4f953c2f5cf6\") " pod="opendatahub/maas-controller-74f4d48d8b-qfxl5"
Sep 15 21:48:02.629143 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.629120    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a55a8124-0a35-4e43-baec-4f953c2f5cf6-webhook-cert\") pod \"maas-controller-74f4d48d8b-qfxl5\" (UID: \"a55a8124-0a35-4e43-baec-4f953c2f5cf6\") " pod="opendatahub/maas-controller-74f4d48d8b-qfxl5"
Sep 15 21:48:02.629253 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.629151    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc2vl\" (UniqueName: \"kubernetes.io/projected/a55a8124-0a35-4e43-baec-4f953c2f5cf6-kube-api-access-tc2vl\") pod \"maas-controller-74f4d48d8b-qfxl5\" (UID: \"a55a8124-0a35-4e43-baec-4f953c2f5cf6\") " pod="opendatahub/maas-controller-74f4d48d8b-qfxl5"
Sep 15 21:48:02.629253 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.629193    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a55a8124-0a35-4e43-baec-4f953c2f5cf6-metrics-certs\") pod \"maas-controller-74f4d48d8b-qfxl5\" (UID: \"a55a8124-0a35-4e43-baec-4f953c2f5cf6\") " pod="opendatahub/maas-controller-74f4d48d8b-qfxl5"
Sep 15 21:48:02.631650 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.631629    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a55a8124-0a35-4e43-baec-4f953c2f5cf6-metrics-certs\") pod \"maas-controller-74f4d48d8b-qfxl5\" (UID: \"a55a8124-0a35-4e43-baec-4f953c2f5cf6\") " pod="opendatahub/maas-controller-74f4d48d8b-qfxl5"
Sep 15 21:48:02.631744 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.631668    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a55a8124-0a35-4e43-baec-4f953c2f5cf6-webhook-cert\") pod \"maas-controller-74f4d48d8b-qfxl5\" (UID: \"a55a8124-0a35-4e43-baec-4f953c2f5cf6\") " pod="opendatahub/maas-controller-74f4d48d8b-qfxl5"
Sep 15 21:48:02.637103 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.637081    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc2vl\" (UniqueName: \"kubernetes.io/projected/a55a8124-0a35-4e43-baec-4f953c2f5cf6-kube-api-access-tc2vl\") pod \"maas-controller-74f4d48d8b-qfxl5\" (UID: \"a55a8124-0a35-4e43-baec-4f953c2f5cf6\") " pod="opendatahub/maas-controller-74f4d48d8b-qfxl5"
Sep 15 21:48:02.672256 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.672197    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="543f6213-a0a9-49c6-97ed-d9ec0b48a54f" path="/var/lib/kubelet/pods/543f6213-a0a9-49c6-97ed-d9ec0b48a54f/volumes"
Sep 15 21:48:02.735681 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:02.735659    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74f4d48d8b-qfxl5"
Sep 15 21:48:03.010960 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:03.010939    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74f4d48d8b-qfxl5"]
Sep 15 21:48:03.012656 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:48:03.012627    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55a8124_0a35_4e43_baec_4f953c2f5cf6.slice/crio-35e27ebd70d4a61e1d2bf4b023809c4b98889df8bc538248f957a2c36ea1e4c9 WatchSource:0}: Error finding container 35e27ebd70d4a61e1d2bf4b023809c4b98889df8bc538248f957a2c36ea1e4c9: Status 404 returned error can't find the container with id 35e27ebd70d4a61e1d2bf4b023809c4b98889df8bc538248f957a2c36ea1e4c9
Sep 15 21:48:03.579171 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:03.579140    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74f4d48d8b-qfxl5" event={"ID":"a55a8124-0a35-4e43-baec-4f953c2f5cf6","Type":"ContainerStarted","Data":"8c322d35f2623a3ad00586d1e410a7996482b9247cd66b161a65688899cbc8ea"}
Sep 15 21:48:03.579579 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:03.579196    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74f4d48d8b-qfxl5" event={"ID":"a55a8124-0a35-4e43-baec-4f953c2f5cf6","Type":"ContainerStarted","Data":"35e27ebd70d4a61e1d2bf4b023809c4b98889df8bc538248f957a2c36ea1e4c9"}
Sep 15 21:48:03.579579 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:03.579294    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74f4d48d8b-qfxl5"
Sep 15 21:48:03.596205 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:03.596155    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-74f4d48d8b-qfxl5" podStartSLOduration=1.158276266 podStartE2EDuration="1.596140859s" totalImagesPullingTime="437.864593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:48:03.594505816 +0000 UTC m=+2251.456439619" watchObservedRunningTime="2026-09-15 21:48:03.596140859 +0000 UTC m=+2251.458074663"
Sep 15 21:48:14.587533 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:48:14.587498    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74f4d48d8b-qfxl5"
Sep 15 21:49:16.243933 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.243888    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"]
Sep 15 21:49:16.244463 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.244213    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg" podUID="fe5a30ee-14a2-4dec-a16b-2a44bd005c6b" containerName="manager" containerID="cri-o://9df2f00b2b99d5bc3c8b2d1fcb12e67320925357e691a67676ac6e97e2f06fa1" gracePeriod=10
Sep 15 21:49:16.700169 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.700146    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"
Sep 15 21:49:16.797153 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.797117    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92q6h\" (UniqueName: \"kubernetes.io/projected/fe5a30ee-14a2-4dec-a16b-2a44bd005c6b-kube-api-access-92q6h\") pod \"fe5a30ee-14a2-4dec-a16b-2a44bd005c6b\" (UID: \"fe5a30ee-14a2-4dec-a16b-2a44bd005c6b\") "
Sep 15 21:49:16.797325 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.797176    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe5a30ee-14a2-4dec-a16b-2a44bd005c6b-extensions-socket-volume\") pod \"fe5a30ee-14a2-4dec-a16b-2a44bd005c6b\" (UID: \"fe5a30ee-14a2-4dec-a16b-2a44bd005c6b\") "
Sep 15 21:49:16.797577 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.797554    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5a30ee-14a2-4dec-a16b-2a44bd005c6b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fe5a30ee-14a2-4dec-a16b-2a44bd005c6b" (UID: "fe5a30ee-14a2-4dec-a16b-2a44bd005c6b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:49:16.799070 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.799044    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5a30ee-14a2-4dec-a16b-2a44bd005c6b-kube-api-access-92q6h" (OuterVolumeSpecName: "kube-api-access-92q6h") pod "fe5a30ee-14a2-4dec-a16b-2a44bd005c6b" (UID: "fe5a30ee-14a2-4dec-a16b-2a44bd005c6b"). InnerVolumeSpecName "kube-api-access-92q6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:49:16.898460 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.898397    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92q6h\" (UniqueName: \"kubernetes.io/projected/fe5a30ee-14a2-4dec-a16b-2a44bd005c6b-kube-api-access-92q6h\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:49:16.898460 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.898423    2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe5a30ee-14a2-4dec-a16b-2a44bd005c6b-extensions-socket-volume\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:49:16.971229 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.971203    2584 generic.go:356] "Generic (PLEG): container finished" podID="fe5a30ee-14a2-4dec-a16b-2a44bd005c6b" containerID="9df2f00b2b99d5bc3c8b2d1fcb12e67320925357e691a67676ac6e97e2f06fa1" exitCode=0
Sep 15 21:49:16.971358 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.971247    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"
Sep 15 21:49:16.971358 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.971288    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg" event={"ID":"fe5a30ee-14a2-4dec-a16b-2a44bd005c6b","Type":"ContainerDied","Data":"9df2f00b2b99d5bc3c8b2d1fcb12e67320925357e691a67676ac6e97e2f06fa1"}
Sep 15 21:49:16.971358 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.971338    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg" event={"ID":"fe5a30ee-14a2-4dec-a16b-2a44bd005c6b","Type":"ContainerDied","Data":"c76547ff3b1ab85d864ceb08cb4448219cd309e560d8f8cc1360cc7f1f9f45b3"}
Sep 15 21:49:16.971504 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.971363    2584 scope.go:117] "RemoveContainer" containerID="9df2f00b2b99d5bc3c8b2d1fcb12e67320925357e691a67676ac6e97e2f06fa1"
Sep 15 21:49:16.981051 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.981036    2584 scope.go:117] "RemoveContainer" containerID="5acdc19d015248d4a85932fa6d46c5088cad6b7f1e9f9c373cdba0d220b9c7ce"
Sep 15 21:49:16.989031 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.989014    2584 scope.go:117] "RemoveContainer" containerID="9df2f00b2b99d5bc3c8b2d1fcb12e67320925357e691a67676ac6e97e2f06fa1"
Sep 15 21:49:16.989292 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:49:16.989274    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df2f00b2b99d5bc3c8b2d1fcb12e67320925357e691a67676ac6e97e2f06fa1\": container with ID starting with 9df2f00b2b99d5bc3c8b2d1fcb12e67320925357e691a67676ac6e97e2f06fa1 not found: ID does not exist" containerID="9df2f00b2b99d5bc3c8b2d1fcb12e67320925357e691a67676ac6e97e2f06fa1"
Sep 15 21:49:16.989337 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.989300    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df2f00b2b99d5bc3c8b2d1fcb12e67320925357e691a67676ac6e97e2f06fa1"} err="failed to get container status \"9df2f00b2b99d5bc3c8b2d1fcb12e67320925357e691a67676ac6e97e2f06fa1\": rpc error: code = NotFound desc = could not find container \"9df2f00b2b99d5bc3c8b2d1fcb12e67320925357e691a67676ac6e97e2f06fa1\": container with ID starting with 9df2f00b2b99d5bc3c8b2d1fcb12e67320925357e691a67676ac6e97e2f06fa1 not found: ID does not exist"
Sep 15 21:49:16.989337 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.989315    2584 scope.go:117] "RemoveContainer" containerID="5acdc19d015248d4a85932fa6d46c5088cad6b7f1e9f9c373cdba0d220b9c7ce"
Sep 15 21:49:16.989574 ip-10-0-139-184 kubenswrapper[2584]: E0915 21:49:16.989558    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5acdc19d015248d4a85932fa6d46c5088cad6b7f1e9f9c373cdba0d220b9c7ce\": container with ID starting with 5acdc19d015248d4a85932fa6d46c5088cad6b7f1e9f9c373cdba0d220b9c7ce not found: ID does not exist" containerID="5acdc19d015248d4a85932fa6d46c5088cad6b7f1e9f9c373cdba0d220b9c7ce"
Sep 15 21:49:16.989613 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.989579    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5acdc19d015248d4a85932fa6d46c5088cad6b7f1e9f9c373cdba0d220b9c7ce"} err="failed to get container status \"5acdc19d015248d4a85932fa6d46c5088cad6b7f1e9f9c373cdba0d220b9c7ce\": rpc error: code = NotFound desc = could not find container \"5acdc19d015248d4a85932fa6d46c5088cad6b7f1e9f9c373cdba0d220b9c7ce\": container with ID starting with 5acdc19d015248d4a85932fa6d46c5088cad6b7f1e9f9c373cdba0d220b9c7ce not found: ID does not exist"
Sep 15 21:49:16.993014 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.992994    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"]
Sep 15 21:49:16.999918 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:16.999900    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gjvxg"]
Sep 15 21:49:18.672566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:49:18.672533    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe5a30ee-14a2-4dec-a16b-2a44bd005c6b" path="/var/lib/kubelet/pods/fe5a30ee-14a2-4dec-a16b-2a44bd005c6b/volumes"
Sep 15 21:50:16.972688 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:16.972658    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swjzd"]
Sep 15 21:50:16.973117 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:16.973064    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5a30ee-14a2-4dec-a16b-2a44bd005c6b" containerName="manager"
Sep 15 21:50:16.973117 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:16.973077    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5a30ee-14a2-4dec-a16b-2a44bd005c6b" containerName="manager"
Sep 15 21:50:16.973117 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:16.973089    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5a30ee-14a2-4dec-a16b-2a44bd005c6b" containerName="manager"
Sep 15 21:50:16.973117 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:16.973097    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5a30ee-14a2-4dec-a16b-2a44bd005c6b" containerName="manager"
Sep 15 21:50:16.973254 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:16.973172    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5a30ee-14a2-4dec-a16b-2a44bd005c6b" containerName="manager"
Sep 15 21:50:16.973254 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:16.973182    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5a30ee-14a2-4dec-a16b-2a44bd005c6b" containerName="manager"
Sep 15 21:50:16.976280 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:16.976264    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swjzd"
Sep 15 21:50:16.978373 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:16.978354    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6z224\""
Sep 15 21:50:16.990283 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:16.990265    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swjzd"]
Sep 15 21:50:17.030355 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:17.030330    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x86lb\" (UniqueName: \"kubernetes.io/projected/58b68082-d855-44f6-a3dc-6b81b1467da5-kube-api-access-x86lb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-swjzd\" (UID: \"58b68082-d855-44f6-a3dc-6b81b1467da5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swjzd"
Sep 15 21:50:17.030498 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:17.030401    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/58b68082-d855-44f6-a3dc-6b81b1467da5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-swjzd\" (UID: \"58b68082-d855-44f6-a3dc-6b81b1467da5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swjzd"
Sep 15 21:50:17.131757 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:17.131721    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x86lb\" (UniqueName: \"kubernetes.io/projected/58b68082-d855-44f6-a3dc-6b81b1467da5-kube-api-access-x86lb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-swjzd\" (UID: \"58b68082-d855-44f6-a3dc-6b81b1467da5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swjzd"
Sep 15 21:50:17.131967 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:17.131816    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/58b68082-d855-44f6-a3dc-6b81b1467da5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-swjzd\" (UID: \"58b68082-d855-44f6-a3dc-6b81b1467da5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swjzd"
Sep 15 21:50:17.132204 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:17.132187    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/58b68082-d855-44f6-a3dc-6b81b1467da5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-swjzd\" (UID: \"58b68082-d855-44f6-a3dc-6b81b1467da5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swjzd"
Sep 15 21:50:17.140856 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:17.140833    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x86lb\" (UniqueName: \"kubernetes.io/projected/58b68082-d855-44f6-a3dc-6b81b1467da5-kube-api-access-x86lb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-swjzd\" (UID: \"58b68082-d855-44f6-a3dc-6b81b1467da5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swjzd"
Sep 15 21:50:17.285637 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:17.285617    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swjzd"
Sep 15 21:50:17.565312 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:17.565282    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swjzd"]
Sep 15 21:50:17.566836 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:50:17.566806    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b68082_d855_44f6_a3dc_6b81b1467da5.slice/crio-d8e037386fd5dfb3855443dcef6a590aa5bfc3be45e6c4a989b05559ab96e3a0 WatchSource:0}: Error finding container d8e037386fd5dfb3855443dcef6a590aa5bfc3be45e6c4a989b05559ab96e3a0: Status 404 returned error can't find the container with id d8e037386fd5dfb3855443dcef6a590aa5bfc3be45e6c4a989b05559ab96e3a0
Sep 15 21:50:18.302517 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:18.302474    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swjzd" event={"ID":"58b68082-d855-44f6-a3dc-6b81b1467da5","Type":"ContainerStarted","Data":"d8a0604083a919ade55293149ee32370ff69b9d1db4687b4b4346cb9f80d2609"}
Sep 15 21:50:18.302901 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:18.302543    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swjzd"
Sep 15 21:50:18.302901 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:18.302562    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swjzd" event={"ID":"58b68082-d855-44f6-a3dc-6b81b1467da5","Type":"ContainerStarted","Data":"d8e037386fd5dfb3855443dcef6a590aa5bfc3be45e6c4a989b05559ab96e3a0"}
Sep 15 21:50:18.320897 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:18.320846    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swjzd" podStartSLOduration=2.320829508 podStartE2EDuration="2.320829508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:50:18.319396273 +0000 UTC m=+2386.181330068" watchObservedRunningTime="2026-09-15 21:50:18.320829508 +0000 UTC m=+2386.182763315"
Sep 15 21:50:29.310493 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:29.310462    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swjzd"
Sep 15 21:50:32.802045 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:32.802012    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:50:32.806138 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:32.806115    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:50:33.317631 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:33.317605    2584 scope.go:117] "RemoveContainer" containerID="683053e5d2fc843b5f5a283c5d455610a6f31711e7224682a8fcaad6eaf9d9f9"
Sep 15 21:50:53.206411 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.206362    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"]
Sep 15 21:50:53.210319 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.210294    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.212562 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.212519    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f74dbafa-kserve-self-signed-certs\""
Sep 15 21:50:53.218110 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.217910    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"]
Sep 15 21:50:53.348552 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.348511    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndwwl\" (UniqueName: \"kubernetes.io/projected/b79ca406-2c5a-4612-b892-43fcad8bde5f-kube-api-access-ndwwl\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.348744 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.348568    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-home\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.348744 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.348628    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b79ca406-2c5a-4612-b892-43fcad8bde5f-tls-certs\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.348744 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.348678    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-model-cache\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.348744 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.348734    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-dshm\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.348916 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.348781    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-tmp-dir\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.450362 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.450328    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwwl\" (UniqueName: \"kubernetes.io/projected/b79ca406-2c5a-4612-b892-43fcad8bde5f-kube-api-access-ndwwl\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.450549 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.450388    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-home\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.450549 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.450440    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b79ca406-2c5a-4612-b892-43fcad8bde5f-tls-certs\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.450549 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.450484    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-model-cache\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.450549 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.450530    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-dshm\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.450784 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.450557    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-tmp-dir\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.451027 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.450837    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-home\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.451126 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.450873    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-model-cache\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.451126 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.451061    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-tmp-dir\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.452831 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.452807    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-dshm\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.453282 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.453260    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b79ca406-2c5a-4612-b892-43fcad8bde5f-tls-certs\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.457763 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.457718    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndwwl\" (UniqueName: \"kubernetes.io/projected/b79ca406-2c5a-4612-b892-43fcad8bde5f-kube-api-access-ndwwl\") pod \"e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") " pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.520889 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.520855    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:50:53.791864 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:53.791837    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"]
Sep 15 21:50:53.793403 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:50:53.793355    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb79ca406_2c5a_4612_b892_43fcad8bde5f.slice/crio-2cb7225b9cedb99eea9f2ec04af42c0e5fd6e913944b41c48a6ff67ee1d7c929 WatchSource:0}: Error finding container 2cb7225b9cedb99eea9f2ec04af42c0e5fd6e913944b41c48a6ff67ee1d7c929: Status 404 returned error can't find the container with id 2cb7225b9cedb99eea9f2ec04af42c0e5fd6e913944b41c48a6ff67ee1d7c929
Sep 15 21:50:54.503344 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:54.503308    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79" event={"ID":"b79ca406-2c5a-4612-b892-43fcad8bde5f","Type":"ContainerStarted","Data":"9537da8f7ec9c5a12ceb6fa4bfa19e7b925f2159db3ef597873496d1423c53d6"}
Sep 15 21:50:54.503344 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:54.503350    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79" event={"ID":"b79ca406-2c5a-4612-b892-43fcad8bde5f","Type":"ContainerStarted","Data":"2cb7225b9cedb99eea9f2ec04af42c0e5fd6e913944b41c48a6ff67ee1d7c929"}
Sep 15 21:50:54.518951 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:50:54.518903    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79" podStartSLOduration=1.518888419 podStartE2EDuration="1.518888419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:50:54.5181798 +0000 UTC m=+2422.380113604" watchObservedRunningTime="2026-09-15 21:50:54.518888419 +0000 UTC m=+2422.380822222"
Sep 15 21:51:03.521171 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:51:03.521084    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:51:03.521589 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:51:03.521194    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:51:03.533434 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:51:03.533410    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:51:03.561856 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:51:03.561829    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:51:33.340651 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:51:33.340619    2584 scope.go:117] "RemoveContainer" containerID="558ddc904931201016b9e438f9a59993a30809afc571f93fc84d31cd1b0fd5f9"
Sep 15 21:51:33.348746 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:51:33.348727    2584 scope.go:117] "RemoveContainer" containerID="80a0d47cbbf42d02d68873874c9b8309371e20ccf784afe40f7b3e3d5815528b"
Sep 15 21:51:33.355951 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:51:33.355926    2584 scope.go:117] "RemoveContainer" containerID="bda957c619952fd54c051bb144bd17dbb9e9c3b38ba4a6f2f33ba0e01828cd61"
Sep 15 21:51:48.681303 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:51:48.681271    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"]
Sep 15 21:52:03.700458 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.700414    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79" podUID="b79ca406-2c5a-4612-b892-43fcad8bde5f" containerName="main" containerID="cri-o://9537da8f7ec9c5a12ceb6fa4bfa19e7b925f2159db3ef597873496d1423c53d6" gracePeriod=45
Sep 15 21:52:03.878491 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.878400    2584 generic.go:356] "Generic (PLEG): container finished" podID="b79ca406-2c5a-4612-b892-43fcad8bde5f" containerID="9537da8f7ec9c5a12ceb6fa4bfa19e7b925f2159db3ef597873496d1423c53d6" exitCode=0
Sep 15 21:52:03.878491 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.878424    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79" event={"ID":"b79ca406-2c5a-4612-b892-43fcad8bde5f","Type":"ContainerDied","Data":"9537da8f7ec9c5a12ceb6fa4bfa19e7b925f2159db3ef597873496d1423c53d6"}
Sep 15 21:52:03.937952 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.937931    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:52:03.994406 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.994308    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-dshm\") pod \"b79ca406-2c5a-4612-b892-43fcad8bde5f\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") "
Sep 15 21:52:03.994406 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.994402    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-tmp-dir\") pod \"b79ca406-2c5a-4612-b892-43fcad8bde5f\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") "
Sep 15 21:52:03.994625 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.994468    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndwwl\" (UniqueName: \"kubernetes.io/projected/b79ca406-2c5a-4612-b892-43fcad8bde5f-kube-api-access-ndwwl\") pod \"b79ca406-2c5a-4612-b892-43fcad8bde5f\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") "
Sep 15 21:52:03.994625 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.994503    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-home\") pod \"b79ca406-2c5a-4612-b892-43fcad8bde5f\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") "
Sep 15 21:52:03.994625 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.994533    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-model-cache\") pod \"b79ca406-2c5a-4612-b892-43fcad8bde5f\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") "
Sep 15 21:52:03.994625 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.994576    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b79ca406-2c5a-4612-b892-43fcad8bde5f-tls-certs\") pod \"b79ca406-2c5a-4612-b892-43fcad8bde5f\" (UID: \"b79ca406-2c5a-4612-b892-43fcad8bde5f\") "
Sep 15 21:52:03.994846 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.994712    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b79ca406-2c5a-4612-b892-43fcad8bde5f" (UID: "b79ca406-2c5a-4612-b892-43fcad8bde5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:52:03.994846 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.994757    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-home" (OuterVolumeSpecName: "home") pod "b79ca406-2c5a-4612-b892-43fcad8bde5f" (UID: "b79ca406-2c5a-4612-b892-43fcad8bde5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:52:03.994846 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.994793    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b79ca406-2c5a-4612-b892-43fcad8bde5f" (UID: "b79ca406-2c5a-4612-b892-43fcad8bde5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:52:03.994999 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.994982    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-home\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:52:03.995035 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.995006    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-model-cache\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:52:03.995073 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.995041    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-tmp-dir\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:52:03.996645 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.996626    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-dshm" (OuterVolumeSpecName: "dshm") pod "b79ca406-2c5a-4612-b892-43fcad8bde5f" (UID: "b79ca406-2c5a-4612-b892-43fcad8bde5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:52:03.996720 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.996661    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b79ca406-2c5a-4612-b892-43fcad8bde5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b79ca406-2c5a-4612-b892-43fcad8bde5f" (UID: "b79ca406-2c5a-4612-b892-43fcad8bde5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:52:03.996806 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:03.996786    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79ca406-2c5a-4612-b892-43fcad8bde5f-kube-api-access-ndwwl" (OuterVolumeSpecName: "kube-api-access-ndwwl") pod "b79ca406-2c5a-4612-b892-43fcad8bde5f" (UID: "b79ca406-2c5a-4612-b892-43fcad8bde5f"). InnerVolumeSpecName "kube-api-access-ndwwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:52:04.095827 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:04.095788    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b79ca406-2c5a-4612-b892-43fcad8bde5f-tls-certs\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:52:04.095827 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:04.095815    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b79ca406-2c5a-4612-b892-43fcad8bde5f-dshm\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:52:04.095827 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:04.095824    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndwwl\" (UniqueName: \"kubernetes.io/projected/b79ca406-2c5a-4612-b892-43fcad8bde5f-kube-api-access-ndwwl\") on node \"ip-10-0-139-184.ec2.internal\" DevicePath \"\""
Sep 15 21:52:04.885105 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:04.885079    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"
Sep 15 21:52:04.885566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:04.885122    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79" event={"ID":"b79ca406-2c5a-4612-b892-43fcad8bde5f","Type":"ContainerDied","Data":"2cb7225b9cedb99eea9f2ec04af42c0e5fd6e913944b41c48a6ff67ee1d7c929"}
Sep 15 21:52:04.885566 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:04.885181    2584 scope.go:117] "RemoveContainer" containerID="9537da8f7ec9c5a12ceb6fa4bfa19e7b925f2159db3ef597873496d1423c53d6"
Sep 15 21:52:04.903287 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:04.903259    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"]
Sep 15 21:52:04.906785 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:04.906763    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f74dbafa-kserve-5b4d8548ff-z6r79"]
Sep 15 21:52:06.672223 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:06.672189    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b79ca406-2c5a-4612-b892-43fcad8bde5f" path="/var/lib/kubelet/pods/b79ca406-2c5a-4612-b892-43fcad8bde5f/volumes"
Sep 15 21:52:21.591910 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.591866    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"]
Sep 15 21:52:21.592604 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.592579    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b79ca406-2c5a-4612-b892-43fcad8bde5f" containerName="main"
Sep 15 21:52:21.592772 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.592606    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b79ca406-2c5a-4612-b892-43fcad8bde5f" containerName="main"
Sep 15 21:52:21.592772 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.592733    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b79ca406-2c5a-4612-b892-43fcad8bde5f" containerName="main"
Sep 15 21:52:21.595772 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.595758    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"
Sep 15 21:52:21.598611 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.598576    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 21:52:21.598611 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.598602    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4hmqz\""
Sep 15 21:52:21.598809 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.598579    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 21:52:21.598809 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.598583    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 21:52:21.598809 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.598576    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:52:21.604597 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.604574    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"]
Sep 15 21:52:21.751194 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.751167    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b97dd01b-1a98-4852-b87b-af845c7ae9b1-metrics-certs\") pod \"maas-api-775cbbbf65-vqm27\" (UID: \"b97dd01b-1a98-4852-b87b-af845c7ae9b1\") " pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"
Sep 15 21:52:21.751367 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.751211    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b97dd01b-1a98-4852-b87b-af845c7ae9b1-maas-api-tls\") pod \"maas-api-775cbbbf65-vqm27\" (UID: \"b97dd01b-1a98-4852-b87b-af845c7ae9b1\") " pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"
Sep 15 21:52:21.751367 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.751281    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zpzn\" (UniqueName: \"kubernetes.io/projected/b97dd01b-1a98-4852-b87b-af845c7ae9b1-kube-api-access-6zpzn\") pod \"maas-api-775cbbbf65-vqm27\" (UID: \"b97dd01b-1a98-4852-b87b-af845c7ae9b1\") " pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"
Sep 15 21:52:21.751367 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.751319    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b97dd01b-1a98-4852-b87b-af845c7ae9b1-openshift-service-ca\") pod \"maas-api-775cbbbf65-vqm27\" (UID: \"b97dd01b-1a98-4852-b87b-af845c7ae9b1\") " pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"
Sep 15 21:52:21.852509 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.852429    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b97dd01b-1a98-4852-b87b-af845c7ae9b1-metrics-certs\") pod \"maas-api-775cbbbf65-vqm27\" (UID: \"b97dd01b-1a98-4852-b87b-af845c7ae9b1\") " pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"
Sep 15 21:52:21.852509 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.852478    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b97dd01b-1a98-4852-b87b-af845c7ae9b1-maas-api-tls\") pod \"maas-api-775cbbbf65-vqm27\" (UID: \"b97dd01b-1a98-4852-b87b-af845c7ae9b1\") " pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"
Sep 15 21:52:21.852509 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.852502    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zpzn\" (UniqueName: \"kubernetes.io/projected/b97dd01b-1a98-4852-b87b-af845c7ae9b1-kube-api-access-6zpzn\") pod \"maas-api-775cbbbf65-vqm27\" (UID: \"b97dd01b-1a98-4852-b87b-af845c7ae9b1\") " pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"
Sep 15 21:52:21.852794 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.852611    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b97dd01b-1a98-4852-b87b-af845c7ae9b1-openshift-service-ca\") pod \"maas-api-775cbbbf65-vqm27\" (UID: \"b97dd01b-1a98-4852-b87b-af845c7ae9b1\") " pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"
Sep 15 21:52:21.853246 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.853227    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b97dd01b-1a98-4852-b87b-af845c7ae9b1-openshift-service-ca\") pod \"maas-api-775cbbbf65-vqm27\" (UID: \"b97dd01b-1a98-4852-b87b-af845c7ae9b1\") " pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"
Sep 15 21:52:21.854874 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.854851    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b97dd01b-1a98-4852-b87b-af845c7ae9b1-metrics-certs\") pod \"maas-api-775cbbbf65-vqm27\" (UID: \"b97dd01b-1a98-4852-b87b-af845c7ae9b1\") " pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"
Sep 15 21:52:21.854964 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.854946    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b97dd01b-1a98-4852-b87b-af845c7ae9b1-maas-api-tls\") pod \"maas-api-775cbbbf65-vqm27\" (UID: \"b97dd01b-1a98-4852-b87b-af845c7ae9b1\") " pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"
Sep 15 21:52:21.859836 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.859814    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zpzn\" (UniqueName: \"kubernetes.io/projected/b97dd01b-1a98-4852-b87b-af845c7ae9b1-kube-api-access-6zpzn\") pod \"maas-api-775cbbbf65-vqm27\" (UID: \"b97dd01b-1a98-4852-b87b-af845c7ae9b1\") " pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"
Sep 15 21:52:21.909853 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:21.909828    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"
Sep 15 21:52:22.178588 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:22.178365    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"]
Sep 15 21:52:22.180912 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:52:22.180876    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97dd01b_1a98_4852_b87b_af845c7ae9b1.slice/crio-c1792d4699dfb34ca5b69fd16d525ea25aaf8136229707fc1613dc15a255b986 WatchSource:0}: Error finding container c1792d4699dfb34ca5b69fd16d525ea25aaf8136229707fc1613dc15a255b986: Status 404 returned error can't find the container with id c1792d4699dfb34ca5b69fd16d525ea25aaf8136229707fc1613dc15a255b986
Sep 15 21:52:22.993164 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:22.993124    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27" event={"ID":"b97dd01b-1a98-4852-b87b-af845c7ae9b1","Type":"ContainerStarted","Data":"c1792d4699dfb34ca5b69fd16d525ea25aaf8136229707fc1613dc15a255b986"}
Sep 15 21:52:24.000357 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:24.000264    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27" event={"ID":"b97dd01b-1a98-4852-b87b-af845c7ae9b1","Type":"ContainerStarted","Data":"c30a7be320cf404f13242d042d98face66a7a8a18193a5dafad2fca5c6c52668"}
Sep 15 21:52:24.000758 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:24.000449    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"
Sep 15 21:52:24.017052 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:24.017010    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27" podStartSLOduration=1.4827364969999999 podStartE2EDuration="3.016995675s" totalImagesPullingTime="1.534259178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:52:24.01527658 +0000 UTC m=+2511.877210382" watchObservedRunningTime="2026-09-15 21:52:24.016995675 +0000 UTC m=+2511.878929477"
Sep 15 21:52:30.010536 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:30.010462    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-775cbbbf65-vqm27"
Sep 15 21:52:33.382394 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:33.382358    2584 scope.go:117] "RemoveContainer" containerID="40ff0c17794a3a70e923a5b8577d2b0a7369f6dfcdc61afaaf477d98b2e7cfe0"
Sep 15 21:52:33.391430 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:33.391405    2584 scope.go:117] "RemoveContainer" containerID="13cf2cb22cf31e6429a760d84667c5027ce2f61010efaef226731d5e2a1af42c"
Sep 15 21:52:33.400098 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:33.400074    2584 scope.go:117] "RemoveContainer" containerID="f6f07b0563d1b4216e6eefaa8a339f01ce710c7466ec16d5beb2ce900e7ad954"
Sep 15 21:52:50.705761 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:50.705733    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-8sdqj_425cc59c-7511-4a61-9ef3-9021837f32f6/manager/0.log"
Sep 15 21:52:50.835094 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:50.835066    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-9rss4_9877a946-ae67-4460-a1f9-8b50498d4332/manager/0.log"
Sep 15 21:52:50.965096 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:50.965021    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-74f4d48d8b-qfxl5_a55a8124-0a35-4e43-baec-4f953c2f5cf6/manager/0.log"
Sep 15 21:52:51.596979 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:51.596950    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-568b8bf8c4-ptqvt_ff759f48-2d5b-4c93-809e-8b363ce8bb6d/copy-manifests/0.log"
Sep 15 21:52:51.603352 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:51.603325    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-568b8bf8c4-ptqvt_ff759f48-2d5b-4c93-809e-8b363ce8bb6d/manager/0.log"
Sep 15 21:52:52.087658 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:52.087628    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-775cbbbf65-vqm27_b97dd01b-1a98-4852-b87b-af845c7ae9b1/maas-api/0.log"
Sep 15 21:52:52.318314 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:52.318285    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29825145-8dchd_c87b7732-e9a7-4653-a499-38caaac5ae6e/cleanup/0.log"
Sep 15 21:52:53.346744 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:53.346715    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-fglmp_a6d036da-41d2-4ef0-a98d-f439302aa413/manager/0.log"
Sep 15 21:52:53.464817 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:53.464789    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-n8nhk_c9dee725-b3fb-445e-bb74-7703ce75d73f/manager/0.log"
Sep 15 21:52:53.582613 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:53.582590    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-65mm8_44d32b0f-59dd-44d9-936c-3ec66bbdbc18/kuadrant-console-plugin/0.log"
Sep 15 21:52:53.722095 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:53.722012    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-swjzd_58b68082-d855-44f6-a3dc-6b81b1467da5/manager/0.log"
Sep 15 21:52:53.971559 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:53.971518    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-twq57_49d2501c-80ab-40c0-bc11-df6414168bd7/manager/0.log"
Sep 15 21:52:54.336491 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:54.336459    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw_be3a7bee-7136-4630-ae69-61061cf917d1/istio-proxy/0.log"
Sep 15 21:52:54.692110 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:54.692012    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cbb5b4bf7-z86xb_72956236-ecac-4bda-af44-f973911d4235/kube-auth-proxy/0.log"
Sep 15 21:52:54.839062 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:54.839033    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c747fcbf8-jcbs4_fd952d22-e6de-4c5d-acdb-8da8fb46ffb1/istio-proxy/1.log"
Sep 15 21:52:54.965701 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:54.965625    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-5wksw_fa501378-f7fd-4481-a046-98e4bfc9bb6d/payload-pre-processing/0.log"
Sep 15 21:52:55.205956 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:55.205927    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-547bb554b6-j9fkz_63176ac2-9b27-4a17-88f4-0a934db86c97/router/0.log"
Sep 15 21:52:55.564174 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:55.564147    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-bg7fz_ac3aa4b4-5064-4610-9b10-d005889f006c/main/0.log"
Sep 15 21:52:55.788104 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:55.788074    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-qnw5j_f91de23a-0f65-4377-8f30-1039e25e8448/main/0.log"
Sep 15 21:52:55.900163 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:52:55.900084    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-vm45z_421c7bc4-ebd3-4f5b-aa03-0a3d112ca2cf/main/0.log"
Sep 15 21:53:05.346449 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:05.346418    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wghbm_a2260413-c64c-4707-92f9-ed64418eeda4/global-pull-secret-syncer/0.log"
Sep 15 21:53:05.467049 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:05.467025    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jrzds_ba752fdf-05e3-4c36-adf6-4f443cf4f2e4/konnectivity-agent/0.log"
Sep 15 21:53:05.512456 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:05.512431    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-184.ec2.internal_eaddcc5db67cf457ad141690276d44eb/haproxy/0.log"
Sep 15 21:53:10.329984 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:10.329952    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-fglmp_a6d036da-41d2-4ef0-a98d-f439302aa413/manager/0.log"
Sep 15 21:53:10.352260 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:10.352238    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-n8nhk_c9dee725-b3fb-445e-bb74-7703ce75d73f/manager/0.log"
Sep 15 21:53:10.377202 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:10.377182    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-65mm8_44d32b0f-59dd-44d9-936c-3ec66bbdbc18/kuadrant-console-plugin/0.log"
Sep 15 21:53:10.424374 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:10.424353    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-swjzd_58b68082-d855-44f6-a3dc-6b81b1467da5/manager/0.log"
Sep 15 21:53:10.538491 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:10.538468    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-twq57_49d2501c-80ab-40c0-bc11-df6414168bd7/manager/0.log"
Sep 15 21:53:12.150011 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:12.149977    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-mfp8r_306fd13e-6e53-43f4-9903-e0a886c375e8/cluster-monitoring-operator/0.log"
Sep 15 21:53:12.253829 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:12.253757    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d57bdd96c-4hnx6_7bd6c5fd-299c-4553-aae7-173fb0142ad5/metrics-server/0.log"
Sep 15 21:53:12.281395 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:12.281357    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-j9smx_27b7e762-2ec9-4277-8c31-53910792ed85/monitoring-plugin/0.log"
Sep 15 21:53:12.329158 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:12.329139    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5tvnc_f1913778-4e8f-4c93-9078-48d67f4884c0/node-exporter/0.log"
Sep 15 21:53:12.349609 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:12.349585    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5tvnc_f1913778-4e8f-4c93-9078-48d67f4884c0/kube-rbac-proxy/0.log"
Sep 15 21:53:12.370201 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:12.370185    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5tvnc_f1913778-4e8f-4c93-9078-48d67f4884c0/init-textfile/0.log"
Sep 15 21:53:12.713092 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:12.713054    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6de4ec41-0eb5-4e60-b967-c56afe68e625/prometheus/0.log"
Sep 15 21:53:12.736747 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:12.736722    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6de4ec41-0eb5-4e60-b967-c56afe68e625/config-reloader/0.log"
Sep 15 21:53:12.760015 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:12.759944    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6de4ec41-0eb5-4e60-b967-c56afe68e625/thanos-sidecar/0.log"
Sep 15 21:53:12.780545 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:12.780520    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6de4ec41-0eb5-4e60-b967-c56afe68e625/kube-rbac-proxy-web/0.log"
Sep 15 21:53:12.805414 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:12.805391    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6de4ec41-0eb5-4e60-b967-c56afe68e625/kube-rbac-proxy/0.log"
Sep 15 21:53:12.826527 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:12.826501    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6de4ec41-0eb5-4e60-b967-c56afe68e625/kube-rbac-proxy-thanos/0.log"
Sep 15 21:53:12.847700 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:12.847674    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6de4ec41-0eb5-4e60-b967-c56afe68e625/init-config-reloader/0.log"
Sep 15 21:53:13.058983 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:13.058957    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79ff6dcd-zr4mh_701bffe8-9eff-4e17-943c-6e005cf74022/thanos-query/0.log"
Sep 15 21:53:13.083218 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:13.083190    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79ff6dcd-zr4mh_701bffe8-9eff-4e17-943c-6e005cf74022/kube-rbac-proxy-web/0.log"
Sep 15 21:53:13.106172 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:13.106151    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79ff6dcd-zr4mh_701bffe8-9eff-4e17-943c-6e005cf74022/kube-rbac-proxy/0.log"
Sep 15 21:53:13.129570 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:13.129554    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79ff6dcd-zr4mh_701bffe8-9eff-4e17-943c-6e005cf74022/prom-label-proxy/0.log"
Sep 15 21:53:13.167564 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:13.167545    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79ff6dcd-zr4mh_701bffe8-9eff-4e17-943c-6e005cf74022/kube-rbac-proxy-rules/0.log"
Sep 15 21:53:13.236674 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:13.236652    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79ff6dcd-zr4mh_701bffe8-9eff-4e17-943c-6e005cf74022/kube-rbac-proxy-metrics/0.log"
Sep 15 21:53:15.242915 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.242887    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/2.log"
Sep 15 21:53:15.249137 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.249111    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4d7vv_96e8185d-5c74-4462-88ef-a4c4e53f65f5/console-operator/3.log"
Sep 15 21:53:15.388078 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.388043    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"]
Sep 15 21:53:15.390920 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.390901    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.392977 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.392957    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ljsqz\"/\"kube-root-ca.crt\""
Sep 15 21:53:15.393725 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.393705    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ljsqz\"/\"openshift-service-ca.crt\""
Sep 15 21:53:15.393828 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.393730    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ljsqz\"/\"default-dockercfg-gfrqc\""
Sep 15 21:53:15.401707 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.401685    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"]
Sep 15 21:53:15.444448 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.444428    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsvqs\" (UniqueName: \"kubernetes.io/projected/94bb44af-9566-4895-bcc0-04b3ed2cfabf-kube-api-access-wsvqs\") pod \"perf-node-gather-daemonset-pxkn9\" (UID: \"94bb44af-9566-4895-bcc0-04b3ed2cfabf\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.444546 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.444468    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/94bb44af-9566-4895-bcc0-04b3ed2cfabf-proc\") pod \"perf-node-gather-daemonset-pxkn9\" (UID: \"94bb44af-9566-4895-bcc0-04b3ed2cfabf\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.444546 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.444528    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/94bb44af-9566-4895-bcc0-04b3ed2cfabf-podres\") pod \"perf-node-gather-daemonset-pxkn9\" (UID: \"94bb44af-9566-4895-bcc0-04b3ed2cfabf\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.444629 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.444554    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94bb44af-9566-4895-bcc0-04b3ed2cfabf-lib-modules\") pod \"perf-node-gather-daemonset-pxkn9\" (UID: \"94bb44af-9566-4895-bcc0-04b3ed2cfabf\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.444629 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.444609    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94bb44af-9566-4895-bcc0-04b3ed2cfabf-sys\") pod \"perf-node-gather-daemonset-pxkn9\" (UID: \"94bb44af-9566-4895-bcc0-04b3ed2cfabf\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.545882 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.545850    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/94bb44af-9566-4895-bcc0-04b3ed2cfabf-podres\") pod \"perf-node-gather-daemonset-pxkn9\" (UID: \"94bb44af-9566-4895-bcc0-04b3ed2cfabf\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.545975 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.545900    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94bb44af-9566-4895-bcc0-04b3ed2cfabf-lib-modules\") pod \"perf-node-gather-daemonset-pxkn9\" (UID: \"94bb44af-9566-4895-bcc0-04b3ed2cfabf\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.545975 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.545925    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94bb44af-9566-4895-bcc0-04b3ed2cfabf-sys\") pod \"perf-node-gather-daemonset-pxkn9\" (UID: \"94bb44af-9566-4895-bcc0-04b3ed2cfabf\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.545975 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.545952    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsvqs\" (UniqueName: \"kubernetes.io/projected/94bb44af-9566-4895-bcc0-04b3ed2cfabf-kube-api-access-wsvqs\") pod \"perf-node-gather-daemonset-pxkn9\" (UID: \"94bb44af-9566-4895-bcc0-04b3ed2cfabf\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.546108 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.545983    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/94bb44af-9566-4895-bcc0-04b3ed2cfabf-proc\") pod \"perf-node-gather-daemonset-pxkn9\" (UID: \"94bb44af-9566-4895-bcc0-04b3ed2cfabf\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.546108 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.546015    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/94bb44af-9566-4895-bcc0-04b3ed2cfabf-podres\") pod \"perf-node-gather-daemonset-pxkn9\" (UID: \"94bb44af-9566-4895-bcc0-04b3ed2cfabf\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.546108 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.546016    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94bb44af-9566-4895-bcc0-04b3ed2cfabf-lib-modules\") pod \"perf-node-gather-daemonset-pxkn9\" (UID: \"94bb44af-9566-4895-bcc0-04b3ed2cfabf\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.546108 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.546051    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/94bb44af-9566-4895-bcc0-04b3ed2cfabf-proc\") pod \"perf-node-gather-daemonset-pxkn9\" (UID: \"94bb44af-9566-4895-bcc0-04b3ed2cfabf\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.546108 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.546051    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94bb44af-9566-4895-bcc0-04b3ed2cfabf-sys\") pod \"perf-node-gather-daemonset-pxkn9\" (UID: \"94bb44af-9566-4895-bcc0-04b3ed2cfabf\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.553126 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.553102    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsvqs\" (UniqueName: \"kubernetes.io/projected/94bb44af-9566-4895-bcc0-04b3ed2cfabf-kube-api-access-wsvqs\") pod \"perf-node-gather-daemonset-pxkn9\" (UID: \"94bb44af-9566-4895-bcc0-04b3ed2cfabf\") " pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.700730 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.700710    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:15.967323 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:15.967299    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"]
Sep 15 21:53:15.968906 ip-10-0-139-184 kubenswrapper[2584]: W0915 21:53:15.968879    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod94bb44af_9566_4895_bcc0_04b3ed2cfabf.slice/crio-049d720ab09b31ebf66bfb23e333bfaed760a63f82a415877318d109ea9c6579 WatchSource:0}: Error finding container 049d720ab09b31ebf66bfb23e333bfaed760a63f82a415877318d109ea9c6579: Status 404 returned error can't find the container with id 049d720ab09b31ebf66bfb23e333bfaed760a63f82a415877318d109ea9c6579
Sep 15 21:53:16.263979 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:16.263914    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9" event={"ID":"94bb44af-9566-4895-bcc0-04b3ed2cfabf","Type":"ContainerStarted","Data":"b3aecaf888f889aba42fb7c83104fddb14dcc3d04fdd30049eb574d4f04ebdd8"}
Sep 15 21:53:16.263979 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:16.263959    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9" event={"ID":"94bb44af-9566-4895-bcc0-04b3ed2cfabf","Type":"ContainerStarted","Data":"049d720ab09b31ebf66bfb23e333bfaed760a63f82a415877318d109ea9c6579"}
Sep 15 21:53:16.264356 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:16.264036    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:16.281370 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:16.281330    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9" podStartSLOduration=1.281318317 podStartE2EDuration="1.281318317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:53:16.280041345 +0000 UTC m=+2564.141975149" watchObservedRunningTime="2026-09-15 21:53:16.281318317 +0000 UTC m=+2564.143252120"
Sep 15 21:53:16.483518 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:16.483487    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-bhlld_8270c2c6-29f0-4ce5-b4a6-ec3b322eccaf/volume-data-source-validator/0.log"
Sep 15 21:53:17.648784 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:17.648754    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vg2k9_62fa88e4-9983-484f-87b7-65a070871dfa/dns/0.log"
Sep 15 21:53:17.670821 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:17.670795    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vg2k9_62fa88e4-9983-484f-87b7-65a070871dfa/kube-rbac-proxy/0.log"
Sep 15 21:53:17.720752 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:17.720726    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q2rvv_c27eb1d3-c379-4e71-9694-7cfce96c4a47/dns-node-resolver/0.log"
Sep 15 21:53:18.363001 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:18.362978    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-69c485f746-vlgs4_ddd07ea7-ab52-4446-a110-4d9ea6c2cc77/registry/0.log"
Sep 15 21:53:18.434943 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:18.434911    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rh5pt_c1ff81ab-8f51-438e-a910-4d6624e8cbbe/node-ca/0.log"
Sep 15 21:53:19.370330 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:19.370283    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw_be3a7bee-7136-4630-ae69-61061cf917d1/istio-proxy/0.log"
Sep 15 21:53:20.224140 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:20.224111    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cbb5b4bf7-z86xb_72956236-ecac-4bda-af44-f973911d4235/kube-auth-proxy/0.log"
Sep 15 21:53:20.251970 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:20.251947    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c747fcbf8-jcbs4_fd952d22-e6de-4c5d-acdb-8da8fb46ffb1/istio-proxy/0.log"
Sep 15 21:53:20.257210 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:20.257187    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c747fcbf8-jcbs4_fd952d22-e6de-4c5d-acdb-8da8fb46ffb1/istio-proxy/1.log"
Sep 15 21:53:20.294828 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:20.294800    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-5wksw_fa501378-f7fd-4481-a046-98e4bfc9bb6d/payload-pre-processing/0.log"
Sep 15 21:53:20.360176 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:20.360153    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-547bb554b6-j9fkz_63176ac2-9b27-4a17-88f4-0a934db86c97/router/0.log"
Sep 15 21:53:21.262804 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:21.262775    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-grkbv_7e36f14f-f82d-49f1-aacf-aba017667671/serve-healthcheck-canary/0.log"
Sep 15 21:53:21.849233 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:21.849205    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4mgr8_2a0964c1-a9bf-466a-a96e-c51abcdb59cc/kube-rbac-proxy/0.log"
Sep 15 21:53:21.869992 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:21.869970    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4mgr8_2a0964c1-a9bf-466a-a96e-c51abcdb59cc/exporter/0.log"
Sep 15 21:53:21.894027 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:21.894005    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4mgr8_2a0964c1-a9bf-466a-a96e-c51abcdb59cc/extractor/0.log"
Sep 15 21:53:22.286603 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:22.286581    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ljsqz/perf-node-gather-daemonset-pxkn9"
Sep 15 21:53:24.200911 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:24.200874    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-8sdqj_425cc59c-7511-4a61-9ef3-9021837f32f6/manager/0.log"
Sep 15 21:53:24.693710 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:24.693682    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-9rss4_9877a946-ae67-4460-a1f9-8b50498d4332/manager/0.log"
Sep 15 21:53:24.775858 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:24.775825    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-74f4d48d8b-qfxl5_a55a8124-0a35-4e43-baec-4f953c2f5cf6/manager/0.log"
Sep 15 21:53:25.046345 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:25.046312    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw_be3a7bee-7136-4630-ae69-61061cf917d1/istio-proxy/0.log"
Sep 15 21:53:25.131829 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:25.131807    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-568b8bf8c4-ptqvt_ff759f48-2d5b-4c93-809e-8b363ce8bb6d/manager/0.log"
Sep 15 21:53:25.154176 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:25.154158    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-568b8bf8c4-ptqvt_ff759f48-2d5b-4c93-809e-8b363ce8bb6d/copy-manifests/0.log"
Sep 15 21:53:25.888541 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:25.888505    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cbb5b4bf7-z86xb_72956236-ecac-4bda-af44-f973911d4235/kube-auth-proxy/0.log"
Sep 15 21:53:25.910148 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:25.910117    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c747fcbf8-jcbs4_fd952d22-e6de-4c5d-acdb-8da8fb46ffb1/istio-proxy/0.log"
Sep 15 21:53:25.914694 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:25.914671    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c747fcbf8-jcbs4_fd952d22-e6de-4c5d-acdb-8da8fb46ffb1/istio-proxy/1.log"
Sep 15 21:53:25.944922 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:25.944897    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-5wksw_fa501378-f7fd-4481-a046-98e4bfc9bb6d/payload-pre-processing/0.log"
Sep 15 21:53:25.987586 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:25.987562    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-547bb554b6-j9fkz_63176ac2-9b27-4a17-88f4-0a934db86c97/router/0.log"
Sep 15 21:53:27.506645 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:27.506619    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6cbcdcfccc6wscw_be3a7bee-7136-4630-ae69-61061cf917d1/istio-proxy/0.log"
Sep 15 21:53:28.360502 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:28.360473    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cbb5b4bf7-z86xb_72956236-ecac-4bda-af44-f973911d4235/kube-auth-proxy/0.log"
Sep 15 21:53:28.380305 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:28.380281    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c747fcbf8-jcbs4_fd952d22-e6de-4c5d-acdb-8da8fb46ffb1/istio-proxy/0.log"
Sep 15 21:53:28.385525 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:28.385494    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c747fcbf8-jcbs4_fd952d22-e6de-4c5d-acdb-8da8fb46ffb1/istio-proxy/1.log"
Sep 15 21:53:28.413775 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:28.413754    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-5wksw_fa501378-f7fd-4481-a046-98e4bfc9bb6d/payload-pre-processing/0.log"
Sep 15 21:53:28.470649 ip-10-0-139-184 kubenswrapper[2584]: I0915 21:53:28.470622    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-547bb554b6-j9fkz_63176ac2-9b27-4a17-88f4-0a934db86c97/router/0.log"