Sep 16 14:14:09.352260 ip-10-0-128-50 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 14:14:09.692900 ip-10-0-128-50 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 14:14:09.692900 ip-10-0-128-50 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 14:14:09.692900 ip-10-0-128-50 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 14:14:09.692900 ip-10-0-128-50 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 14:14:09.692900 ip-10-0-128-50 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 14:14:09.693531 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.692923    2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 14:14:09.696305 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696288    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:14:09.696305 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696303    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:14:09.696305 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696307    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696311    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696315    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696319    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696324    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696327    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696330    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696333    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696335    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696339    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696342    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696346    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696348    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696351    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696354    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696356    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696359    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696362    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696364    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696367    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:14:09.696396 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696369    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696372    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696375    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696377    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696380    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696383    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696385    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696388    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696391    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696393    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696397    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696399    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696402    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696405    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696408    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696411    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696413    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696416    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696419    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696421    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:14:09.696872 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696425    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696427    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696430    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696433    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696435    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696438    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696440    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696443    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696445    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696448    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696450    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696455    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696458    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696461    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696464    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696468    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696471    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696474    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696477    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:14:09.697363 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696480    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696483    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696486    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696488    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696491    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696493    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696496    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696498    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696501    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696503    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696506    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696509    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696511    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696514    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696517    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696520    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696525    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696528    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696531    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:14:09.697838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696534    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696538    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696541    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696544    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696546    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696549    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696552    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696555    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696559    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696562    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696564    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696567    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696570    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696573    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696576    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696578    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696581    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696583    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.696586    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697263    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:14:09.698308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697269    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697272    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697275    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697278    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697281    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697283    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697286    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697289    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697303    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697306    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697309    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697312    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697315    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697317    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697320    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697322    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697325    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697328    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697330    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697333    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:14:09.698810 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697337    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697340    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697343    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697345    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697348    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697351    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697353    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697355    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697360    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697364    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697367    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697370    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697372    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697375    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697378    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697380    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697383    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697386    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697388    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:14:09.699343 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697391    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697394    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697397    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697400    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697402    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697405    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697407    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697411    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697414    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697416    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697419    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697422    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697424    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697430    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697433    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697436    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697439    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697442    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697445    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:14:09.699814 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697448    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697452    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697455    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697457    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697460    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697462    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697465    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697467    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697470    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697474    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697476    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697479    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697481    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697484    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697487    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697489    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697493    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697495    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697498    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697500    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:14:09.700286 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697503    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697505    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697508    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697511    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697513    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697516    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697519    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697521    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697523    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697526    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697528    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697531    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697534    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697536    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697538    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697541    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697543    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697546    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697548    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697551    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:14:09.700809 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697945    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697950    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697953    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697956    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697958    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697961    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697963    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697966    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697969    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697972    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697975    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697977    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697980    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697983    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697986    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697988    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697991    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697994    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.697998    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698002    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:14:09.701308 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698005    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698008    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698011    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698014    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698017    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698020    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698023    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698026    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698028    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698031    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698034    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698037    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698039    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698042    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698044    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698047    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698049    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698052    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698054    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:14:09.701800 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698056    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698059    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698063    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698065    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698068    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698070    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698072    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698076    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698078    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698081    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698083    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698086    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698088    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698091    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698093    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698095    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698098    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698102    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698104    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698108    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:14:09.702266 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698110    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698113    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698115    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698118    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698120    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698123    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698125    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698127    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698133    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698136    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698139    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698142    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698144    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698147    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698152    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698155    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698157    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698160    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698162    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:14:09.702739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698165    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698167    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698170    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698172    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698175    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698177    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698180    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698182    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698185    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698187    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698189    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698192    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698194    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698197    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698199    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698202    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698204    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698207    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698209    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698211    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:14:09.703210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698214    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698286    2553 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698317    2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698326    2553 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698331    2553 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698335    2553 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698338    2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698343    2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698347    2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698352    2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698355    2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698358    2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698362    2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698365    2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698368    2553 flags.go:64] FLAG: --cgroup-root=""
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698371    2553 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698374    2553 flags.go:64] FLAG: --client-ca-file=""
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698377    2553 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698380    2553 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698385    2553 flags.go:64] FLAG: --cluster-domain=""
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698388    2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698391    2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698394    2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698397    2553 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 14:14:09.703791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698401    2553 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698403    2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698406    2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698410    2553 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698413    2553 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698416    2553 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698419    2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698422    2553 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698425    2553 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698428    2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698431    2553 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698434    2553 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698437    2553 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698440    2553 flags.go:64] FLAG: --enable-server="true"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698443    2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698447    2553 flags.go:64] FLAG: --event-burst="100"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698450    2553 flags.go:64] FLAG: --event-qps="50"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698454    2553 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698457    2553 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698460    2553 flags.go:64] FLAG: --eviction-hard=""
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698464    2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698467    2553 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698470    2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698473    2553 flags.go:64] FLAG: --eviction-soft=""
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698476    2553 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 14:14:09.704399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698479    2553 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698482    2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698484    2553 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698487    2553 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698490    2553 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698493    2553 flags.go:64] FLAG: --feature-gates=""
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698496    2553 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698499    2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698502    2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698505    2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698508    2553 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698512    2553 flags.go:64] FLAG: --help="false"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698514    2553 flags.go:64] FLAG: --hostname-override="ip-10-0-128-50.ec2.internal"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698517    2553 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698521    2553 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698524    2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698527    2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698531    2553 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698534    2553 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698537    2553 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698540    2553 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698542    2553 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698545    2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698548    2553 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 14:14:09.705084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698551    2553 flags.go:64] FLAG: --kube-reserved=""
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698554    2553 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698557    2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698562    2553 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698565    2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698567    2553 flags.go:64] FLAG: --lock-file=""
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698570    2553 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698573    2553 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698576    2553 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698581    2553 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698584    2553 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698587    2553 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698590    2553 flags.go:64] FLAG: --logging-format="text"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698592    2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698596    2553 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698599    2553 flags.go:64] FLAG: --manifest-url=""
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698602    2553 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698610    2553 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698613    2553 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698617    2553 flags.go:64] FLAG: --max-pods="110"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698620    2553 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698623    2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698626    2553 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698629    2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698632    2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 14:14:09.705701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698635    2553 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698637    2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698645    2553 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698648    2553 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698651    2553 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698654    2553 flags.go:64] FLAG: --pod-cidr=""
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698657    2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698662    2553 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698664    2553 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698667    2553 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698671    2553 flags.go:64] FLAG: --port="10250"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698676    2553 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698679    2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07963b18cd5f1bb39"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698682    2553 flags.go:64] FLAG: --qos-reserved=""
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698685    2553 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698688    2553 flags.go:64] FLAG: --register-node="true"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698691    2553 flags.go:64] FLAG: --register-with-taints=""
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698698    2553 flags.go:64] FLAG: --registry-burst="10"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698701    2553 flags.go:64] FLAG: --registry-qps="5"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698704    2553 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698707    2553 flags.go:64] FLAG: --reserved-memory=""
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698710    2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698713    2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698716    2553 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698719    2553 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 14:14:09.706332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698721    2553 flags.go:64] FLAG: --runonce="false"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698724    2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698727    2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698730    2553 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698734    2553 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698737    2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698740    2553 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698743    2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698746    2553 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698749    2553 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698752    2553 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698754    2553 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698757    2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698760    2553 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698763    2553 flags.go:64] FLAG: --system-cgroups=""
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698765    2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698770    2553 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698773    2553 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698778    2553 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698782    2553 flags.go:64] FLAG: --tls-min-version=""
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698785    2553 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698787    2553 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698790    2553 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698793    2553 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698796    2553 flags.go:64] FLAG: --v="2"
Sep 16 14:14:09.706956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698800    2553 flags.go:64] FLAG: --version="false"
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698804    2553 flags.go:64] FLAG: --vmodule=""
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698807    2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.698810    2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698958    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698962    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698965    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698968    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698971    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698974    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698977    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698980    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698983    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698986    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698989    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698992    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698994    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698997    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.698999    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699003    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:14:09.707558 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699005    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699008    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699010    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699013    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699016    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699020    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699023    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699026    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699028    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699031    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699034    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699037    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699040    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699042    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699045    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699047    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699050    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699052    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699055    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699058    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:14:09.708093 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699061    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699064    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699066    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699073    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699076    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699078    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699081    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699084    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699086    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699089    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699091    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699094    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699097    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699099    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699102    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699104    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699107    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699110    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699113    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:14:09.708579 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699116    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699119    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699122    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699124    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699127    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699130    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699132    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699135    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699137    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699140    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699142    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699145    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699147    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699152    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699156    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699160    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699163    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699167    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699171    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:14:09.709061 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699174    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699177    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699180    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699183    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699185    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699188    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699191    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699193    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699196    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699198    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699201    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699205    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699207    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699210    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699213    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699215    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699218    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699220    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699223    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699226    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:14:09.709518 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699228    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:14:09.710059 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699231    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:14:09.710059 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699233    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:14:09.710059 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699236    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:14:09.710059 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.699238    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:14:09.710059 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.699636    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 14:14:09.710059 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.706588    2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 14:14:09.710059 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.706601    2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 14:14:09.710059 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.707285    2553 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 14:14:09.710059 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.707293    2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 14:14:09.710059 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707350    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:14:09.710059 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707354    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:14:09.710059 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707358    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:14:09.710059 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707361    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:14:09.710059 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707364    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:14:09.710059 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707367    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707370    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707374    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707377    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707380    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707383    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707386    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707389    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707393    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707397    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707400    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707403    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707406    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707409    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707411    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707414    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707416    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707419    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707423    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:14:09.710440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707426    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707429    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707432    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707435    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707437    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707441    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707443    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707446    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707449    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707451    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707454    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707457    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707459    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707462    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707464    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707467    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707469    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707472    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707475    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707477    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:14:09.710913 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707480    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707482    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707486    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707489    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707492    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707494    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707497    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707499    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707502    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707504    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707507    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707510    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707512    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707515    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707517    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707520    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707522    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707525    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707528    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707530    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:14:09.711399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707533    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707535    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707538    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707540    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707543    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707545    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707548    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707550    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707553    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707556    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707558    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707561    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707572    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707575    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707578    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707581    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707583    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707586    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707588    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:14:09.711900 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707591    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:14:09.712358 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707594    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:14:09.712358 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707597    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:14:09.712358 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707599    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:14:09.712358 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707602    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:14:09.712358 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707604    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:14:09.712358 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707607    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:14:09.712358 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707610    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:14:09.712358 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707612    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:14:09.712358 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707615    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:14:09.712358 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707617    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:14:09.712358 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707620    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:14:09.712358 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707624    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:14:09.712358 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707626    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:14:09.712358 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707629    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:14:09.712358 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707632    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:14:09.712718 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.707637    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 14:14:09.712718 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707745    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:14:09.712718 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707750    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:14:09.712718 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707753    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:14:09.712718 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707756    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:14:09.712718 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707759    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:14:09.712718 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707762    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:14:09.712718 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707764    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:14:09.712718 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707767    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:14:09.712718 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707770    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:14:09.712718 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707772    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:14:09.712718 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707775    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:14:09.712718 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707778    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:14:09.712718 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707782    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:14:09.712718 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707785    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707788    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707791    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707793    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707796    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707799    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707802    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707804    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707807    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707809    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707812    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707814    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707817    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707819    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707822    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707824    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707827    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707830    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707832    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707835    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:14:09.713097 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707837    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707840    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707842    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707845    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707847    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707850    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707853    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707856    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707858    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707861    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707863    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707866    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707869    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707871    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707874    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707876    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707879    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707881    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707884    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:14:09.713595 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707886    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707907    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707909    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707912    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707914    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707917    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707920    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707922    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707925    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707928    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707930    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707933    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707936    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707938    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707941    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707943    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707946    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707948    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707951    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707956    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:14:09.714066 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707958    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707961    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707964    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707966    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707969    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707971    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707974    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707976    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707979    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707982    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707984    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707989    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707992    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707995    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.707998    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.708001    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.708004    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.708008    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:14:09.714537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.708011    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:14:09.715181 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.708014    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:14:09.715181 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.708017    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:14:09.715181 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.708019    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:14:09.715181 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.708023    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:14:09.715181 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.708025    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:14:09.715181 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.708029    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:14:09.715181 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.708032    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:14:09.715181 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:09.708034    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:14:09.715181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.708039    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 14:14:09.715181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.708139    2553 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 14:14:09.715181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.709863    2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 14:14:09.715181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.710525    2553 server.go:1019] "Starting client certificate rotation"
Sep 16 14:14:09.715181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.710618    2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 14:14:09.715181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.711290    2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 14:14:09.715777 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.713159    2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 14:14:09.715777 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.714860    2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 14:14:09.724219 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.724201    2553 log.go:25] "Validated CRI v1 runtime API"
Sep 16 14:14:09.729783 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.729768    2553 log.go:25] "Validated CRI v1 image API"
Sep 16 14:14:09.731175 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.731158    2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 14:14:09.736463 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.736438    2553 fs.go:135] Filesystem UUIDs: map[565626d3-0890-4482-b0bf-452d9198961b:/dev/nvme0n1p4 77cb9083-572c-489c-bf92-9989240ced80:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 16 14:14:09.736551 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.736460    2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 14:14:09.742011 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.741901    2553 manager.go:216] Machine: {Timestamp:2026-09-16 14:14:09.740611613 +0000 UTC m=+0.287353536 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119363 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26259a24253f1fdcca6594a37b58a9 SystemUUID:ec26259a-2425-3f1f-dcca-6594a37b58a9 BootID:48e63891-6083-4dd1-90b8-1c27ecbaf2ac Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:f9:89:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:f9:89:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:2e:25:a5:86:bd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 14:14:09.742011 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.742005    2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 14:14:09.742126 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.742084    2553 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 14:14:09.744521 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.744495    2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 14:14:09.744662 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.744522    2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-50.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 14:14:09.745174 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.745165    2553 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 14:14:09.745211 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.745177    2553 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 14:14:09.745211 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.745185    2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 14:14:09.745730 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.745720    2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 14:14:09.746359 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.746349    2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 14:14:09.747821 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.747803    2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 14:14:09.749426 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.749414    2553 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 14:14:09.749539 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.749530    2553 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 14:14:09.749582 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.749560    2553 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 14:14:09.749582 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.749570    2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 14:14:09.749655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.749582    2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 14:14:09.749655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.749591    2553 kubelet.go:398] "Adding apiserver pod source"
Sep 16 14:14:09.749655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.749601    2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 14:14:09.749947 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.749934    2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 14:14:09.750007 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.749951    2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 14:14:09.752933 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.752914    2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 14:14:09.754464 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.754449    2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 14:14:09.754523 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.754478    2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 14:14:09.756362 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.756347    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 14:14:09.756419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.756366    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 14:14:09.756419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.756374    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 14:14:09.756419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.756382    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 14:14:09.756419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.756393    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 14:14:09.756419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.756400    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 14:14:09.756419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.756414    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 14:14:09.756419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.756419    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 14:14:09.756605 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.756426    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 14:14:09.756605 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.756431    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 14:14:09.756605 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.756439    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 14:14:09.756605 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.756445    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 14:14:09.757104 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.757090    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 14:14:09.757104 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.757105    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 14:14:09.762340 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.762323    2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-50.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 14:14:09.762340 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.762287    2553 server.go:1268] "Started kubelet"
Sep 16 14:14:09.762502 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.762324    2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 14:14:09.762502 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.762389    2553 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 14:14:09.762502 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.762451    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 14:14:09.762651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.762506    2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 14:14:09.762651 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.762585    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-50.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 14:14:09.763338 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.763323    2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 14:14:09.763376 ip-10-0-128-50 systemd[1]: Started Kubernetes Kubelet.
Sep 16 14:14:09.767374 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.767349    2553 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 14:14:09.767840 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.767820    2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 14:14:09.768657 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.768643    2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 14:14:09.768712 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.768661    2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 14:14:09.768809 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.768790    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-50.ec2.internal\" not found"
Sep 16 14:14:09.769166 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.769137    2553 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 14:14:09.769166 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.769156    2553 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 14:14:09.769303 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.769213    2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 14:14:09.769303 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.769266    2553 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 14:14:09.769303 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.769275    2553 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 14:14:09.769583 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.769565    2553 factory.go:55] Registering systemd factory
Sep 16 14:14:09.769583 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.769585    2553 factory.go:223] Registration of the systemd container factory successfully
Sep 16 14:14:09.771182 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.769966    2553 factory.go:153] Registering CRI-O factory
Sep 16 14:14:09.771182 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.769984    2553 factory.go:223] Registration of the crio container factory successfully
Sep 16 14:14:09.771182 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.770032    2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 14:14:09.771182 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.770051    2553 factory.go:103] Registering Raw factory
Sep 16 14:14:09.771182 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.770065    2553 manager.go:1196] Started watching for new ooms in manager
Sep 16 14:14:09.771182 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.770493    2553 manager.go:349] Starting recovery of all containers
Sep 16 14:14:09.771482 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.771265    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 14:14:09.772468 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.771406    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-50.ec2.internal.18d5d2a8e376ba70  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-50.ec2.internal,UID:ip-10-0-128-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-50.ec2.internal,},FirstTimestamp:2026-09-16 14:14:09.762187888 +0000 UTC m=+0.308929810,LastTimestamp:2026-09-16 14:14:09.762187888 +0000 UTC m=+0.308929810,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-50.ec2.internal,}"
Sep 16 14:14:09.772468 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.772450    2553 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-50.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 14:14:09.783202 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.783177    2553 manager.go:354] Recovery completed
Sep 16 14:14:09.788882 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.788401    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:14:09.790253 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.790234    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:14:09.790323 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.790265    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:14:09.790323 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.790275    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:14:09.790793 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.790779    2553 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 14:14:09.790793 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.790790    2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 14:14:09.790793 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.790791    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:14:09.790980 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.790804    2553 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 14:14:09.791608 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.791592    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:14:09.791673 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.791621    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:14:09.791673 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.791634    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:14:09.792529 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.792456    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-50.ec2.internal.18d5d2a8e522f2c6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-50.ec2.internal,UID:ip-10-0-128-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-50.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-50.ec2.internal,},FirstTimestamp:2026-09-16 14:14:09.790251718 +0000 UTC m=+0.336993642,LastTimestamp:2026-09-16 14:14:09.790251718 +0000 UTC m=+0.336993642,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-50.ec2.internal,}"
Sep 16 14:14:09.793632 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.793618    2553 policy_none.go:49] "None policy: Start"
Sep 16 14:14:09.793699 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.793634    2553 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 14:14:09.793699 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.793646    2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 14:14:09.794728 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.794717    2553 policy_none.go:47] "Start"
Sep 16 14:14:09.802787 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.802528    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-50.ec2.internal.18d5d2a8e5233836  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-50.ec2.internal,UID:ip-10-0-128-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-50.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-50.ec2.internal,},FirstTimestamp:2026-09-16 14:14:09.790269494 +0000 UTC m=+0.337011416,LastTimestamp:2026-09-16 14:14:09.790269494 +0000 UTC m=+0.337011416,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-50.ec2.internal,}"
Sep 16 14:14:09.813157 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.813093    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-50.ec2.internal.18d5d2a8e52360e6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-50.ec2.internal,UID:ip-10-0-128-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-50.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-50.ec2.internal,},FirstTimestamp:2026-09-16 14:14:09.79027991 +0000 UTC m=+0.337021832,LastTimestamp:2026-09-16 14:14:09.79027991 +0000 UTC m=+0.337021832,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-50.ec2.internal,}"
Sep 16 14:14:09.815311 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.815293    2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f792x"
Sep 16 14:14:09.821923 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.821901    2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f792x"
Sep 16 14:14:09.833280 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.821947    2553 manager.go:337] "Starting Device Plugin manager"
Sep 16 14:14:09.833280 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.821998    2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 14:14:09.833280 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.822012    2553 server.go:87] "Starting device plugin registration server"
Sep 16 14:14:09.833280 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.822317    2553 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 14:14:09.833280 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.822326    2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 14:14:09.833280 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.822417    2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 14:14:09.833280 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.822481    2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 14:14:09.833280 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.822489    2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 14:14:09.833280 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.823122    2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 14:14:09.833280 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.823193    2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-50.ec2.internal\" not found"
Sep 16 14:14:09.833280 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.823486    2553 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-50.ec2.internal.18d5d2a8e522f2c6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-50.ec2.internal.18d5d2a8e522f2c6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-50.ec2.internal,UID:ip-10-0-128-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-50.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-50.ec2.internal,},FirstTimestamp:2026-09-16 14:14:09.790251718 +0000 UTC m=+0.336993642,LastTimestamp:2026-09-16 14:14:09.791607029 +0000 UTC m=+0.338348956,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-50.ec2.internal,}"
Sep 16 14:14:09.909256 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.909215    2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 14:14:09.910547 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.910532    2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 14:14:09.910617 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.910580    2553 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 14:14:09.910617 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.910602    2553 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 14:14:09.910707 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.910695    2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 14:14:09.913122 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.913104    2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:14:09.923017 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.923000    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:14:09.923866 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.923849    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:14:09.923960 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.923882    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:14:09.923960 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.923916    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:14:09.923960 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.923940    2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-50.ec2.internal"
Sep 16 14:14:09.933248 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:09.933230    2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-50.ec2.internal"
Sep 16 14:14:09.933315 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.933253    2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-50.ec2.internal\": node \"ip-10-0-128-50.ec2.internal\" not found"
Sep 16 14:14:09.965147 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:09.965074    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-50.ec2.internal\" not found"
Sep 16 14:14:10.010972 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.010932    2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-50.ec2.internal"]
Sep 16 14:14:10.011049 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.011020    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:14:10.011959 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.011942    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:14:10.012052 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.011977    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:14:10.012052 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.011992    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:14:10.013439 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.013424    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:14:10.013561 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.013545    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal"
Sep 16 14:14:10.013612 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.013583    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:14:10.014175 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.014159    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:14:10.014259 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.014190    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:14:10.014259 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.014204    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:14:10.014259 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.014159    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:14:10.014259 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.014259    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:14:10.014384 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.014277    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:14:10.015197 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.015180    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-50.ec2.internal"
Sep 16 14:14:10.015292 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.015215    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:14:10.015820 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.015807    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:14:10.015865 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.015834    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:14:10.015865 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.015848    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:14:10.031412 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:10.031389    2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-50.ec2.internal\" not found" node="ip-10-0-128-50.ec2.internal"
Sep 16 14:14:10.034579 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:10.034559    2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-50.ec2.internal\" not found" node="ip-10-0-128-50.ec2.internal"
Sep 16 14:14:10.065733 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:10.065714    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-50.ec2.internal\" not found"
Sep 16 14:14:10.070768 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.070743    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc9a0b5b683b9efc648017b7df9cf169-config\") pod \"kube-apiserver-proxy-ip-10-0-128-50.ec2.internal\" (UID: \"cc9a0b5b683b9efc648017b7df9cf169\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-50.ec2.internal"
Sep 16 14:14:10.070873 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.070799    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fd7d5957c3c1631a2eca1f67dcc24ff3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal\" (UID: \"fd7d5957c3c1631a2eca1f67dcc24ff3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal"
Sep 16 14:14:10.070873 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.070817    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd7d5957c3c1631a2eca1f67dcc24ff3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal\" (UID: \"fd7d5957c3c1631a2eca1f67dcc24ff3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal"
Sep 16 14:14:10.166644 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:10.166609    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-50.ec2.internal\" not found"
Sep 16 14:14:10.171979 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.171956    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fd7d5957c3c1631a2eca1f67dcc24ff3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal\" (UID: \"fd7d5957c3c1631a2eca1f67dcc24ff3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal"
Sep 16 14:14:10.172082 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.171989    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd7d5957c3c1631a2eca1f67dcc24ff3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal\" (UID: \"fd7d5957c3c1631a2eca1f67dcc24ff3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal"
Sep 16 14:14:10.172082 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.172018    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc9a0b5b683b9efc648017b7df9cf169-config\") pod \"kube-apiserver-proxy-ip-10-0-128-50.ec2.internal\" (UID: \"cc9a0b5b683b9efc648017b7df9cf169\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-50.ec2.internal"
Sep 16 14:14:10.172082 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.172078    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc9a0b5b683b9efc648017b7df9cf169-config\") pod \"kube-apiserver-proxy-ip-10-0-128-50.ec2.internal\" (UID: \"cc9a0b5b683b9efc648017b7df9cf169\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-50.ec2.internal"
Sep 16 14:14:10.172218 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.172090    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd7d5957c3c1631a2eca1f67dcc24ff3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal\" (UID: \"fd7d5957c3c1631a2eca1f67dcc24ff3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal"
Sep 16 14:14:10.172218 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.172100    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fd7d5957c3c1631a2eca1f67dcc24ff3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal\" (UID: \"fd7d5957c3c1631a2eca1f67dcc24ff3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal"
Sep 16 14:14:10.267411 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:10.267329    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-50.ec2.internal\" not found"
Sep 16 14:14:10.333830 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.333792    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal"
Sep 16 14:14:10.336661 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.336646    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-50.ec2.internal"
Sep 16 14:14:10.368022 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:10.367995    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-50.ec2.internal\" not found"
Sep 16 14:14:10.469424 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:10.469392    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-50.ec2.internal\" not found"
Sep 16 14:14:10.571517 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:10.571440    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-50.ec2.internal\" not found"
Sep 16 14:14:10.672148 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:10.672117    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-50.ec2.internal\" not found"
Sep 16 14:14:10.677954 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.676633    2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:14:10.711037 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.711013    2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 14:14:10.711440 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.711145    2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 14:14:10.711440 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.711184    2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 14:14:10.772733 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:10.772710    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-50.ec2.internal\" not found"
Sep 16 14:14:10.827209 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.827133    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 14:09:09 +0000 UTC" deadline="2028-02-19 15:50:46.577574698 +0000 UTC"
Sep 16 14:14:10.827209 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.827170    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12505h36m35.75040927s"
Sep 16 14:14:10.873567 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:10.873539    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-50.ec2.internal\" not found"
Sep 16 14:14:10.883076 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.883063    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:14:10.884215 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.884192    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:14:10.884319 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.884233    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:14:10.884319 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.884249    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:14:10.925109 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:10.925076    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7d5957c3c1631a2eca1f67dcc24ff3.slice/crio-cfe11d3cdf352b27104b6bd7411b3a0a59b4e26e65e80d950a9a29672c16d9ee WatchSource:0}: Error finding container cfe11d3cdf352b27104b6bd7411b3a0a59b4e26e65e80d950a9a29672c16d9ee: Status 404 returned error can't find the container with id cfe11d3cdf352b27104b6bd7411b3a0a59b4e26e65e80d950a9a29672c16d9ee
Sep 16 14:14:10.925526 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:10.925504    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9a0b5b683b9efc648017b7df9cf169.slice/crio-1b6159100b748bff5095617ebd7800630d05bc1a1fae08699d7e0a7d9330e8fb WatchSource:0}: Error finding container 1b6159100b748bff5095617ebd7800630d05bc1a1fae08699d7e0a7d9330e8fb: Status 404 returned error can't find the container with id 1b6159100b748bff5095617ebd7800630d05bc1a1fae08699d7e0a7d9330e8fb
Sep 16 14:14:10.930557 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:10.930539    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:14:10.974272 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:10.974246    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-50.ec2.internal\" not found"
Sep 16 14:14:11.074842 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:11.074815    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-50.ec2.internal\" not found"
Sep 16 14:14:11.076347 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.076331    2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:14:11.175345 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.175277    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal"
Sep 16 14:14:11.191677 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.191651    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 14:14:11.192691 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.192676    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-50.ec2.internal"
Sep 16 14:14:11.193025 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.193007    2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:14:11.200387 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.200370    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 14:14:11.729714 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.729681    2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:14:11.751972 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.751950    2553 apiserver.go:52] "Watching apiserver"
Sep 16 14:14:11.760502 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.760480    2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 14:14:11.762820 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.762791    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-50.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7","openshift-image-registry/node-ca-255dt","openshift-multus/multus-8b5hz","openshift-multus/multus-additional-cni-plugins-jccdd","openshift-network-diagnostics/network-check-target-ttfnh","openshift-network-operator/iptables-alerter-brglq","openshift-ovn-kubernetes/ovnkube-node-lnrqf","kube-system/global-pull-secret-syncer-kdnvs","kube-system/konnectivity-agent-kswqv","openshift-cluster-node-tuning-operator/tuned-fkk7b","openshift-dns/node-resolver-s79nx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal","openshift-multus/network-metrics-daemon-5shg9"]
Sep 16 14:14:11.764308 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.764287    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-brglq"
Sep 16 14:14:11.765843 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.765823    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.766978 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.766958    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-255dt"
Sep 16 14:14:11.767145 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.767074    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 14:14:11.767145 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.767107    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-crhd9\""
Sep 16 14:14:11.767371 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.767329    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 14:14:11.767480 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.767455    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 14:14:11.768688 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.768316    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.768688 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.768436    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 14:14:11.769377 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.769360    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2f6v9\""
Sep 16 14:14:11.769807 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.769780    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.769912 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.769839    2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 14:14:11.770709 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.770666    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 14:14:11.771085 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.771070    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 14:14:11.771244 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.771098    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 14:14:11.771456 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.771438    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 14:14:11.771526 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.771470    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 14:14:11.771579 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.771440    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rcp94\""
Sep 16 14:14:11.772248 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.771642    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 14:14:11.775008 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.774987    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:11.775123 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:11.775099    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttfnh" podUID="163028d4-d9b5-4feb-9c49-3af6793693c4"
Sep 16 14:14:11.775187 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.775111    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 14:14:11.775239 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.775195    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9w52m\""
Sep 16 14:14:11.775292 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.775259    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 14:14:11.775356 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.775307    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tv9ch\""
Sep 16 14:14:11.775556 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.775499    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 14:14:11.775706 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.775658    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 14:14:11.776502 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.776484    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:11.776590 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:11.776554    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdnvs" podUID="c23242c8-2ee7-4041-b310-6538da68c470"
Sep 16 14:14:11.777635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.777533    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:11.777635 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:11.777597    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5shg9" podUID="4e432d05-f97c-4799-8a7c-d660680b6298"
Sep 16 14:14:11.778825 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.778805    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.779990 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.779973    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kswqv"
Sep 16 14:14:11.780902 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.780866    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g66b\" (UniqueName: \"kubernetes.io/projected/5cb644a0-2a32-41ef-aee9-7e4943300ac4-kube-api-access-5g66b\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.780986 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.780926    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-run-multus-certs\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.780986 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.780954    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-systemd\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.780986 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.780977    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cb644a0-2a32-41ef-aee9-7e4943300ac4-tmp\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.781172 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.780999    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/279e9549-530e-4e54-afbe-9c0954c59441-host\") pod \"node-ca-255dt\" (UID: \"279e9549-530e-4e54-afbe-9c0954c59441\") " pod="openshift-image-registry/node-ca-255dt"
Sep 16 14:14:11.781172 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781041    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/279e9549-530e-4e54-afbe-9c0954c59441-serviceca\") pod \"node-ca-255dt\" (UID: \"279e9549-530e-4e54-afbe-9c0954c59441\") " pod="openshift-image-registry/node-ca-255dt"
Sep 16 14:14:11.781172 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781095    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-run-netns\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.781172 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781136    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9adcfc8-62d7-451e-9756-510427c1434a-multus-daemon-config\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.781360 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781184    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe-iptables-alerter-script\") pod \"iptables-alerter-brglq\" (UID: \"2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe\") " pod="openshift-network-operator/iptables-alerter-brglq"
Sep 16 14:14:11.781360 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781211    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-kubernetes\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.781360 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781249    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-lib-modules\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.781360 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781280    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-etc-kubernetes\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.781360 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781304    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29b51897-d00a-432c-92eb-29f6a9059b20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.781360 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781327    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-modprobe-d\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.781360 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781348    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-sys\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781368    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-host\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781395    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-var-lib-cni-bin\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781402    2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781418    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-hostroot\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781455    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29b51897-d00a-432c-92eb-29f6a9059b20-cnibin\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781478    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-sysctl-conf\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781487    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781502    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-var-lib-kubelet\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781508    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q9gn8\""
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781524    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-tuned\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781544    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-multus-cni-dir\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781557    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-cnibin\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781575    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-multus-socket-dir-parent\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781589    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-multus-conf-dir\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781603    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29b51897-d00a-432c-92eb-29f6a9059b20-cni-binary-copy\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781616    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-sysctl-d\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781663    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt77z\" (UniqueName: \"kubernetes.io/projected/279e9549-530e-4e54-afbe-9c0954c59441-kube-api-access-xt77z\") pod \"node-ca-255dt\" (UID: \"279e9549-530e-4e54-afbe-9c0954c59441\") " pod="openshift-image-registry/node-ca-255dt"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781691    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-system-cni-dir\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.781733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781694    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 14:14:11.782655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781719    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9adcfc8-62d7-451e-9756-510427c1434a-cni-binary-copy\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.782655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781700    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 14:14:11.782655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781737    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/29b51897-d00a-432c-92eb-29f6a9059b20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.782655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781761    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff4hn\" (UniqueName: \"kubernetes.io/projected/29b51897-d00a-432c-92eb-29f6a9059b20-kube-api-access-ff4hn\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.782655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781784    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-run-k8s-cni-cncf-io\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.782655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781808    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-var-lib-cni-multus\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.782655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781843    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx4wq\" (UniqueName: \"kubernetes.io/projected/a9adcfc8-62d7-451e-9756-510427c1434a-kube-api-access-sx4wq\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.782655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781876    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29b51897-d00a-432c-92eb-29f6a9059b20-system-cni-dir\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.782655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781915    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/29b51897-d00a-432c-92eb-29f6a9059b20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.782655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781948    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcw7t\" (UniqueName: \"kubernetes.io/projected/2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe-kube-api-access-hcw7t\") pod \"iptables-alerter-brglq\" (UID: \"2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe\") " pod="openshift-network-operator/iptables-alerter-brglq"
Sep 16 14:14:11.782655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.781986    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-run\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.782655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.782027    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-var-lib-kubelet\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.782655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.782061    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe-host-slash\") pod \"iptables-alerter-brglq\" (UID: \"2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe\") " pod="openshift-network-operator/iptables-alerter-brglq"
Sep 16 14:14:11.782655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.782083    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-sysconfig\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.782655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.782412    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.783215 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.782826    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 14:14:11.783215 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.782885    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 14:14:11.783215 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.782824    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-78lrp\""
Sep 16 14:14:11.784141 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.784123    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s79nx"
Sep 16 14:14:11.784798 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.784781    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 14:14:11.785066 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.785047    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 14:14:11.785144 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.785094    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 14:14:11.785214 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.785047    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ppct7\""
Sep 16 14:14:11.786334 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.786315    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 14:14:11.786334 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.786327    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 14:14:11.786480 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.786396    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 14:14:11.786740 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.786724    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 14:14:11.786812 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.786752    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 14:14:11.786812 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.786757    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9qxnn\""
Sep 16 14:14:11.807088 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.807072    2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kjlnq"
Sep 16 14:14:11.815953 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.815938    2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kjlnq"
Sep 16 14:14:11.870510 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.870490    2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 14:14:11.883651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883312    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29b51897-d00a-432c-92eb-29f6a9059b20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.883651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883372    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.883651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883413    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-hostroot\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.883651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883448    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29b51897-d00a-432c-92eb-29f6a9059b20-cnibin\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.883651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883482    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-sysctl-conf\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.883651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883472    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29b51897-d00a-432c-92eb-29f6a9059b20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.883651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883507    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-tuned\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.883651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883539    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxqhb\" (UniqueName: \"kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb\") pod \"network-check-target-ttfnh\" (UID: \"163028d4-d9b5-4feb-9c49-3af6793693c4\") " pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:11.883651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883576    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-cnibin\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.883651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883558    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-hostroot\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.883651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883612    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29b51897-d00a-432c-92eb-29f6a9059b20-cni-binary-copy\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.883651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883648    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c23242c8-2ee7-4041-b310-6538da68c470-dbus\") pod \"global-pull-secret-syncer-kdnvs\" (UID: \"c23242c8-2ee7-4041-b310-6538da68c470\") " pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883678    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883717    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t25g8\" (UniqueName: \"kubernetes.io/projected/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-kube-api-access-t25g8\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883753    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-system-cni-dir\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883785    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9adcfc8-62d7-451e-9756-510427c1434a-cni-binary-copy\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883812    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-run-netns\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883834    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-env-overrides\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883863    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g66b\" (UniqueName: \"kubernetes.io/projected/5cb644a0-2a32-41ef-aee9-7e4943300ac4-kube-api-access-5g66b\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883921    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff4hn\" (UniqueName: \"kubernetes.io/projected/29b51897-d00a-432c-92eb-29f6a9059b20-kube-api-access-ff4hn\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883954    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-run-k8s-cni-cncf-io\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.883984    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-var-lib-cni-multus\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884011    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx4wq\" (UniqueName: \"kubernetes.io/projected/a9adcfc8-62d7-451e-9756-510427c1434a-kube-api-access-sx4wq\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884041    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29b51897-d00a-432c-92eb-29f6a9059b20-system-cni-dir\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884074    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcw7t\" (UniqueName: \"kubernetes.io/projected/2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe-kube-api-access-hcw7t\") pod \"iptables-alerter-brglq\" (UID: \"2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe\") " pod="openshift-network-operator/iptables-alerter-brglq"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884104    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-slash\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884106    2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884135    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-run-systemd\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.884295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884161    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-etc-openvswitch\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884192    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884221    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cb644a0-2a32-41ef-aee9-7e4943300ac4-tmp\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884249    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/279e9549-530e-4e54-afbe-9c0954c59441-host\") pod \"node-ca-255dt\" (UID: \"279e9549-530e-4e54-afbe-9c0954c59441\") " pod="openshift-image-registry/node-ca-255dt"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884278    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvphn\" (UniqueName: \"kubernetes.io/projected/6fb3478f-9802-48c0-96a3-9bcb4283e030-kube-api-access-wvphn\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884303    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/345b6071-6885-4fa9-8a8f-cc1b40368688-konnectivity-ca\") pod \"konnectivity-agent-kswqv\" (UID: \"345b6071-6885-4fa9-8a8f-cc1b40368688\") " pod="kube-system/konnectivity-agent-kswqv"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884333    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-systemd-units\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884379    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-run-netns\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884409    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9adcfc8-62d7-451e-9756-510427c1434a-multus-daemon-config\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884443    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-lib-modules\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884470    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c23242c8-2ee7-4041-b310-6538da68c470-kubelet-config\") pod \"global-pull-secret-syncer-kdnvs\" (UID: \"c23242c8-2ee7-4041-b310-6538da68c470\") " pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884501    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-device-dir\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884534    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-ovn-node-metrics-cert\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884561    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/846a7905-fecc-4b70-b055-ee249dde8cb6-hosts-file\") pod \"node-resolver-s79nx\" (UID: \"846a7905-fecc-4b70-b055-ee249dde8cb6\") " pod="openshift-dns/node-resolver-s79nx"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884590    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-etc-kubernetes\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884620    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-modprobe-d\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884643    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-sys\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.885000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884670    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-host\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884697    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/345b6071-6885-4fa9-8a8f-cc1b40368688-agent-certs\") pod \"konnectivity-agent-kswqv\" (UID: \"345b6071-6885-4fa9-8a8f-cc1b40368688\") " pod="kube-system/konnectivity-agent-kswqv"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884727    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-var-lib-cni-bin\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884757    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-var-lib-kubelet\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884782    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs\") pod \"network-metrics-daemon-5shg9\" (UID: \"4e432d05-f97c-4799-8a7c-d660680b6298\") " pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884811    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-log-socket\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884841    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-ovnkube-script-lib\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884869    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/846a7905-fecc-4b70-b055-ee249dde8cb6-tmp-dir\") pod \"node-resolver-s79nx\" (UID: \"846a7905-fecc-4b70-b055-ee249dde8cb6\") " pod="openshift-dns/node-resolver-s79nx"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884914    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-multus-cni-dir\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884941    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-multus-socket-dir-parent\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.884993    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-multus-conf-dir\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885029    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-sysctl-d\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885072    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt77z\" (UniqueName: \"kubernetes.io/projected/279e9549-530e-4e54-afbe-9c0954c59441-kube-api-access-xt77z\") pod \"node-ca-255dt\" (UID: \"279e9549-530e-4e54-afbe-9c0954c59441\") " pod="openshift-image-registry/node-ca-255dt"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885110    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwd9\" (UniqueName: \"kubernetes.io/projected/4e432d05-f97c-4799-8a7c-d660680b6298-kube-api-access-lzwd9\") pod \"network-metrics-daemon-5shg9\" (UID: \"4e432d05-f97c-4799-8a7c-d660680b6298\") " pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885140    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-run-ovn\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885169    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-ovnkube-config\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885202    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/29b51897-d00a-432c-92eb-29f6a9059b20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.885683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885232    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-socket-dir\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885255    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-run-openvswitch\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885285    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret\") pod \"global-pull-secret-syncer-kdnvs\" (UID: \"c23242c8-2ee7-4041-b310-6538da68c470\") " pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885320    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/29b51897-d00a-432c-92eb-29f6a9059b20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885350    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-run\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885378    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-etc-selinux\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885402    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgbz2\" (UniqueName: \"kubernetes.io/projected/846a7905-fecc-4b70-b055-ee249dde8cb6-kube-api-access-qgbz2\") pod \"node-resolver-s79nx\" (UID: \"846a7905-fecc-4b70-b055-ee249dde8cb6\") " pod="openshift-dns/node-resolver-s79nx"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885434    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-var-lib-kubelet\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885555    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe-host-slash\") pod \"iptables-alerter-brglq\" (UID: \"2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe\") " pod="openshift-network-operator/iptables-alerter-brglq"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885588    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-sysconfig\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885616    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-run-multus-certs\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885650    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-systemd\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885717    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/279e9549-530e-4e54-afbe-9c0954c59441-serviceca\") pod \"node-ca-255dt\" (UID: \"279e9549-530e-4e54-afbe-9c0954c59441\") " pod="openshift-image-registry/node-ca-255dt"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885793    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-sys-fs\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885819    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-var-lib-openvswitch\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885849    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-cni-bin\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885881    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe-iptables-alerter-script\") pod \"iptables-alerter-brglq\" (UID: \"2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe\") " pod="openshift-network-operator/iptables-alerter-brglq"
Sep 16 14:14:11.886525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885927    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-kubernetes\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.887305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.885991    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-registration-dir\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.887305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.886015    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-kubelet\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.887305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.886042    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-node-log\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.887305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.886069    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-cni-netd\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.887305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.886235    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-cnibin\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.887305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.886313    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-var-lib-cni-bin\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.887305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.886353    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-var-lib-kubelet\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.887305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.887068    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-run-k8s-cni-cncf-io\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.887305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.887138    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-run\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.887305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.887160    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-var-lib-cni-multus\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.887305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.887253    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-var-lib-kubelet\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.887305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.887264    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-sysctl-conf\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.887305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.887306    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe-host-slash\") pod \"iptables-alerter-brglq\" (UID: \"2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe\") " pod="openshift-network-operator/iptables-alerter-brglq"
Sep 16 14:14:11.888225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.887366    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-sysconfig\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.888225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.887412    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-run-multus-certs\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.888225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.887456    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29b51897-d00a-432c-92eb-29f6a9059b20-system-cni-dir\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.888225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.887491    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-systemd\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.888225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.888025    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/279e9549-530e-4e54-afbe-9c0954c59441-serviceca\") pod \"node-ca-255dt\" (UID: \"279e9549-530e-4e54-afbe-9c0954c59441\") " pod="openshift-image-registry/node-ca-255dt"
Sep 16 14:14:11.888225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.888041    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/29b51897-d00a-432c-92eb-29f6a9059b20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.888225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.888066    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29b51897-d00a-432c-92eb-29f6a9059b20-cni-binary-copy\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.888248    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-tuned\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.888329    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29b51897-d00a-432c-92eb-29f6a9059b20-cnibin\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.888340    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/279e9549-530e-4e54-afbe-9c0954c59441-host\") pod \"node-ca-255dt\" (UID: \"279e9549-530e-4e54-afbe-9c0954c59441\") " pod="openshift-image-registry/node-ca-255dt"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.888428    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-host-run-netns\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.888441    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-lib-modules\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.888548    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-system-cni-dir\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.888755    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-kubernetes\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.888783    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-etc-kubernetes\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.888837    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-multus-conf-dir\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.888906    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-host\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.888991    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-modprobe-d\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.889073    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-sys\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.889138    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9adcfc8-62d7-451e-9756-510427c1434a-cni-binary-copy\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.889173    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cb644a0-2a32-41ef-aee9-7e4943300ac4-etc-sysctl-d\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.889343    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cb644a0-2a32-41ef-aee9-7e4943300ac4-tmp\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.889359    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe-iptables-alerter-script\") pod \"iptables-alerter-brglq\" (UID: \"2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe\") " pod="openshift-network-operator/iptables-alerter-brglq"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.889436    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-multus-socket-dir-parent\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.889669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.889503    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9adcfc8-62d7-451e-9756-510427c1434a-multus-cni-dir\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.890612 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.889508    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/29b51897-d00a-432c-92eb-29f6a9059b20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.890612 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.890071    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9adcfc8-62d7-451e-9756-510427c1434a-multus-daemon-config\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.895923 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.895881    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff4hn\" (UniqueName: \"kubernetes.io/projected/29b51897-d00a-432c-92eb-29f6a9059b20-kube-api-access-ff4hn\") pod \"multus-additional-cni-plugins-jccdd\" (UID: \"29b51897-d00a-432c-92eb-29f6a9059b20\") " pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:11.896005 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.895983    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g66b\" (UniqueName: \"kubernetes.io/projected/5cb644a0-2a32-41ef-aee9-7e4943300ac4-kube-api-access-5g66b\") pod \"tuned-fkk7b\" (UID: \"5cb644a0-2a32-41ef-aee9-7e4943300ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:11.896910 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.896872    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx4wq\" (UniqueName: \"kubernetes.io/projected/a9adcfc8-62d7-451e-9756-510427c1434a-kube-api-access-sx4wq\") pod \"multus-8b5hz\" (UID: \"a9adcfc8-62d7-451e-9756-510427c1434a\") " pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:11.898482 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.898466    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt77z\" (UniqueName: \"kubernetes.io/projected/279e9549-530e-4e54-afbe-9c0954c59441-kube-api-access-xt77z\") pod \"node-ca-255dt\" (UID: \"279e9549-530e-4e54-afbe-9c0954c59441\") " pod="openshift-image-registry/node-ca-255dt"
Sep 16 14:14:11.899459 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.899438    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcw7t\" (UniqueName: \"kubernetes.io/projected/2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe-kube-api-access-hcw7t\") pod \"iptables-alerter-brglq\" (UID: \"2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe\") " pod="openshift-network-operator/iptables-alerter-brglq"
Sep 16 14:14:11.917401 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.917356    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-50.ec2.internal" event={"ID":"cc9a0b5b683b9efc648017b7df9cf169","Type":"ContainerStarted","Data":"1b6159100b748bff5095617ebd7800630d05bc1a1fae08699d7e0a7d9330e8fb"}
Sep 16 14:14:11.918561 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.918537    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal" event={"ID":"fd7d5957c3c1631a2eca1f67dcc24ff3","Type":"ContainerStarted","Data":"cfe11d3cdf352b27104b6bd7411b3a0a59b4e26e65e80d950a9a29672c16d9ee"}
Sep 16 14:14:11.986706 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.986634    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwd9\" (UniqueName: \"kubernetes.io/projected/4e432d05-f97c-4799-8a7c-d660680b6298-kube-api-access-lzwd9\") pod \"network-metrics-daemon-5shg9\" (UID: \"4e432d05-f97c-4799-8a7c-d660680b6298\") " pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:11.986706 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.986676    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-run-ovn\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.986706 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.986702    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-ovnkube-config\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.986727    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-socket-dir\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.987036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.986752    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-run-openvswitch\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.986777    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret\") pod \"global-pull-secret-syncer-kdnvs\" (UID: \"c23242c8-2ee7-4041-b310-6538da68c470\") " pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:11.987036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.986803    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-etc-selinux\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.987036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.986826    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgbz2\" (UniqueName: \"kubernetes.io/projected/846a7905-fecc-4b70-b055-ee249dde8cb6-kube-api-access-qgbz2\") pod \"node-resolver-s79nx\" (UID: \"846a7905-fecc-4b70-b055-ee249dde8cb6\") " pod="openshift-dns/node-resolver-s79nx"
Sep 16 14:14:11.987036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.986854    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-sys-fs\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.987036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.986904    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-var-lib-openvswitch\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.986928    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-cni-bin\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.986776    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-run-ovn\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.986956    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-registration-dir\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.987036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987016    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-kubelet\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987024    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-registration-dir\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987057    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-node-log\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987080    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-kubelet\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987087    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-cni-netd\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987113    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987145    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqhb\" (UniqueName: \"kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb\") pod \"network-check-target-ttfnh\" (UID: \"163028d4-d9b5-4feb-9c49-3af6793693c4\") " pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987154    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-etc-selinux\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987172    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c23242c8-2ee7-4041-b310-6538da68c470-dbus\") pod \"global-pull-secret-syncer-kdnvs\" (UID: \"c23242c8-2ee7-4041-b310-6538da68c470\") " pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987198    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987225    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t25g8\" (UniqueName: \"kubernetes.io/projected/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-kube-api-access-t25g8\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987252    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-run-netns\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987276    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-env-overrides\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987323    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-slash\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987348    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-run-systemd\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987373    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-etc-openvswitch\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987384    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-run-openvswitch\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987396    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.987604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987426    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvphn\" (UniqueName: \"kubernetes.io/projected/6fb3478f-9802-48c0-96a3-9bcb4283e030-kube-api-access-wvphn\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987430    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-var-lib-openvswitch\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987452    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/345b6071-6885-4fa9-8a8f-cc1b40368688-konnectivity-ca\") pod \"konnectivity-agent-kswqv\" (UID: \"345b6071-6885-4fa9-8a8f-cc1b40368688\") " pod="kube-system/konnectivity-agent-kswqv"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:11.987466    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987480    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-systemd-units\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987483    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-socket-dir\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987498    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-node-log\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987510    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c23242c8-2ee7-4041-b310-6538da68c470-kubelet-config\") pod \"global-pull-secret-syncer-kdnvs\" (UID: \"c23242c8-2ee7-4041-b310-6538da68c470\") " pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987512    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-cni-netd\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987534    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-device-dir\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987566    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987575    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-ovn-node-metrics-cert\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987582    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-cni-bin\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987058    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-sys-fs\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987599    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/846a7905-fecc-4b70-b055-ee249dde8cb6-hosts-file\") pod \"node-resolver-s79nx\" (UID: \"846a7905-fecc-4b70-b055-ee249dde8cb6\") " pod="openshift-dns/node-resolver-s79nx"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987609    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987471    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c23242c8-2ee7-4041-b310-6538da68c470-dbus\") pod \"global-pull-secret-syncer-kdnvs\" (UID: \"c23242c8-2ee7-4041-b310-6538da68c470\") " pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987631    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-ovnkube-config\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.988389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987640    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/345b6071-6885-4fa9-8a8f-cc1b40368688-agent-certs\") pod \"konnectivity-agent-kswqv\" (UID: \"345b6071-6885-4fa9-8a8f-cc1b40368688\") " pod="kube-system/konnectivity-agent-kswqv"
Sep 16 14:14:11.989232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987688    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-run-systemd\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.989232 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:11.987704    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret podName:c23242c8-2ee7-4041-b310-6538da68c470 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:12.487668737 +0000 UTC m=+3.034410654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret") pod "global-pull-secret-syncer-kdnvs" (UID: "c23242c8-2ee7-4041-b310-6538da68c470") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:11.989232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987726    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs\") pod \"network-metrics-daemon-5shg9\" (UID: \"4e432d05-f97c-4799-8a7c-d660680b6298\") " pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:11.989232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987734    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-device-dir\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.989232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987765    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-slash\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.989232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987642    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/846a7905-fecc-4b70-b055-ee249dde8cb6-hosts-file\") pod \"node-resolver-s79nx\" (UID: \"846a7905-fecc-4b70-b055-ee249dde8cb6\") " pod="openshift-dns/node-resolver-s79nx"
Sep 16 14:14:11.989232 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:11.987819    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:11.989232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987836    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-host-run-netns\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.989232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987838    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-etc-openvswitch\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.989232 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:11.987864    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs podName:4e432d05-f97c-4799-8a7c-d660680b6298 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:12.487850813 +0000 UTC m=+3.034592735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs") pod "network-metrics-daemon-5shg9" (UID: "4e432d05-f97c-4799-8a7c-d660680b6298") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:11.989232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987552    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6fb3478f-9802-48c0-96a3-9bcb4283e030-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:11.989232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987875    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-systemd-units\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.989232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987925    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-log-socket\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.989232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987934    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c23242c8-2ee7-4041-b310-6538da68c470-kubelet-config\") pod \"global-pull-secret-syncer-kdnvs\" (UID: \"c23242c8-2ee7-4041-b310-6538da68c470\") " pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:11.989232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987951    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-ovnkube-script-lib\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.989232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.987977    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/846a7905-fecc-4b70-b055-ee249dde8cb6-tmp-dir\") pod \"node-resolver-s79nx\" (UID: \"846a7905-fecc-4b70-b055-ee249dde8cb6\") " pod="openshift-dns/node-resolver-s79nx"
Sep 16 14:14:11.990005 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.988016    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-log-socket\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.990005 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.988080    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-env-overrides\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.990005 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.988288    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/846a7905-fecc-4b70-b055-ee249dde8cb6-tmp-dir\") pod \"node-resolver-s79nx\" (UID: \"846a7905-fecc-4b70-b055-ee249dde8cb6\") " pod="openshift-dns/node-resolver-s79nx"
Sep 16 14:14:11.990005 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.988427    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/345b6071-6885-4fa9-8a8f-cc1b40368688-konnectivity-ca\") pod \"konnectivity-agent-kswqv\" (UID: \"345b6071-6885-4fa9-8a8f-cc1b40368688\") " pod="kube-system/konnectivity-agent-kswqv"
Sep 16 14:14:11.990005 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.988494    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-ovnkube-script-lib\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.990754 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.990731    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/345b6071-6885-4fa9-8a8f-cc1b40368688-agent-certs\") pod \"konnectivity-agent-kswqv\" (UID: \"345b6071-6885-4fa9-8a8f-cc1b40368688\") " pod="kube-system/konnectivity-agent-kswqv"
Sep 16 14:14:11.992305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:11.992121    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-ovn-node-metrics-cert\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:11.998181 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:11.998162    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:14:11.998181 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:11.998182    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:14:11.998402 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:11.998194    2553 projected.go:196] Error preparing data for projected volume kube-api-access-gxqhb for pod openshift-network-diagnostics/network-check-target-ttfnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:11.998402 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:11.998236    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb podName:163028d4-d9b5-4feb-9c49-3af6793693c4 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:12.498224488 +0000 UTC m=+3.044966396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gxqhb" (UniqueName: "kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb") pod "network-check-target-ttfnh" (UID: "163028d4-d9b5-4feb-9c49-3af6793693c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:12.002068 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.002044    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvphn\" (UniqueName: \"kubernetes.io/projected/6fb3478f-9802-48c0-96a3-9bcb4283e030-kube-api-access-wvphn\") pod \"aws-ebs-csi-driver-node-bcss7\" (UID: \"6fb3478f-9802-48c0-96a3-9bcb4283e030\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:12.002787 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.002543    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwd9\" (UniqueName: \"kubernetes.io/projected/4e432d05-f97c-4799-8a7c-d660680b6298-kube-api-access-lzwd9\") pod \"network-metrics-daemon-5shg9\" (UID: \"4e432d05-f97c-4799-8a7c-d660680b6298\") " pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:12.004163 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.004113    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t25g8\" (UniqueName: \"kubernetes.io/projected/c26c7c6d-beb0-47be-9d7f-d57f0772d6e3-kube-api-access-t25g8\") pod \"ovnkube-node-lnrqf\" (UID: \"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:12.004163 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.004125    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgbz2\" (UniqueName: \"kubernetes.io/projected/846a7905-fecc-4b70-b055-ee249dde8cb6-kube-api-access-qgbz2\") pod \"node-resolver-s79nx\" (UID: \"846a7905-fecc-4b70-b055-ee249dde8cb6\") " pod="openshift-dns/node-resolver-s79nx"
Sep 16 14:14:12.077704 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.077637    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-brglq"
Sep 16 14:14:12.085626 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.085562    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8b5hz"
Sep 16 14:14:12.093830 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.093653    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fkk7b"
Sep 16 14:14:12.098195 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.098146    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-255dt"
Sep 16 14:14:12.103234 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.103213    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jccdd"
Sep 16 14:14:12.108316 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.108152    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7"
Sep 16 14:14:12.113240 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.113222    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kswqv"
Sep 16 14:14:12.117960 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.117886    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:12.121797 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.121606    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s79nx"
Sep 16 14:14:12.304566 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:12.304337    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed9360d_0c6a_4b41_9f4e_eaf22a1eb0fe.slice/crio-e828c90a2f4d4ffb6912474b29bc3f9da7d69d73c4eaa19642b790284193b2a5 WatchSource:0}: Error finding container e828c90a2f4d4ffb6912474b29bc3f9da7d69d73c4eaa19642b790284193b2a5: Status 404 returned error can't find the container with id e828c90a2f4d4ffb6912474b29bc3f9da7d69d73c4eaa19642b790284193b2a5
Sep 16 14:14:12.305766 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:12.305647    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb3478f_9802_48c0_96a3_9bcb4283e030.slice/crio-591ff4969e7fb72720b5da3303ef3e89b60c1b2d649b3d54c7a24bed1d568474 WatchSource:0}: Error finding container 591ff4969e7fb72720b5da3303ef3e89b60c1b2d649b3d54c7a24bed1d568474: Status 404 returned error can't find the container with id 591ff4969e7fb72720b5da3303ef3e89b60c1b2d649b3d54c7a24bed1d568474
Sep 16 14:14:12.312131 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:12.311294    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b51897_d00a_432c_92eb_29f6a9059b20.slice/crio-af78224bbccd0345c8ce774d166d4212a465f671c4e0a5dafe9009d1bb9b039c WatchSource:0}: Error finding container af78224bbccd0345c8ce774d166d4212a465f671c4e0a5dafe9009d1bb9b039c: Status 404 returned error can't find the container with id af78224bbccd0345c8ce774d166d4212a465f671c4e0a5dafe9009d1bb9b039c
Sep 16 14:14:12.312696 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:12.312401    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345b6071_6885_4fa9_8a8f_cc1b40368688.slice/crio-a49c91c5c8070b39b6cef4a9f2f36e86b9d249816832e453e66ee9524387ac73 WatchSource:0}: Error finding container a49c91c5c8070b39b6cef4a9f2f36e86b9d249816832e453e66ee9524387ac73: Status 404 returned error can't find the container with id a49c91c5c8070b39b6cef4a9f2f36e86b9d249816832e453e66ee9524387ac73
Sep 16 14:14:12.313815 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:12.313141    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9adcfc8_62d7_451e_9756_510427c1434a.slice/crio-c39c3bf16d0f21d9fdeec29d708766630456b66f361203b8a8aa515194ee5a44 WatchSource:0}: Error finding container c39c3bf16d0f21d9fdeec29d708766630456b66f361203b8a8aa515194ee5a44: Status 404 returned error can't find the container with id c39c3bf16d0f21d9fdeec29d708766630456b66f361203b8a8aa515194ee5a44
Sep 16 14:14:12.313815 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:12.313405    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26c7c6d_beb0_47be_9d7f_d57f0772d6e3.slice/crio-493eae77023f27fa0357459e075918250e1b7dc848bfa9c3b5f0be34c8f00f23 WatchSource:0}: Error finding container 493eae77023f27fa0357459e075918250e1b7dc848bfa9c3b5f0be34c8f00f23: Status 404 returned error can't find the container with id 493eae77023f27fa0357459e075918250e1b7dc848bfa9c3b5f0be34c8f00f23
Sep 16 14:14:12.314876 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:12.314855    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279e9549_530e_4e54_afbe_9c0954c59441.slice/crio-03d652ff46940881d2e40a86b95b70e112e0d1bd020d6701290741b6593a7480 WatchSource:0}: Error finding container 03d652ff46940881d2e40a86b95b70e112e0d1bd020d6701290741b6593a7480: Status 404 returned error can't find the container with id 03d652ff46940881d2e40a86b95b70e112e0d1bd020d6701290741b6593a7480
Sep 16 14:14:12.491579 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.491382    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs\") pod \"network-metrics-daemon-5shg9\" (UID: \"4e432d05-f97c-4799-8a7c-d660680b6298\") " pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:12.491579 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.491443    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret\") pod \"global-pull-secret-syncer-kdnvs\" (UID: \"c23242c8-2ee7-4041-b310-6538da68c470\") " pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:12.491579 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:12.491557    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:12.491881 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:12.491629    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs podName:4e432d05-f97c-4799-8a7c-d660680b6298 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:13.49161191 +0000 UTC m=+4.038353823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs") pod "network-metrics-daemon-5shg9" (UID: "4e432d05-f97c-4799-8a7c-d660680b6298") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:12.491881 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:12.491560    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:12.491881 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:12.491724    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret podName:c23242c8-2ee7-4041-b310-6538da68c470 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:13.491707452 +0000 UTC m=+4.038449361 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret") pod "global-pull-secret-syncer-kdnvs" (UID: "c23242c8-2ee7-4041-b310-6538da68c470") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:12.573856 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.573834    2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:14:12.592305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.592261    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqhb\" (UniqueName: \"kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb\") pod \"network-check-target-ttfnh\" (UID: \"163028d4-d9b5-4feb-9c49-3af6793693c4\") " pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:12.592453 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:12.592432    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:14:12.592513 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:12.592463    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:14:12.592513 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:12.592477    2553 projected.go:196] Error preparing data for projected volume kube-api-access-gxqhb for pod openshift-network-diagnostics/network-check-target-ttfnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:12.592589 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:12.592531    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb podName:163028d4-d9b5-4feb-9c49-3af6793693c4 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:13.592513419 +0000 UTC m=+4.139255330 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gxqhb" (UniqueName: "kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb") pod "network-check-target-ttfnh" (UID: "163028d4-d9b5-4feb-9c49-3af6793693c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:12.818053 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.817948    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 14:09:11 +0000 UTC" deadline="2028-05-27 00:20:42.149511758 +0000 UTC"
Sep 16 14:14:12.818053 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.817998    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14842h6m29.331516278s"
Sep 16 14:14:12.911436 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.910953    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:12.911436 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:12.911099    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdnvs" podUID="c23242c8-2ee7-4041-b310-6538da68c470"
Sep 16 14:14:12.923080 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.922991    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8b5hz" event={"ID":"a9adcfc8-62d7-451e-9756-510427c1434a","Type":"ContainerStarted","Data":"c39c3bf16d0f21d9fdeec29d708766630456b66f361203b8a8aa515194ee5a44"}
Sep 16 14:14:12.924632 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.924542    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf" event={"ID":"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3","Type":"ContainerStarted","Data":"493eae77023f27fa0357459e075918250e1b7dc848bfa9c3b5f0be34c8f00f23"}
Sep 16 14:14:12.926324 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.926258    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kswqv" event={"ID":"345b6071-6885-4fa9-8a8f-cc1b40368688","Type":"ContainerStarted","Data":"a49c91c5c8070b39b6cef4a9f2f36e86b9d249816832e453e66ee9524387ac73"}
Sep 16 14:14:12.927993 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.927960    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jccdd" event={"ID":"29b51897-d00a-432c-92eb-29f6a9059b20","Type":"ContainerStarted","Data":"af78224bbccd0345c8ce774d166d4212a465f671c4e0a5dafe9009d1bb9b039c"}
Sep 16 14:14:12.929750 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.929709    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-brglq" event={"ID":"2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe","Type":"ContainerStarted","Data":"e828c90a2f4d4ffb6912474b29bc3f9da7d69d73c4eaa19642b790284193b2a5"}
Sep 16 14:14:12.931332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.931301    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-255dt" event={"ID":"279e9549-530e-4e54-afbe-9c0954c59441","Type":"ContainerStarted","Data":"03d652ff46940881d2e40a86b95b70e112e0d1bd020d6701290741b6593a7480"}
Sep 16 14:14:12.933020 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.932963    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s79nx" event={"ID":"846a7905-fecc-4b70-b055-ee249dde8cb6","Type":"ContainerStarted","Data":"a8d2e5636d48a2bc1319e22b996e5c047eb0ee503932f1c8ee3098fa5bb3801c"}
Sep 16 14:14:12.934365 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.934343    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7" event={"ID":"6fb3478f-9802-48c0-96a3-9bcb4283e030","Type":"ContainerStarted","Data":"591ff4969e7fb72720b5da3303ef3e89b60c1b2d649b3d54c7a24bed1d568474"}
Sep 16 14:14:12.936877 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.936851    2553 generic.go:356] "Generic (PLEG): container finished" podID="fd7d5957c3c1631a2eca1f67dcc24ff3" containerID="72e4a13847282d1935db5a87250e14eba031365a08c86c33e1ee63783bed3fe9" exitCode=0
Sep 16 14:14:12.937454 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.937429    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal" event={"ID":"fd7d5957c3c1631a2eca1f67dcc24ff3","Type":"ContainerDied","Data":"72e4a13847282d1935db5a87250e14eba031365a08c86c33e1ee63783bed3fe9"}
Sep 16 14:14:12.939046 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:12.938997    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fkk7b" event={"ID":"5cb644a0-2a32-41ef-aee9-7e4943300ac4","Type":"ContainerStarted","Data":"6c7f0d53572f495590c177ab9d13ff62a774f58badea04ffdc5a1990745291ff"}
Sep 16 14:14:13.499945 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:13.499046    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs\") pod \"network-metrics-daemon-5shg9\" (UID: \"4e432d05-f97c-4799-8a7c-d660680b6298\") " pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:13.499945 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:13.499131    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret\") pod \"global-pull-secret-syncer-kdnvs\" (UID: \"c23242c8-2ee7-4041-b310-6538da68c470\") " pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:13.499945 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:13.499328    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:13.499945 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:13.499392    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret podName:c23242c8-2ee7-4041-b310-6538da68c470 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:15.499374024 +0000 UTC m=+6.046115952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret") pod "global-pull-secret-syncer-kdnvs" (UID: "c23242c8-2ee7-4041-b310-6538da68c470") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:13.499945 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:13.499781    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:13.499945 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:13.499829    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs podName:4e432d05-f97c-4799-8a7c-d660680b6298 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:15.499813106 +0000 UTC m=+6.046555028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs") pod "network-metrics-daemon-5shg9" (UID: "4e432d05-f97c-4799-8a7c-d660680b6298") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:13.600943 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:13.600314    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqhb\" (UniqueName: \"kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb\") pod \"network-check-target-ttfnh\" (UID: \"163028d4-d9b5-4feb-9c49-3af6793693c4\") " pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:13.600943 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:13.600491    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:14:13.600943 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:13.600516    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:14:13.600943 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:13.600529    2553 projected.go:196] Error preparing data for projected volume kube-api-access-gxqhb for pod openshift-network-diagnostics/network-check-target-ttfnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:13.600943 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:13.600584    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb podName:163028d4-d9b5-4feb-9c49-3af6793693c4 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:15.600565878 +0000 UTC m=+6.147307792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gxqhb" (UniqueName: "kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb") pod "network-check-target-ttfnh" (UID: "163028d4-d9b5-4feb-9c49-3af6793693c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:13.818816 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:13.818663    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 14:09:11 +0000 UTC" deadline="2028-03-18 21:27:42.352963327 +0000 UTC"
Sep 16 14:14:13.818816 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:13.818717    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13183h13m28.534251118s"
Sep 16 14:14:13.912778 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:13.912005    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:13.912778 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:13.912132    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttfnh" podUID="163028d4-d9b5-4feb-9c49-3af6793693c4"
Sep 16 14:14:13.912778 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:13.912613    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:13.912778 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:13.912734    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5shg9" podUID="4e432d05-f97c-4799-8a7c-d660680b6298"
Sep 16 14:14:13.958532 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:13.958473    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal" event={"ID":"fd7d5957c3c1631a2eca1f67dcc24ff3","Type":"ContainerStarted","Data":"15999ce954383ff206caf59c9100c8e3cfdbe515bbb5d25886bc6f3c6a89028d"}
Sep 16 14:14:13.974769 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:13.963959    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-50.ec2.internal" event={"ID":"cc9a0b5b683b9efc648017b7df9cf169","Type":"ContainerStarted","Data":"3051cbf6c3260b71d7a7c611905a49d5fe332d540667ca54b82129a10fc27891"}
Sep 16 14:14:13.980321 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:13.980153    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-50.ec2.internal" podStartSLOduration=2.98013783 podStartE2EDuration="2.98013783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:13.980108775 +0000 UTC m=+4.526850707" watchObservedRunningTime="2026-09-16 14:14:13.98013783 +0000 UTC m=+4.526879764"
Sep 16 14:14:14.003999 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:14.003143    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-50.ec2.internal" podStartSLOduration=3.00312413 podStartE2EDuration="3.00312413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:14.003013943 +0000 UTC m=+4.549755900" watchObservedRunningTime="2026-09-16 14:14:14.00312413 +0000 UTC m=+4.549866062"
Sep 16 14:14:14.912056 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:14.911520    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:14.912056 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:14.911679    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdnvs" podUID="c23242c8-2ee7-4041-b310-6538da68c470"
Sep 16 14:14:15.515885 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:15.515848    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs\") pod \"network-metrics-daemon-5shg9\" (UID: \"4e432d05-f97c-4799-8a7c-d660680b6298\") " pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:15.516106 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:15.515920    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret\") pod \"global-pull-secret-syncer-kdnvs\" (UID: \"c23242c8-2ee7-4041-b310-6538da68c470\") " pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:15.516106 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:15.516040    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:15.516106 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:15.516098    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret podName:c23242c8-2ee7-4041-b310-6538da68c470 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:19.516080445 +0000 UTC m=+10.062822376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret") pod "global-pull-secret-syncer-kdnvs" (UID: "c23242c8-2ee7-4041-b310-6538da68c470") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:15.516490 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:15.516386    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:15.516490 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:15.516448    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs podName:4e432d05-f97c-4799-8a7c-d660680b6298 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:19.516436239 +0000 UTC m=+10.063178152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs") pod "network-metrics-daemon-5shg9" (UID: "4e432d05-f97c-4799-8a7c-d660680b6298") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:15.616905 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:15.616861    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqhb\" (UniqueName: \"kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb\") pod \"network-check-target-ttfnh\" (UID: \"163028d4-d9b5-4feb-9c49-3af6793693c4\") " pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:15.617079 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:15.617035    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:14:15.617079 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:15.617058    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:14:15.617079 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:15.617067    2553 projected.go:196] Error preparing data for projected volume kube-api-access-gxqhb for pod openshift-network-diagnostics/network-check-target-ttfnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:15.617199 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:15.617120    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb podName:163028d4-d9b5-4feb-9c49-3af6793693c4 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:19.617105893 +0000 UTC m=+10.163847823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gxqhb" (UniqueName: "kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb") pod "network-check-target-ttfnh" (UID: "163028d4-d9b5-4feb-9c49-3af6793693c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:15.911232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:15.911158    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:15.911394 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:15.911308    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttfnh" podUID="163028d4-d9b5-4feb-9c49-3af6793693c4"
Sep 16 14:14:15.911689 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:15.911667    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:15.911768 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:15.911752    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5shg9" podUID="4e432d05-f97c-4799-8a7c-d660680b6298"
Sep 16 14:14:16.911378 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:16.911318    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:16.911845 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:16.911452    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdnvs" podUID="c23242c8-2ee7-4041-b310-6538da68c470"
Sep 16 14:14:17.911653 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:17.911611    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:17.912125 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:17.911744    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttfnh" podUID="163028d4-d9b5-4feb-9c49-3af6793693c4"
Sep 16 14:14:17.912125 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:17.911806    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:17.912125 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:17.911943    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5shg9" podUID="4e432d05-f97c-4799-8a7c-d660680b6298"
Sep 16 14:14:18.911584 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:18.911552    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:18.911759 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:18.911684    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdnvs" podUID="c23242c8-2ee7-4041-b310-6538da68c470"
Sep 16 14:14:19.549791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:19.549758    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs\") pod \"network-metrics-daemon-5shg9\" (UID: \"4e432d05-f97c-4799-8a7c-d660680b6298\") " pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:19.549791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:19.549798    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret\") pod \"global-pull-secret-syncer-kdnvs\" (UID: \"c23242c8-2ee7-4041-b310-6538da68c470\") " pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:19.550054 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:19.549932    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:19.550054 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:19.549985    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret podName:c23242c8-2ee7-4041-b310-6538da68c470 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:27.549970211 +0000 UTC m=+18.096712121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret") pod "global-pull-secret-syncer-kdnvs" (UID: "c23242c8-2ee7-4041-b310-6538da68c470") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:19.550054 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:19.550034    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:19.550054 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:19.550057    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs podName:4e432d05-f97c-4799-8a7c-d660680b6298 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:27.550050718 +0000 UTC m=+18.096792627 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs") pod "network-metrics-daemon-5shg9" (UID: "4e432d05-f97c-4799-8a7c-d660680b6298") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:19.650497 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:19.650459    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqhb\" (UniqueName: \"kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb\") pod \"network-check-target-ttfnh\" (UID: \"163028d4-d9b5-4feb-9c49-3af6793693c4\") " pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:19.650727 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:19.650702    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:14:19.650848 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:19.650733    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:14:19.650848 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:19.650746    2553 projected.go:196] Error preparing data for projected volume kube-api-access-gxqhb for pod openshift-network-diagnostics/network-check-target-ttfnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:19.650848 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:19.650810    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb podName:163028d4-d9b5-4feb-9c49-3af6793693c4 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:27.650791083 +0000 UTC m=+18.197533013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gxqhb" (UniqueName: "kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb") pod "network-check-target-ttfnh" (UID: "163028d4-d9b5-4feb-9c49-3af6793693c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:19.911523 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:19.911443    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:19.912840 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:19.912479    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttfnh" podUID="163028d4-d9b5-4feb-9c49-3af6793693c4"
Sep 16 14:14:19.912840 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:19.912583    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:19.912840 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:19.912681    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5shg9" podUID="4e432d05-f97c-4799-8a7c-d660680b6298"
Sep 16 14:14:20.911057 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:20.911028    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:20.911235 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:20.911146    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdnvs" podUID="c23242c8-2ee7-4041-b310-6538da68c470"
Sep 16 14:14:21.911082 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:21.910998    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:21.911500 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:21.910998    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:21.911500 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:21.911147    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5shg9" podUID="4e432d05-f97c-4799-8a7c-d660680b6298"
Sep 16 14:14:21.911500 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:21.911167    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttfnh" podUID="163028d4-d9b5-4feb-9c49-3af6793693c4"
Sep 16 14:14:22.911783 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:22.911741    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:22.912257 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:22.911850    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdnvs" podUID="c23242c8-2ee7-4041-b310-6538da68c470"
Sep 16 14:14:23.911128 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:23.911094    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:23.911320 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:23.911234    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttfnh" podUID="163028d4-d9b5-4feb-9c49-3af6793693c4"
Sep 16 14:14:23.911320 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:23.911278    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:23.911429 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:23.911408    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5shg9" podUID="4e432d05-f97c-4799-8a7c-d660680b6298"
Sep 16 14:14:24.911093 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:24.911058    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:24.911526 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:24.911187    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdnvs" podUID="c23242c8-2ee7-4041-b310-6538da68c470"
Sep 16 14:14:25.910992 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:25.910957    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:25.911156 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:25.911078    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:25.911479 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:25.911186    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5shg9" podUID="4e432d05-f97c-4799-8a7c-d660680b6298"
Sep 16 14:14:25.911479 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:25.911074    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttfnh" podUID="163028d4-d9b5-4feb-9c49-3af6793693c4"
Sep 16 14:14:26.910748 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:26.910718    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:26.910976 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:26.910818    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdnvs" podUID="c23242c8-2ee7-4041-b310-6538da68c470"
Sep 16 14:14:27.608305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:27.608267    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs\") pod \"network-metrics-daemon-5shg9\" (UID: \"4e432d05-f97c-4799-8a7c-d660680b6298\") " pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:27.608305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:27.608316    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret\") pod \"global-pull-secret-syncer-kdnvs\" (UID: \"c23242c8-2ee7-4041-b310-6538da68c470\") " pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:27.608771 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:27.608452    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:27.608771 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:27.608495    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:27.608771 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:27.608529    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs podName:4e432d05-f97c-4799-8a7c-d660680b6298 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:43.60850731 +0000 UTC m=+34.155249230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs") pod "network-metrics-daemon-5shg9" (UID: "4e432d05-f97c-4799-8a7c-d660680b6298") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:27.608771 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:27.608550    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret podName:c23242c8-2ee7-4041-b310-6538da68c470 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:43.608540102 +0000 UTC m=+34.155282017 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret") pod "global-pull-secret-syncer-kdnvs" (UID: "c23242c8-2ee7-4041-b310-6538da68c470") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:27.709381 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:27.709342    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqhb\" (UniqueName: \"kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb\") pod \"network-check-target-ttfnh\" (UID: \"163028d4-d9b5-4feb-9c49-3af6793693c4\") " pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:27.709542 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:27.709487    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:14:27.709542 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:27.709504    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:14:27.709542 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:27.709514    2553 projected.go:196] Error preparing data for projected volume kube-api-access-gxqhb for pod openshift-network-diagnostics/network-check-target-ttfnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:27.709705 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:27.709583    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb podName:163028d4-d9b5-4feb-9c49-3af6793693c4 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:43.709569836 +0000 UTC m=+34.256311767 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gxqhb" (UniqueName: "kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb") pod "network-check-target-ttfnh" (UID: "163028d4-d9b5-4feb-9c49-3af6793693c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:27.911706 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:27.911621    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:27.911866 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:27.911626    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:27.911866 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:27.911759    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttfnh" podUID="163028d4-d9b5-4feb-9c49-3af6793693c4"
Sep 16 14:14:27.911866 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:27.911845    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5shg9" podUID="4e432d05-f97c-4799-8a7c-d660680b6298"
Sep 16 14:14:28.911599 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:28.911565    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:28.912027 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:28.911694    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdnvs" podUID="c23242c8-2ee7-4041-b310-6538da68c470"
Sep 16 14:14:29.911815 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:29.911627    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:29.912292 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:29.911679    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:29.912292 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:29.911900    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5shg9" podUID="4e432d05-f97c-4799-8a7c-d660680b6298"
Sep 16 14:14:29.912292 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:29.911956    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttfnh" podUID="163028d4-d9b5-4feb-9c49-3af6793693c4"
Sep 16 14:14:29.998159 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:29.998124    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fkk7b" event={"ID":"5cb644a0-2a32-41ef-aee9-7e4943300ac4","Type":"ContainerStarted","Data":"cd9af31f4b484ae17103048b4ee3325f1684aa0aaa8c6c88bc683321e43a046b"}
Sep 16 14:14:29.999867 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:29.999839    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8b5hz" event={"ID":"a9adcfc8-62d7-451e-9756-510427c1434a","Type":"ContainerStarted","Data":"a0bcaedaeb27ff422b96732ddc87d3fae466c9c6d4112e0bf3e37487f05606a1"}
Sep 16 14:14:30.003519 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.003493    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf" event={"ID":"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3","Type":"ContainerStarted","Data":"65780f99446d2c79475832a68944e06223258c080d07726060651f1ebc072c8a"}
Sep 16 14:14:30.003794 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.003561    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf" event={"ID":"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3","Type":"ContainerStarted","Data":"85a8eb929c2f99da98350420baf03975f380c91a8c2425aff3e36d33e2d3f47e"}
Sep 16 14:14:30.003794 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.003601    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf" event={"ID":"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3","Type":"ContainerStarted","Data":"e62c0b2be13dc558d134f720eb4956efe3cf0ca5793a35bfcaf22fbeba6db188"}
Sep 16 14:14:30.003794 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.003635    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf" event={"ID":"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3","Type":"ContainerStarted","Data":"59972c631aeb87c29f2a327ad60a161b4959197176c865329f46fd6d171be841"}
Sep 16 14:14:30.003794 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.003664    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf" event={"ID":"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3","Type":"ContainerStarted","Data":"dea1f6dfec96d75e090e52bcda3358c3e92d2f7c4d6c256e49e1ca64a32ab052"}
Sep 16 14:14:30.003794 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.003699    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf" event={"ID":"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3","Type":"ContainerStarted","Data":"18561113a74ac3ce7708b1ec38951562f6f06cbf81d4887b096590e046c30f7c"}
Sep 16 14:14:30.005063 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.005036    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kswqv" event={"ID":"345b6071-6885-4fa9-8a8f-cc1b40368688","Type":"ContainerStarted","Data":"947fa04f16065cf718de87a97b8425cee980e6accbdd03be0be5e72c86c2808a"}
Sep 16 14:14:30.006508 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.006487    2553 generic.go:356] "Generic (PLEG): container finished" podID="29b51897-d00a-432c-92eb-29f6a9059b20" containerID="4ba7a0341ed233592c156db23d7762440e73ebb494c53ed1df1454aa96fafafe" exitCode=0
Sep 16 14:14:30.006597 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.006555    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jccdd" event={"ID":"29b51897-d00a-432c-92eb-29f6a9059b20","Type":"ContainerDied","Data":"4ba7a0341ed233592c156db23d7762440e73ebb494c53ed1df1454aa96fafafe"}
Sep 16 14:14:30.008268 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.008241    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-255dt" event={"ID":"279e9549-530e-4e54-afbe-9c0954c59441","Type":"ContainerStarted","Data":"e46ea8ceb196e310e7d820eea49119eae65eda67ad4d0672602171d7f39d9db6"}
Sep 16 14:14:30.009737 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.009705    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s79nx" event={"ID":"846a7905-fecc-4b70-b055-ee249dde8cb6","Type":"ContainerStarted","Data":"721bdc8ba09a37091c4d2a8f19c968cb9eb8eb58a04ea3c5c974009c6bbc5821"}
Sep 16 14:14:30.011335 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.011312    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7" event={"ID":"6fb3478f-9802-48c0-96a3-9bcb4283e030","Type":"ContainerStarted","Data":"4703440a5106e3bcdc7fec792cb77c6092f83cda859b92670f844c0b4c93208d"}
Sep 16 14:14:30.021069 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.021031    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fkk7b" podStartSLOduration=2.8600826 podStartE2EDuration="20.021005129s" totalImagesPullingTime="17.160922529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:30.019715696 +0000 UTC m=+20.566457626" watchObservedRunningTime="2026-09-16 14:14:30.021005129 +0000 UTC m=+20.567747059"
Sep 16 14:14:30.073393 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.073349    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kswqv" podStartSLOduration=10.889415286 podStartE2EDuration="20.073338245s" totalImagesPullingTime="9.183922959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:30.044357314 +0000 UTC m=+20.591099255" watchObservedRunningTime="2026-09-16 14:14:30.073338245 +0000 UTC m=+20.620080176"
Sep 16 14:14:30.073672 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.073635    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8b5hz" podStartSLOduration=2.846765957 podStartE2EDuration="20.073631029s" totalImagesPullingTime="17.226865072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:30.073334912 +0000 UTC m=+20.620076845" watchObservedRunningTime="2026-09-16 14:14:30.073631029 +0000 UTC m=+20.620372959"
Sep 16 14:14:30.093386 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.093348    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-s79nx" podStartSLOduration=10.895595009 podStartE2EDuration="20.09333563s" totalImagesPullingTime="9.197740621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:30.093036159 +0000 UTC m=+20.639778089" watchObservedRunningTime="2026-09-16 14:14:30.09333563 +0000 UTC m=+20.640077561"
Sep 16 14:14:30.136192 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.136156    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-255dt" podStartSLOduration=10.949313635 podStartE2EDuration="20.136145144s" totalImagesPullingTime="9.186831509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:30.113223241 +0000 UTC m=+20.659965172" watchObservedRunningTime="2026-09-16 14:14:30.136145144 +0000 UTC m=+20.682887076"
Sep 16 14:14:30.848475 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.848280    2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 14:14:30.911551 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:30.911523    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:30.911693 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:30.911629    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdnvs" podUID="c23242c8-2ee7-4041-b310-6538da68c470"
Sep 16 14:14:31.015674 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:31.015637    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-brglq" event={"ID":"2ed9360d-0c6a-4b41-9f4e-eaf22a1eb0fe","Type":"ContainerStarted","Data":"35a177bc5071e80a13bc463a3f99ffc98c5490869a19a1a57e5f67fa498ef3a8"}
Sep 16 14:14:31.017674 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:31.017637    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7" event={"ID":"6fb3478f-9802-48c0-96a3-9bcb4283e030","Type":"ContainerStarted","Data":"459bf6a9e6217b7d3c4afc792ff46a3db9089887d1e3d076f9a6e8ef30a6fd1d"}
Sep 16 14:14:31.033147 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:31.033105    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-brglq" podStartSLOduration=11.837205049 podStartE2EDuration="21.033090073s" totalImagesPullingTime="9.195885024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:31.032842454 +0000 UTC m=+21.579584389" watchObservedRunningTime="2026-09-16 14:14:31.033090073 +0000 UTC m=+21.579832005"
Sep 16 14:14:31.833607 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:31.833495    2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T14:14:30.848473273Z","UUID":"c04e3937-0efd-4611-93fd-95dd55cc8475","Handler":null,"Name":"","Endpoint":""}
Sep 16 14:14:31.835351 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:31.835329    2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 14:14:31.835501 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:31.835364    2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 14:14:31.911974 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:31.911476    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:31.911974 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:31.911603    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttfnh" podUID="163028d4-d9b5-4feb-9c49-3af6793693c4"
Sep 16 14:14:31.912212 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:31.912108    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:31.912266 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:31.912226    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5shg9" podUID="4e432d05-f97c-4799-8a7c-d660680b6298"
Sep 16 14:14:31.972520 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:31.972406    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kswqv"
Sep 16 14:14:31.972885 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:31.972861    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kswqv"
Sep 16 14:14:32.020810 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:32.020704    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kswqv"
Sep 16 14:14:32.021239 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:32.021120    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kswqv"
Sep 16 14:14:32.911305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:32.911277    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:32.911466 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:32.911392    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdnvs" podUID="c23242c8-2ee7-4041-b310-6538da68c470"
Sep 16 14:14:33.024805 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:33.024762    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7" event={"ID":"6fb3478f-9802-48c0-96a3-9bcb4283e030","Type":"ContainerStarted","Data":"d28a65f5e6c01d6d23ac3ce06a2f1f322a24d45c87d6f50f210c99e312937829"}
Sep 16 14:14:33.028658 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:33.028634    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf" event={"ID":"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3","Type":"ContainerStarted","Data":"79e9b68809801cbf4662406ec96577dba3631bf2d2c8a04dfe17a924e127eb53"}
Sep 16 14:14:33.046919 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:33.046857    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcss7" podStartSLOduration=11.628798949 podStartE2EDuration="23.04683852s" totalImagesPullingTime="11.418039571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:33.046745266 +0000 UTC m=+23.593487198" watchObservedRunningTime="2026-09-16 14:14:33.04683852 +0000 UTC m=+23.593580452"
Sep 16 14:14:33.911445 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:33.911411    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:33.911725 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:33.911418    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:33.911725 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:33.911552    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5shg9" podUID="4e432d05-f97c-4799-8a7c-d660680b6298"
Sep 16 14:14:33.911725 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:33.911628    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttfnh" podUID="163028d4-d9b5-4feb-9c49-3af6793693c4"
Sep 16 14:14:34.911167 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:34.911137    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:34.911513 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:34.911236    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdnvs" podUID="c23242c8-2ee7-4041-b310-6538da68c470"
Sep 16 14:14:35.036069 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:35.036034    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf" event={"ID":"c26c7c6d-beb0-47be-9d7f-d57f0772d6e3","Type":"ContainerStarted","Data":"bdf71e9cd0f8c33d76499e390a181358c11e3e86211be6deb37aeca30b28dcba"}
Sep 16 14:14:35.036325 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:35.036289    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:35.038181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:35.038157    2553 generic.go:356] "Generic (PLEG): container finished" podID="29b51897-d00a-432c-92eb-29f6a9059b20" containerID="872380e27b9bcfab7c17ba9a72c7dd7c6961b1cb073b56a8650caac53afb2efe" exitCode=0
Sep 16 14:14:35.038292 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:35.038188    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jccdd" event={"ID":"29b51897-d00a-432c-92eb-29f6a9059b20","Type":"ContainerDied","Data":"872380e27b9bcfab7c17ba9a72c7dd7c6961b1cb073b56a8650caac53afb2efe"}
Sep 16 14:14:35.052328 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:35.052306    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:35.065056 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:35.065021    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf" podStartSLOduration=7.829915025 podStartE2EDuration="25.065006854s" totalImagesPullingTime="17.235091829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:35.064527162 +0000 UTC m=+25.611269092" watchObservedRunningTime="2026-09-16 14:14:35.065006854 +0000 UTC m=+25.611748785"
Sep 16 14:14:35.911858 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:35.911825    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:35.912342 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:35.911968    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttfnh" podUID="163028d4-d9b5-4feb-9c49-3af6793693c4"
Sep 16 14:14:35.912342 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:35.912016    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:35.912342 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:35.912113    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5shg9" podUID="4e432d05-f97c-4799-8a7c-d660680b6298"
Sep 16 14:14:36.041107 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:36.041076    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:36.041260 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:36.041156    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:36.042534 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:36.042508    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kdnvs"]
Sep 16 14:14:36.042625 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:36.042614    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:36.042737 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:36.042715    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdnvs" podUID="c23242c8-2ee7-4041-b310-6538da68c470"
Sep 16 14:14:36.046489 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:36.046412    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ttfnh"]
Sep 16 14:14:36.046607 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:36.046502    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:36.046607 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:36.046595    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttfnh" podUID="163028d4-d9b5-4feb-9c49-3af6793693c4"
Sep 16 14:14:36.047346 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:36.047324    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5shg9"]
Sep 16 14:14:36.047428 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:36.047416    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:36.047573 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:36.047549    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5shg9" podUID="4e432d05-f97c-4799-8a7c-d660680b6298"
Sep 16 14:14:36.057752 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:36.057732    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:14:37.911807 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:37.911778    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:37.912178 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:37.911776    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:37.912178 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:37.911881    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdnvs" podUID="c23242c8-2ee7-4041-b310-6538da68c470"
Sep 16 14:14:37.912178 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:37.911973    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttfnh" podUID="163028d4-d9b5-4feb-9c49-3af6793693c4"
Sep 16 14:14:37.912178 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:37.911770    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:37.912178 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:37.912053    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5shg9" podUID="4e432d05-f97c-4799-8a7c-d660680b6298"
Sep 16 14:14:39.049760 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:39.049560    2553 generic.go:356] "Generic (PLEG): container finished" podID="29b51897-d00a-432c-92eb-29f6a9059b20" containerID="c4d9ca25d2c146ef5caec2405cbb5796599f4476d9a269b3c7e9b7b1fe4d6bcc" exitCode=0
Sep 16 14:14:39.049760 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:39.049650    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jccdd" event={"ID":"29b51897-d00a-432c-92eb-29f6a9059b20","Type":"ContainerDied","Data":"c4d9ca25d2c146ef5caec2405cbb5796599f4476d9a269b3c7e9b7b1fe4d6bcc"}
Sep 16 14:14:39.912298 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:39.912266    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:39.912436 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:39.912354    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:39.912436 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:39.912391    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttfnh" podUID="163028d4-d9b5-4feb-9c49-3af6793693c4"
Sep 16 14:14:39.912510 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:39.912434    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kdnvs" podUID="c23242c8-2ee7-4041-b310-6538da68c470"
Sep 16 14:14:39.912510 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:39.912442    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:39.912587 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:39.912535    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5shg9" podUID="4e432d05-f97c-4799-8a7c-d660680b6298"
Sep 16 14:14:41.055935 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.055906    2553 generic.go:356] "Generic (PLEG): container finished" podID="29b51897-d00a-432c-92eb-29f6a9059b20" containerID="fb8ba67616b81b365d2bcf1e0272433c26bde26f1615647c08758e0db1ec625d" exitCode=0
Sep 16 14:14:41.056277 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.055969    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jccdd" event={"ID":"29b51897-d00a-432c-92eb-29f6a9059b20","Type":"ContainerDied","Data":"fb8ba67616b81b365d2bcf1e0272433c26bde26f1615647c08758e0db1ec625d"}
Sep 16 14:14:41.123941 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.123918    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-50.ec2.internal" event="NodeReady"
Sep 16 14:14:41.124054 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.124025    2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 14:14:41.168627 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.168600    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-598474997c-5r45z"]
Sep 16 14:14:41.190534 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.190512    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9x4k6"]
Sep 16 14:14:41.190654 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.190637    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.200207 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.200189    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-566s2\""
Sep 16 14:14:41.200297 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.200191    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 14:14:41.200453 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.200438    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 14:14:41.201031 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.201014    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 14:14:41.203089 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.203074    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 14:14:41.212581 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.212555    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-598474997c-5r45z"]
Sep 16 14:14:41.212581 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.212583    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x7sq6"]
Sep 16 14:14:41.212714 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.212654    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:41.217475 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.216747    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 14:14:41.217475 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.216837    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9f4mh\""
Sep 16 14:14:41.217475 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.217023    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 14:14:41.228581 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.228562    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x7sq6"]
Sep 16 14:14:41.228654 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.228590    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9x4k6"]
Sep 16 14:14:41.228698 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.228666    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x7sq6"
Sep 16 14:14:41.232485 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.232463    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 14:14:41.232811 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.232798    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 14:14:41.233699 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.233677    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xnd4l\""
Sep 16 14:14:41.233987 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.233968    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 14:14:41.310209 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.310142    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b095cf19-aa32-4e5c-af3c-7bd969dd0895-image-registry-private-configuration\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.310209 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.310173    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-config-volume\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:41.310209 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.310190    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-tmp-dir\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:41.310378 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.310217    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.310378 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.310269    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d906b113-d506-4232-8e95-23f47268744c-cert\") pod \"ingress-canary-x7sq6\" (UID: \"d906b113-d506-4232-8e95-23f47268744c\") " pod="openshift-ingress-canary/ingress-canary-x7sq6"
Sep 16 14:14:41.310378 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.310286    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm5w8\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-kube-api-access-rm5w8\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.310378 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.310341    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-certificates\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.310492 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.310380    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b095cf19-aa32-4e5c-af3c-7bd969dd0895-installation-pull-secrets\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.310492 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.310412    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b095cf19-aa32-4e5c-af3c-7bd969dd0895-ca-trust-extracted\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.310492 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.310428    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b095cf19-aa32-4e5c-af3c-7bd969dd0895-trusted-ca\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.310492 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.310445    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-bound-sa-token\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.310492 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.310478    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8lsg\" (UniqueName: \"kubernetes.io/projected/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-kube-api-access-m8lsg\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:41.310622 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.310506    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbpwq\" (UniqueName: \"kubernetes.io/projected/d906b113-d506-4232-8e95-23f47268744c-kube-api-access-rbpwq\") pod \"ingress-canary-x7sq6\" (UID: \"d906b113-d506-4232-8e95-23f47268744c\") " pod="openshift-ingress-canary/ingress-canary-x7sq6"
Sep 16 14:14:41.310622 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.310529    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:41.343414 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.343379    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kswqv_345b6071-6885-4fa9-8a8f-cc1b40368688/konnectivity-agent/0.log" limit=1024
Sep 16 14:14:41.411357 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.411328    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b095cf19-aa32-4e5c-af3c-7bd969dd0895-ca-trust-extracted\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.411488 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.411362    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b095cf19-aa32-4e5c-af3c-7bd969dd0895-trusted-ca\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.411488 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.411383    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-bound-sa-token\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.411598 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.411500    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8lsg\" (UniqueName: \"kubernetes.io/projected/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-kube-api-access-m8lsg\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:41.411598 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.411541    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbpwq\" (UniqueName: \"kubernetes.io/projected/d906b113-d506-4232-8e95-23f47268744c-kube-api-access-rbpwq\") pod \"ingress-canary-x7sq6\" (UID: \"d906b113-d506-4232-8e95-23f47268744c\") " pod="openshift-ingress-canary/ingress-canary-x7sq6"
Sep 16 14:14:41.411598 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.411574    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:41.411758 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.411651    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b095cf19-aa32-4e5c-af3c-7bd969dd0895-image-registry-private-configuration\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.411758 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.411679    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-config-volume\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:41.411758 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.411701    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-tmp-dir\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:41.411758 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.411729    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.411758 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:41.411741    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 14:14:41.411758 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.411749    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b095cf19-aa32-4e5c-af3c-7bd969dd0895-ca-trust-extracted\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.412059 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.411784    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d906b113-d506-4232-8e95-23f47268744c-cert\") pod \"ingress-canary-x7sq6\" (UID: \"d906b113-d506-4232-8e95-23f47268744c\") " pod="openshift-ingress-canary/ingress-canary-x7sq6"
Sep 16 14:14:41.412059 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:41.411806    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls podName:68b65eb9-84ac-48c6-8ffc-49c9eee408b3 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:41.911785823 +0000 UTC m=+32.458527750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls") pod "dns-default-9x4k6" (UID: "68b65eb9-84ac-48c6-8ffc-49c9eee408b3") : secret "dns-default-metrics-tls" not found
Sep 16 14:14:41.412059 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.411838    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm5w8\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-kube-api-access-rm5w8\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.412059 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:41.411941    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 14:14:41.412059 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:41.411957    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-598474997c-5r45z: secret "image-registry-tls" not found
Sep 16 14:14:41.412059 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:41.412016    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls podName:b095cf19-aa32-4e5c-af3c-7bd969dd0895 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:41.911998837 +0000 UTC m=+32.458740749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls") pod "image-registry-598474997c-5r45z" (UID: "b095cf19-aa32-4e5c-af3c-7bd969dd0895") : secret "image-registry-tls" not found
Sep 16 14:14:41.412059 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.412041    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-certificates\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.412522 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.412080    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b095cf19-aa32-4e5c-af3c-7bd969dd0895-installation-pull-secrets\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.412522 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.412094    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-tmp-dir\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:41.412522 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.412316    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-config-volume\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:41.412522 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.412380    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b095cf19-aa32-4e5c-af3c-7bd969dd0895-trusted-ca\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.412640 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.412559    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-certificates\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.415907 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.415877    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b095cf19-aa32-4e5c-af3c-7bd969dd0895-image-registry-private-configuration\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.415982 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.415912    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b095cf19-aa32-4e5c-af3c-7bd969dd0895-installation-pull-secrets\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.416026 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.416006    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d906b113-d506-4232-8e95-23f47268744c-cert\") pod \"ingress-canary-x7sq6\" (UID: \"d906b113-d506-4232-8e95-23f47268744c\") " pod="openshift-ingress-canary/ingress-canary-x7sq6"
Sep 16 14:14:41.423358 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.423326    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8lsg\" (UniqueName: \"kubernetes.io/projected/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-kube-api-access-m8lsg\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:41.423692 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.423666    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-bound-sa-token\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.424081 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.424065    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm5w8\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-kube-api-access-rm5w8\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.424231 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.424215    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbpwq\" (UniqueName: \"kubernetes.io/projected/d906b113-d506-4232-8e95-23f47268744c-kube-api-access-rbpwq\") pod \"ingress-canary-x7sq6\" (UID: \"d906b113-d506-4232-8e95-23f47268744c\") " pod="openshift-ingress-canary/ingress-canary-x7sq6"
Sep 16 14:14:41.493878 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.493845    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kswqv_345b6071-6885-4fa9-8a8f-cc1b40368688/konnectivity-agent/0.log" limit=1024
Sep 16 14:14:41.536060 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.536020    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x7sq6"
Sep 16 14:14:41.563213 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.563143    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kswqv_345b6071-6885-4fa9-8a8f-cc1b40368688/konnectivity-agent/0.log" limit=1024
Sep 16 14:14:41.780308 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.780131    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kswqv_345b6071-6885-4fa9-8a8f-cc1b40368688/konnectivity-agent/0.log" limit=1024
Sep 16 14:14:41.862145 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.862071    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x7sq6"]
Sep 16 14:14:41.866943 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:41.866918    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd906b113_d506_4232_8e95_23f47268744c.slice/crio-10f2d26302e4bb7f2e3eed6dede9754ca035f9c4b81e128528ab11f809a36547 WatchSource:0}: Error finding container 10f2d26302e4bb7f2e3eed6dede9754ca035f9c4b81e128528ab11f809a36547: Status 404 returned error can't find the container with id 10f2d26302e4bb7f2e3eed6dede9754ca035f9c4b81e128528ab11f809a36547
Sep 16 14:14:41.868380 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.868292    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kswqv_345b6071-6885-4fa9-8a8f-cc1b40368688/konnectivity-agent/0.log" limit=1024
Sep 16 14:14:41.911442 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.911419    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:41.911442 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.911417    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:41.911584 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.911425    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:41.914629 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.914611    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 14:14:41.916153 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.916134    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 14:14:41.916246 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.916137    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 14:14:41.916246 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.916171    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-999gw\""
Sep 16 14:14:41.916246 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.916166    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 14:14:41.916391 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.916293    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8dcj9\""
Sep 16 14:14:41.916532 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.916516    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:41.916604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:41.916566    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:41.916686 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:41.916668    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 14:14:41.916775 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:41.916726    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls podName:68b65eb9-84ac-48c6-8ffc-49c9eee408b3 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:42.916706978 +0000 UTC m=+33.463448887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls") pod "dns-default-9x4k6" (UID: "68b65eb9-84ac-48c6-8ffc-49c9eee408b3") : secret "dns-default-metrics-tls" not found
Sep 16 14:14:41.916775 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:41.916673    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 14:14:41.916775 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:41.916768    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-598474997c-5r45z: secret "image-registry-tls" not found
Sep 16 14:14:41.916917 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:41.916810    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls podName:b095cf19-aa32-4e5c-af3c-7bd969dd0895 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:42.916796483 +0000 UTC m=+33.463538396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls") pod "image-registry-598474997c-5r45z" (UID: "b095cf19-aa32-4e5c-af3c-7bd969dd0895") : secret "image-registry-tls" not found
Sep 16 14:14:42.060577 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:42.060544    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x7sq6" event={"ID":"d906b113-d506-4232-8e95-23f47268744c","Type":"ContainerStarted","Data":"10f2d26302e4bb7f2e3eed6dede9754ca035f9c4b81e128528ab11f809a36547"}
Sep 16 14:14:42.265645 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:42.265607    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kswqv_345b6071-6885-4fa9-8a8f-cc1b40368688/konnectivity-agent/0.log" limit=1024
Sep 16 14:14:42.923489 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:42.923455    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:42.923676 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:42.923541    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:42.923676 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:42.923624    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 14:14:42.923676 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:42.923650    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-598474997c-5r45z: secret "image-registry-tls" not found
Sep 16 14:14:42.923676 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:42.923667    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 14:14:42.923853 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:42.923718    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls podName:b095cf19-aa32-4e5c-af3c-7bd969dd0895 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:44.923699206 +0000 UTC m=+35.470441120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls") pod "image-registry-598474997c-5r45z" (UID: "b095cf19-aa32-4e5c-af3c-7bd969dd0895") : secret "image-registry-tls" not found
Sep 16 14:14:42.923853 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:42.923735    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls podName:68b65eb9-84ac-48c6-8ffc-49c9eee408b3 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:44.9237269 +0000 UTC m=+35.470468815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls") pod "dns-default-9x4k6" (UID: "68b65eb9-84ac-48c6-8ffc-49c9eee408b3") : secret "dns-default-metrics-tls" not found
Sep 16 14:14:43.046375 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:43.046341    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kswqv_345b6071-6885-4fa9-8a8f-cc1b40368688/konnectivity-agent/0.log" limit=1024
Sep 16 14:14:43.121231 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:43.121191    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kswqv_345b6071-6885-4fa9-8a8f-cc1b40368688/konnectivity-agent/0.log" limit=1024
Sep 16 14:14:43.206204 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:43.206025    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kswqv_345b6071-6885-4fa9-8a8f-cc1b40368688/konnectivity-agent/0.log" limit=1024
Sep 16 14:14:43.278278 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:43.278246    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kswqv_345b6071-6885-4fa9-8a8f-cc1b40368688/konnectivity-agent/0.log" limit=1024
Sep 16 14:14:43.631099 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:43.631059    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs\") pod \"network-metrics-daemon-5shg9\" (UID: \"4e432d05-f97c-4799-8a7c-d660680b6298\") " pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:14:43.631289 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:43.631121    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret\") pod \"global-pull-secret-syncer-kdnvs\" (UID: \"c23242c8-2ee7-4041-b310-6538da68c470\") " pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:43.631289 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:43.631244    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 14:14:43.631404 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:43.631327    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs podName:4e432d05-f97c-4799-8a7c-d660680b6298 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:15.631305786 +0000 UTC m=+66.178047698 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs") pod "network-metrics-daemon-5shg9" (UID: "4e432d05-f97c-4799-8a7c-d660680b6298") : secret "metrics-daemon-secret" not found
Sep 16 14:14:43.635083 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:43.635058    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c23242c8-2ee7-4041-b310-6538da68c470-original-pull-secret\") pod \"global-pull-secret-syncer-kdnvs\" (UID: \"c23242c8-2ee7-4041-b310-6538da68c470\") " pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:43.722125 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:43.722093    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kdnvs"
Sep 16 14:14:43.731640 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:43.731612    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqhb\" (UniqueName: \"kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb\") pod \"network-check-target-ttfnh\" (UID: \"163028d4-d9b5-4feb-9c49-3af6793693c4\") " pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:43.734153 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:43.734131    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxqhb\" (UniqueName: \"kubernetes.io/projected/163028d4-d9b5-4feb-9c49-3af6793693c4-kube-api-access-gxqhb\") pod \"network-check-target-ttfnh\" (UID: \"163028d4-d9b5-4feb-9c49-3af6793693c4\") " pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:44.024685 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:44.024652    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kdnvs"]
Sep 16 14:14:44.027542 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:44.027518    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:44.604444 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:44.604412    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23242c8_2ee7_4041_b310_6538da68c470.slice/crio-e772301d3510c4cf442250cffd978b774161e784d177a3ebf9a03f8bcfe7f548 WatchSource:0}: Error finding container e772301d3510c4cf442250cffd978b774161e784d177a3ebf9a03f8bcfe7f548: Status 404 returned error can't find the container with id e772301d3510c4cf442250cffd978b774161e784d177a3ebf9a03f8bcfe7f548
Sep 16 14:14:44.941746 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:44.941646    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:44.941933 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:44.941746    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:44.941933 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:44.941823    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 14:14:44.941933 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:44.941868    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 14:14:44.941933 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:44.941881    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-598474997c-5r45z: secret "image-registry-tls" not found
Sep 16 14:14:44.941933 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:44.941919    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls podName:68b65eb9-84ac-48c6-8ffc-49c9eee408b3 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:48.941886516 +0000 UTC m=+39.488628430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls") pod "dns-default-9x4k6" (UID: "68b65eb9-84ac-48c6-8ffc-49c9eee408b3") : secret "dns-default-metrics-tls" not found
Sep 16 14:14:44.942224 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:44.941943    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls podName:b095cf19-aa32-4e5c-af3c-7bd969dd0895 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:48.941928212 +0000 UTC m=+39.488670141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls") pod "image-registry-598474997c-5r45z" (UID: "b095cf19-aa32-4e5c-af3c-7bd969dd0895") : secret "image-registry-tls" not found
Sep 16 14:14:45.069122 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:45.069087    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kdnvs" event={"ID":"c23242c8-2ee7-4041-b310-6538da68c470","Type":"ContainerStarted","Data":"e772301d3510c4cf442250cffd978b774161e784d177a3ebf9a03f8bcfe7f548"}
Sep 16 14:14:46.638730 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:46.638494    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ttfnh"]
Sep 16 14:14:46.674849 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:14:46.674814    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163028d4_d9b5_4feb_9c49_3af6793693c4.slice/crio-3e4ec3ce3ebf88b280a637194d889123da8c411360846aa96f07590ec84ab37f WatchSource:0}: Error finding container 3e4ec3ce3ebf88b280a637194d889123da8c411360846aa96f07590ec84ab37f: Status 404 returned error can't find the container with id 3e4ec3ce3ebf88b280a637194d889123da8c411360846aa96f07590ec84ab37f
Sep 16 14:14:47.077994 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:47.077957    2553 generic.go:356] "Generic (PLEG): container finished" podID="29b51897-d00a-432c-92eb-29f6a9059b20" containerID="84fac1f2bf9401240390b51c1d194056a5abde0a230f0e85cd53691aac552e18" exitCode=0
Sep 16 14:14:47.078187 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:47.078049    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jccdd" event={"ID":"29b51897-d00a-432c-92eb-29f6a9059b20","Type":"ContainerDied","Data":"84fac1f2bf9401240390b51c1d194056a5abde0a230f0e85cd53691aac552e18"}
Sep 16 14:14:47.079908 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:47.079869    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x7sq6" event={"ID":"d906b113-d506-4232-8e95-23f47268744c","Type":"ContainerStarted","Data":"f2e86ebcc31faf361fc21ce2810b63f492deb6f85e9d5d770b41ed6f6cd93a9a"}
Sep 16 14:14:47.081313 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:47.081287    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ttfnh" event={"ID":"163028d4-d9b5-4feb-9c49-3af6793693c4","Type":"ContainerStarted","Data":"3e4ec3ce3ebf88b280a637194d889123da8c411360846aa96f07590ec84ab37f"}
Sep 16 14:14:47.148148 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:47.148098    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x7sq6" podStartSLOduration=1.6452295239999999 podStartE2EDuration="6.148076921s" totalImagesPullingTime="4.502847397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:47.146328135 +0000 UTC m=+37.693070067" watchObservedRunningTime="2026-09-16 14:14:47.148076921 +0000 UTC m=+37.694818853"
Sep 16 14:14:47.836753 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:47.836711    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kswqv_345b6071-6885-4fa9-8a8f-cc1b40368688/konnectivity-agent/0.log" limit=1024
Sep 16 14:14:48.088303 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:48.088222    2553 generic.go:356] "Generic (PLEG): container finished" podID="29b51897-d00a-432c-92eb-29f6a9059b20" containerID="ef0a30110f107785b0d0ce1a0a2dc3a9cbb987dc507612f78b35973fea547a22" exitCode=0
Sep 16 14:14:48.088455 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:48.088323    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jccdd" event={"ID":"29b51897-d00a-432c-92eb-29f6a9059b20","Type":"ContainerDied","Data":"ef0a30110f107785b0d0ce1a0a2dc3a9cbb987dc507612f78b35973fea547a22"}
Sep 16 14:14:48.977914 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:48.977866    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:48.978330 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:48.977953    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:48.978330 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:48.978005    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 14:14:48.978330 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:48.978028    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-598474997c-5r45z: secret "image-registry-tls" not found
Sep 16 14:14:48.978330 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:48.978097    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls podName:b095cf19-aa32-4e5c-af3c-7bd969dd0895 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:56.978077294 +0000 UTC m=+47.524819206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls") pod "image-registry-598474997c-5r45z" (UID: "b095cf19-aa32-4e5c-af3c-7bd969dd0895") : secret "image-registry-tls" not found
Sep 16 14:14:48.978330 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:48.978105    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 14:14:48.978330 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:48.978158    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls podName:68b65eb9-84ac-48c6-8ffc-49c9eee408b3 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:56.978142302 +0000 UTC m=+47.524884228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls") pod "dns-default-9x4k6" (UID: "68b65eb9-84ac-48c6-8ffc-49c9eee408b3") : secret "dns-default-metrics-tls" not found
Sep 16 14:14:49.096347 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:49.096279    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jccdd" event={"ID":"29b51897-d00a-432c-92eb-29f6a9059b20","Type":"ContainerStarted","Data":"e7f8d360a69bcf91c08512fd32399b73f334956798c776e95175a016519bdb1b"}
Sep 16 14:14:49.123415 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:49.123353    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jccdd" podStartSLOduration=8.044533341 podStartE2EDuration="39.12333197s" totalImagesPullingTime="31.078798629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:49.122002076 +0000 UTC m=+39.668744007" watchObservedRunningTime="2026-09-16 14:14:49.12333197 +0000 UTC m=+39.670073921"
Sep 16 14:14:49.251667 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:49.251583    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kswqv_345b6071-6885-4fa9-8a8f-cc1b40368688/konnectivity-agent/0.log" limit=1024
Sep 16 14:14:49.326470 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:49.326435    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kswqv_345b6071-6885-4fa9-8a8f-cc1b40368688/konnectivity-agent/0.log" limit=1024
Sep 16 14:14:49.400366 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:49.400326    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kswqv_345b6071-6885-4fa9-8a8f-cc1b40368688/konnectivity-agent/0.log" limit=1024
Sep 16 14:14:53.108505 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:53.108458    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kdnvs" event={"ID":"c23242c8-2ee7-4041-b310-6538da68c470","Type":"ContainerStarted","Data":"40ab797df846cb9374ce0eba9c08d13964b8cd300f949c93ae1bd3d8943ed9d5"}
Sep 16 14:14:53.110022 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:53.109992    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ttfnh" event={"ID":"163028d4-d9b5-4feb-9c49-3af6793693c4","Type":"ContainerStarted","Data":"37eec4802c53ffccdb41d4f87f031d8856d4b324e0198b05c9b66650a802f03c"}
Sep 16 14:14:53.110158 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:53.110137    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:14:53.127882 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:53.127840    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kdnvs" podStartSLOduration=35.688747699 podStartE2EDuration="43.127827442s" totalImagesPullingTime="7.439079743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:53.127038519 +0000 UTC m=+43.673780450" watchObservedRunningTime="2026-09-16 14:14:53.127827442 +0000 UTC m=+43.674569373"
Sep 16 14:14:53.150188 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:53.150147    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ttfnh" podStartSLOduration=37.471011658 podStartE2EDuration="43.15013704s" totalImagesPullingTime="5.679125382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:53.14965031 +0000 UTC m=+43.696392240" watchObservedRunningTime="2026-09-16 14:14:53.15013704 +0000 UTC m=+43.696878972"
Sep 16 14:14:56.208265 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:56.208221    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kswqv_345b6071-6885-4fa9-8a8f-cc1b40368688/konnectivity-agent/0.log" limit=1024
Sep 16 14:14:57.039030 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:57.038994    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:14:57.039219 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:14:57.039042    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:14:57.039219 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:57.039145    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 14:14:57.039219 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:57.039147    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 14:14:57.039377 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:57.039220    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls podName:68b65eb9-84ac-48c6-8ffc-49c9eee408b3 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:13.03920136 +0000 UTC m=+63.585943283 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls") pod "dns-default-9x4k6" (UID: "68b65eb9-84ac-48c6-8ffc-49c9eee408b3") : secret "dns-default-metrics-tls" not found
Sep 16 14:14:57.039377 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:57.039157    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-598474997c-5r45z: secret "image-registry-tls" not found
Sep 16 14:14:57.039377 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:14:57.039284    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls podName:b095cf19-aa32-4e5c-af3c-7bd969dd0895 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:13.039272868 +0000 UTC m=+63.586014776 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls") pod "image-registry-598474997c-5r45z" (UID: "b095cf19-aa32-4e5c-af3c-7bd969dd0895") : secret "image-registry-tls" not found
Sep 16 14:15:07.886650 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.886619    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-st2qc"]
Sep 16 14:15:07.888957 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.888935    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-st2qc"
Sep 16 14:15:07.892282 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.892264    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-l5lvq\""
Sep 16 14:15:07.898756 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.898737    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh"]
Sep 16 14:15:07.900140 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.900126    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh"
Sep 16 14:15:07.902866 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.902845    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 14:15:07.903082 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.903064    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 14:15:07.903664 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.903619    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-m44pb\""
Sep 16 14:15:07.903755 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.903691    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-st2qc"]
Sep 16 14:15:07.903847 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.903826    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 14:15:07.903933 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.903870    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 14:15:07.904694 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.904673    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"]
Sep 16 14:15:07.906329 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.906313    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-s4pkv"]
Sep 16 14:15:07.906448 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.906432    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:07.907379 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.907359    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/556cb41f-d181-44fd-84e4-2a86e3b35965-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbfkh\" (UID: \"556cb41f-d181-44fd-84e4-2a86e3b35965\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh"
Sep 16 14:15:07.907473 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.907409    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/556cb41f-d181-44fd-84e4-2a86e3b35965-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbfkh\" (UID: \"556cb41f-d181-44fd-84e4-2a86e3b35965\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh"
Sep 16 14:15:07.907958 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.907477    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5l2fd\" (UID: \"136f6c0b-4717-4e2b-b788-fa72b68f6eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:07.908049 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.908012    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlh8c\" (UniqueName: \"kubernetes.io/projected/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-kube-api-access-hlh8c\") pod \"cluster-monitoring-operator-6cd954b897-5l2fd\" (UID: \"136f6c0b-4717-4e2b-b788-fa72b68f6eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:07.908108 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.908077    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-5l2fd\" (UID: \"136f6c0b-4717-4e2b-b788-fa72b68f6eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:07.908159 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.908116    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98f62\" (UniqueName: \"kubernetes.io/projected/556cb41f-d181-44fd-84e4-2a86e3b35965-kube-api-access-98f62\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbfkh\" (UID: \"556cb41f-d181-44fd-84e4-2a86e3b35965\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh"
Sep 16 14:15:07.909799 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.908793    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:07.909799 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.909717    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 14:15:07.912341 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.912319    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 14:15:07.913585 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.913568    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 14:15:07.913674 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.913600    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 14:15:07.913674 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.913616    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m7tsr\""
Sep 16 14:15:07.916379 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.916362    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 14:15:07.916857 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.916837    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 14:15:07.916949 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.916843    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xsts4\""
Sep 16 14:15:07.917226 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.917210    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 14:15:07.917428 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.917414    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 14:15:07.924426 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.924408    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 14:15:07.924554 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.924537    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh"]
Sep 16 14:15:07.926226 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.926204    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"]
Sep 16 14:15:07.926989 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.926967    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-s4pkv"]
Sep 16 14:15:07.996355 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.996329    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-25dg9"]
Sep 16 14:15:07.998023 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:07.998008    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-25dg9"
Sep 16 14:15:08.000583 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.000563    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9s46l\""
Sep 16 14:15:08.001035 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.001018    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 14:15:08.001163 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.001148    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 14:15:08.009258 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.009240    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-5l2fd\" (UID: \"136f6c0b-4717-4e2b-b788-fa72b68f6eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:08.009351 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.009270    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98f62\" (UniqueName: \"kubernetes.io/projected/556cb41f-d181-44fd-84e4-2a86e3b35965-kube-api-access-98f62\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbfkh\" (UID: \"556cb41f-d181-44fd-84e4-2a86e3b35965\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh"
Sep 16 14:15:08.009351 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.009300    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe-trusted-ca\") pod \"console-operator-696884fb9c-s4pkv\" (UID: \"ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe\") " pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:08.009448 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.009431    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe-serving-cert\") pod \"console-operator-696884fb9c-s4pkv\" (UID: \"ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe\") " pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:08.009501 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.009467    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btvqj\" (UniqueName: \"kubernetes.io/projected/57ee7262-da79-4676-b886-a297c0899b45-kube-api-access-btvqj\") pod \"network-check-source-6f8b768455-st2qc\" (UID: \"57ee7262-da79-4676-b886-a297c0899b45\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-st2qc"
Sep 16 14:15:08.009560 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.009538    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/556cb41f-d181-44fd-84e4-2a86e3b35965-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbfkh\" (UID: \"556cb41f-d181-44fd-84e4-2a86e3b35965\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh"
Sep 16 14:15:08.009594 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.009568    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe-config\") pod \"console-operator-696884fb9c-s4pkv\" (UID: \"ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe\") " pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:08.009638 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.009598    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/556cb41f-d181-44fd-84e4-2a86e3b35965-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbfkh\" (UID: \"556cb41f-d181-44fd-84e4-2a86e3b35965\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh"
Sep 16 14:15:08.009638 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.009617    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2jrk\" (UniqueName: \"kubernetes.io/projected/ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe-kube-api-access-c2jrk\") pod \"console-operator-696884fb9c-s4pkv\" (UID: \"ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe\") " pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:08.009739 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.009714    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sp9p\" (UniqueName: \"kubernetes.io/projected/667e0c83-a29e-44a0-8cf7-936e046f6f05-kube-api-access-9sp9p\") pod \"volume-data-source-validator-86695b7cc7-25dg9\" (UID: \"667e0c83-a29e-44a0-8cf7-936e046f6f05\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-25dg9"
Sep 16 14:15:08.009790 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.009741    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5l2fd\" (UID: \"136f6c0b-4717-4e2b-b788-fa72b68f6eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:08.009790 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.009761    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlh8c\" (UniqueName: \"kubernetes.io/projected/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-kube-api-access-hlh8c\") pod \"cluster-monitoring-operator-6cd954b897-5l2fd\" (UID: \"136f6c0b-4717-4e2b-b788-fa72b68f6eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:08.009910 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:08.009814    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 14:15:08.009910 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:08.009903    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls podName:136f6c0b-4717-4e2b-b788-fa72b68f6eb8 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:08.509872219 +0000 UTC m=+59.056614129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5l2fd" (UID: "136f6c0b-4717-4e2b-b788-fa72b68f6eb8") : secret "cluster-monitoring-operator-tls" not found
Sep 16 14:15:08.010025 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.009999    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-5l2fd\" (UID: \"136f6c0b-4717-4e2b-b788-fa72b68f6eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:08.010185 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.010165    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/556cb41f-d181-44fd-84e4-2a86e3b35965-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbfkh\" (UID: \"556cb41f-d181-44fd-84e4-2a86e3b35965\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh"
Sep 16 14:15:08.012866 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.012846    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/556cb41f-d181-44fd-84e4-2a86e3b35965-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbfkh\" (UID: \"556cb41f-d181-44fd-84e4-2a86e3b35965\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh"
Sep 16 14:15:08.014814 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.014793    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-25dg9"]
Sep 16 14:15:08.024978 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.024959    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98f62\" (UniqueName: \"kubernetes.io/projected/556cb41f-d181-44fd-84e4-2a86e3b35965-kube-api-access-98f62\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xbfkh\" (UID: \"556cb41f-d181-44fd-84e4-2a86e3b35965\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh"
Sep 16 14:15:08.028258 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.028230    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlh8c\" (UniqueName: \"kubernetes.io/projected/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-kube-api-access-hlh8c\") pod \"cluster-monitoring-operator-6cd954b897-5l2fd\" (UID: \"136f6c0b-4717-4e2b-b788-fa72b68f6eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:08.055858 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.055837    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnrqf"
Sep 16 14:15:08.110843 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.110814    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe-config\") pod \"console-operator-696884fb9c-s4pkv\" (UID: \"ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe\") " pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:08.110974 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.110854    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2jrk\" (UniqueName: \"kubernetes.io/projected/ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe-kube-api-access-c2jrk\") pod \"console-operator-696884fb9c-s4pkv\" (UID: \"ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe\") " pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:08.111032 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.110999    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sp9p\" (UniqueName: \"kubernetes.io/projected/667e0c83-a29e-44a0-8cf7-936e046f6f05-kube-api-access-9sp9p\") pod \"volume-data-source-validator-86695b7cc7-25dg9\" (UID: \"667e0c83-a29e-44a0-8cf7-936e046f6f05\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-25dg9"
Sep 16 14:15:08.111139 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.111121    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe-trusted-ca\") pod \"console-operator-696884fb9c-s4pkv\" (UID: \"ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe\") " pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:08.111202 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.111169    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe-serving-cert\") pod \"console-operator-696884fb9c-s4pkv\" (UID: \"ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe\") " pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:08.111252 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.111215    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btvqj\" (UniqueName: \"kubernetes.io/projected/57ee7262-da79-4676-b886-a297c0899b45-kube-api-access-btvqj\") pod \"network-check-source-6f8b768455-st2qc\" (UID: \"57ee7262-da79-4676-b886-a297c0899b45\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-st2qc"
Sep 16 14:15:08.111565 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.111541    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe-config\") pod \"console-operator-696884fb9c-s4pkv\" (UID: \"ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe\") " pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:08.111807 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.111788    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe-trusted-ca\") pod \"console-operator-696884fb9c-s4pkv\" (UID: \"ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe\") " pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:08.113405 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.113385    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe-serving-cert\") pod \"console-operator-696884fb9c-s4pkv\" (UID: \"ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe\") " pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:08.119636 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.119606    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btvqj\" (UniqueName: \"kubernetes.io/projected/57ee7262-da79-4676-b886-a297c0899b45-kube-api-access-btvqj\") pod \"network-check-source-6f8b768455-st2qc\" (UID: \"57ee7262-da79-4676-b886-a297c0899b45\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-st2qc"
Sep 16 14:15:08.119737 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.119645    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sp9p\" (UniqueName: \"kubernetes.io/projected/667e0c83-a29e-44a0-8cf7-936e046f6f05-kube-api-access-9sp9p\") pod \"volume-data-source-validator-86695b7cc7-25dg9\" (UID: \"667e0c83-a29e-44a0-8cf7-936e046f6f05\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-25dg9"
Sep 16 14:15:08.120048 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.120030    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2jrk\" (UniqueName: \"kubernetes.io/projected/ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe-kube-api-access-c2jrk\") pod \"console-operator-696884fb9c-s4pkv\" (UID: \"ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe\") " pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:08.198185 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.198129    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-st2qc"
Sep 16 14:15:08.212709 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.212687    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh"
Sep 16 14:15:08.224591 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.224566    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:08.306370 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.306346    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-25dg9"
Sep 16 14:15:08.469126 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.469059    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-st2qc"]
Sep 16 14:15:08.471998 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:15:08.471969    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ee7262_da79_4676_b886_a297c0899b45.slice/crio-1d2ac327f4785ff2c0d7e490cdb0d3574cc9faa308fde4f0a5ae974432c12c30 WatchSource:0}: Error finding container 1d2ac327f4785ff2c0d7e490cdb0d3574cc9faa308fde4f0a5ae974432c12c30: Status 404 returned error can't find the container with id 1d2ac327f4785ff2c0d7e490cdb0d3574cc9faa308fde4f0a5ae974432c12c30
Sep 16 14:15:08.494147 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.494087    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh"]
Sep 16 14:15:08.504936 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:15:08.504876    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556cb41f_d181_44fd_84e4_2a86e3b35965.slice/crio-719c8bbdcb59db7a9b8d6d150d344488fcbd22059924a7739d612200e27a5d59 WatchSource:0}: Error finding container 719c8bbdcb59db7a9b8d6d150d344488fcbd22059924a7739d612200e27a5d59: Status 404 returned error can't find the container with id 719c8bbdcb59db7a9b8d6d150d344488fcbd22059924a7739d612200e27a5d59
Sep 16 14:15:08.513633 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.513609    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5l2fd\" (UID: \"136f6c0b-4717-4e2b-b788-fa72b68f6eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:08.513752 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:08.513717    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 14:15:08.513801 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:08.513770    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls podName:136f6c0b-4717-4e2b-b788-fa72b68f6eb8 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:09.513757429 +0000 UTC m=+60.060499341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5l2fd" (UID: "136f6c0b-4717-4e2b-b788-fa72b68f6eb8") : secret "cluster-monitoring-operator-tls" not found
Sep 16 14:15:08.520994 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.520973    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-s4pkv"]
Sep 16 14:15:08.524064 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:15:08.524042    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0aa50c_e70c_4d0a_86e2_ee90fa008ebe.slice/crio-278874afa919ce9fecc06dcb9182265ce84ad94629062ddb41cd0f3d20d3c9ad WatchSource:0}: Error finding container 278874afa919ce9fecc06dcb9182265ce84ad94629062ddb41cd0f3d20d3c9ad: Status 404 returned error can't find the container with id 278874afa919ce9fecc06dcb9182265ce84ad94629062ddb41cd0f3d20d3c9ad
Sep 16 14:15:08.584089 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:15:08.584057    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667e0c83_a29e_44a0_8cf7_936e046f6f05.slice/crio-4c05617b690ada3e66abc344697ce942429e3b5495f58773cfd5e6d259bada89 WatchSource:0}: Error finding container 4c05617b690ada3e66abc344697ce942429e3b5495f58773cfd5e6d259bada89: Status 404 returned error can't find the container with id 4c05617b690ada3e66abc344697ce942429e3b5495f58773cfd5e6d259bada89
Sep 16 14:15:08.584413 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:08.584392    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-25dg9"]
Sep 16 14:15:09.152354 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:09.152312    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-st2qc" event={"ID":"57ee7262-da79-4676-b886-a297c0899b45","Type":"ContainerStarted","Data":"ef2f70f65731b046e5952b6ff2e328a5bb77c5fa594686386fb77ab4262227fb"}
Sep 16 14:15:09.152781 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:09.152367    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-st2qc" event={"ID":"57ee7262-da79-4676-b886-a297c0899b45","Type":"ContainerStarted","Data":"1d2ac327f4785ff2c0d7e490cdb0d3574cc9faa308fde4f0a5ae974432c12c30"}
Sep 16 14:15:09.153460 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:09.153433    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-25dg9" event={"ID":"667e0c83-a29e-44a0-8cf7-936e046f6f05","Type":"ContainerStarted","Data":"4c05617b690ada3e66abc344697ce942429e3b5495f58773cfd5e6d259bada89"}
Sep 16 14:15:09.154472 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:09.154451    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-s4pkv" event={"ID":"ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe","Type":"ContainerStarted","Data":"278874afa919ce9fecc06dcb9182265ce84ad94629062ddb41cd0f3d20d3c9ad"}
Sep 16 14:15:09.155445 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:09.155424    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh" event={"ID":"556cb41f-d181-44fd-84e4-2a86e3b35965","Type":"ContainerStarted","Data":"719c8bbdcb59db7a9b8d6d150d344488fcbd22059924a7739d612200e27a5d59"}
Sep 16 14:15:09.521103 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:09.521065    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5l2fd\" (UID: \"136f6c0b-4717-4e2b-b788-fa72b68f6eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:09.521253 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:09.521175    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 14:15:09.521253 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:09.521230    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls podName:136f6c0b-4717-4e2b-b788-fa72b68f6eb8 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:11.521216456 +0000 UTC m=+62.067958370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5l2fd" (UID: "136f6c0b-4717-4e2b-b788-fa72b68f6eb8") : secret "cluster-monitoring-operator-tls" not found
Sep 16 14:15:11.538142 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:11.538103    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5l2fd\" (UID: \"136f6c0b-4717-4e2b-b788-fa72b68f6eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:11.538561 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:11.538252    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 14:15:11.538561 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:11.538324    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls podName:136f6c0b-4717-4e2b-b788-fa72b68f6eb8 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:15.538307588 +0000 UTC m=+66.085049498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5l2fd" (UID: "136f6c0b-4717-4e2b-b788-fa72b68f6eb8") : secret "cluster-monitoring-operator-tls" not found
Sep 16 14:15:12.168387 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.168352    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-25dg9" event={"ID":"667e0c83-a29e-44a0-8cf7-936e046f6f05","Type":"ContainerStarted","Data":"28c698bf893c9608ba0dcfb8e46abfa5f1c5bfd7abb4bb509ca89c69ae1d10ec"}
Sep 16 14:15:12.169905 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.169872    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/0.log"
Sep 16 14:15:12.170010 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.169921    2553 generic.go:356] "Generic (PLEG): container finished" podID="ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe" containerID="55848c6ee48e75e66c91eefe148d428e0acce941072ed28448cddaf6bd5c1149" exitCode=255
Sep 16 14:15:12.170010 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.169974    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-s4pkv" event={"ID":"ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe","Type":"ContainerDied","Data":"55848c6ee48e75e66c91eefe148d428e0acce941072ed28448cddaf6bd5c1149"}
Sep 16 14:15:12.170135 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.170120    2553 scope.go:117] "RemoveContainer" containerID="55848c6ee48e75e66c91eefe148d428e0acce941072ed28448cddaf6bd5c1149"
Sep 16 14:15:12.184867 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.184827    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-st2qc" podStartSLOduration=5.184814604 podStartE2EDuration="5.184814604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:09.169418304 +0000 UTC m=+59.716160234" watchObservedRunningTime="2026-09-16 14:15:12.184814604 +0000 UTC m=+62.731556537"
Sep 16 14:15:12.185009 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.184934    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-25dg9" podStartSLOduration=2.5717568269999997 podStartE2EDuration="5.184923538s" totalImagesPullingTime="2.613166711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:12.184102534 +0000 UTC m=+62.730844466" watchObservedRunningTime="2026-09-16 14:15:12.184923538 +0000 UTC m=+62.731665488"
Sep 16 14:15:12.871830 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.871793    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mbq9r"]
Sep 16 14:15:12.874030 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.874004    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:12.876653 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.876576    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 14:15:12.876783 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.876658    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 14:15:12.876783 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.876690    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 14:15:12.876946 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.876932    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 14:15:12.877832 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.877815    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qst7x\""
Sep 16 14:15:12.891449 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.891394    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mbq9r"]
Sep 16 14:15:12.947414 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.947383    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbxp8\" (UniqueName: \"kubernetes.io/projected/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-api-access-cbxp8\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:12.947565 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.947425    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b2463f9-3054-4d2d-94c5-b07c406a9187-crio-socket\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:12.947565 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.947484    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:12.947676 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.947610    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:12.947729 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:12.947676    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b2463f9-3054-4d2d-94c5-b07c406a9187-data-volume\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:13.048245 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.048209    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbxp8\" (UniqueName: \"kubernetes.io/projected/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-api-access-cbxp8\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:13.048395 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.048270    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b2463f9-3054-4d2d-94c5-b07c406a9187-crio-socket\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:13.048395 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.048313    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:15:13.048395 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.048339    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:13.048395 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.048385    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:13.048527 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.048400    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b2463f9-3054-4d2d-94c5-b07c406a9187-crio-socket\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:13.048527 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:13.048431    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 14:15:13.048527 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:13.048454    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-598474997c-5r45z: secret "image-registry-tls" not found
Sep 16 14:15:13.048527 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:13.048485    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 14:15:13.048527 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:13.048492    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:13.048527 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.048408    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:15:13.048709 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:13.048533    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 14:15:13.048709 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:13.048500    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls podName:b095cf19-aa32-4e5c-af3c-7bd969dd0895 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:45.048486629 +0000 UTC m=+95.595228538 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls") pod "image-registry-598474997c-5r45z" (UID: "b095cf19-aa32-4e5c-af3c-7bd969dd0895") : secret "image-registry-tls" not found
Sep 16 14:15:13.048709 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:13.048573    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls podName:3b2463f9-3054-4d2d-94c5-b07c406a9187 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:13.548554766 +0000 UTC m=+64.095296684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mbq9r" (UID: "3b2463f9-3054-4d2d-94c5-b07c406a9187") : secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:13.048709 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:13.048592    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls podName:68b65eb9-84ac-48c6-8ffc-49c9eee408b3 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:45.048582932 +0000 UTC m=+95.595324845 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls") pod "dns-default-9x4k6" (UID: "68b65eb9-84ac-48c6-8ffc-49c9eee408b3") : secret "dns-default-metrics-tls" not found
Sep 16 14:15:13.048709 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:13.048605    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm podName:3b2463f9-3054-4d2d-94c5-b07c406a9187 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:13.548599402 +0000 UTC m=+64.095341315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mbq9r" (UID: "3b2463f9-3054-4d2d-94c5-b07c406a9187") : configmap "kube-rbac-proxy" not found
Sep 16 14:15:13.048709 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.048643    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b2463f9-3054-4d2d-94c5-b07c406a9187-data-volume\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:13.048929 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.048914    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b2463f9-3054-4d2d-94c5-b07c406a9187-data-volume\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:13.057004 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.056981    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbxp8\" (UniqueName: \"kubernetes.io/projected/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-api-access-cbxp8\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:13.176733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.176664    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:15:13.177115 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.177099    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/0.log"
Sep 16 14:15:13.177170 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.177136    2553 generic.go:356] "Generic (PLEG): container finished" podID="ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe" containerID="1886f74f7e4d112fa3def3db8d552bf24f9c4fdcc762b8c952a5aefbba7d07e4" exitCode=255
Sep 16 14:15:13.177230 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.177208    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-s4pkv" event={"ID":"ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe","Type":"ContainerDied","Data":"1886f74f7e4d112fa3def3db8d552bf24f9c4fdcc762b8c952a5aefbba7d07e4"}
Sep 16 14:15:13.178066 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.178045    2553 scope.go:117] "RemoveContainer" containerID="55848c6ee48e75e66c91eefe148d428e0acce941072ed28448cddaf6bd5c1149"
Sep 16 14:15:13.178168 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.178134    2553 scope.go:117] "RemoveContainer" containerID="1886f74f7e4d112fa3def3db8d552bf24f9c4fdcc762b8c952a5aefbba7d07e4"
Sep 16 14:15:13.178815 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:13.178786    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-s4pkv_openshift-console-operator(ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe)\"" pod="openshift-console-operator/console-operator-696884fb9c-s4pkv" podUID="ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe"
Sep 16 14:15:13.182463 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.179208    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh" event={"ID":"556cb41f-d181-44fd-84e4-2a86e3b35965","Type":"ContainerStarted","Data":"a64d1f8dbf9abea6a4aa8e71a136d867d32c08915c1931519b24bdc77acc331d"}
Sep 16 14:15:13.224319 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.224279    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh" podStartSLOduration=1.657019822 podStartE2EDuration="6.224266803s" totalImagesPullingTime="4.567246981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:13.223439652 +0000 UTC m=+63.770181583" watchObservedRunningTime="2026-09-16 14:15:13.224266803 +0000 UTC m=+63.771008734"
Sep 16 14:15:13.551838 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.551803    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:13.552006 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:13.551849    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:13.552006 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:13.551950    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 14:15:13.552081 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:13.552006    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:13.552081 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:13.552035    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm podName:3b2463f9-3054-4d2d-94c5-b07c406a9187 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:14.552016967 +0000 UTC m=+65.098758875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mbq9r" (UID: "3b2463f9-3054-4d2d-94c5-b07c406a9187") : configmap "kube-rbac-proxy" not found
Sep 16 14:15:13.552081 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:13.552063    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls podName:3b2463f9-3054-4d2d-94c5-b07c406a9187 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:14.552050728 +0000 UTC m=+65.098792637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mbq9r" (UID: "3b2463f9-3054-4d2d-94c5-b07c406a9187") : secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:14.183998 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:14.183969    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:15:14.184395 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:14.184318    2553 scope.go:117] "RemoveContainer" containerID="1886f74f7e4d112fa3def3db8d552bf24f9c4fdcc762b8c952a5aefbba7d07e4"
Sep 16 14:15:14.184487 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:14.184468    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-s4pkv_openshift-console-operator(ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe)\"" pod="openshift-console-operator/console-operator-696884fb9c-s4pkv" podUID="ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe"
Sep 16 14:15:14.437907 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:14.437819    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s79nx_846a7905-fecc-4b70-b055-ee249dde8cb6/dns-node-resolver/0.log"
Sep 16 14:15:14.560046 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:14.560006    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:14.560222 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:14.560056    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:14.560222 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:14.560145    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 14:15:14.560222 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:14.560215    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm podName:3b2463f9-3054-4d2d-94c5-b07c406a9187 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:16.56019614 +0000 UTC m=+67.106938066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mbq9r" (UID: "3b2463f9-3054-4d2d-94c5-b07c406a9187") : configmap "kube-rbac-proxy" not found
Sep 16 14:15:14.560222 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:14.560153    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:14.560393 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:14.560280    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls podName:3b2463f9-3054-4d2d-94c5-b07c406a9187 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:16.560265348 +0000 UTC m=+67.107007258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mbq9r" (UID: "3b2463f9-3054-4d2d-94c5-b07c406a9187") : secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:15.237463 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:15.237433    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-255dt_279e9549-530e-4e54-afbe-9c0954c59441/node-ca/0.log"
Sep 16 14:15:15.567870 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:15.567782    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5l2fd\" (UID: \"136f6c0b-4717-4e2b-b788-fa72b68f6eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:15.568005 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:15.567960    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 14:15:15.568058 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:15.568036    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls podName:136f6c0b-4717-4e2b-b788-fa72b68f6eb8 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:23.568018565 +0000 UTC m=+74.114760480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5l2fd" (UID: "136f6c0b-4717-4e2b-b788-fa72b68f6eb8") : secret "cluster-monitoring-operator-tls" not found
Sep 16 14:15:15.668706 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:15.668663    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs\") pod \"network-metrics-daemon-5shg9\" (UID: \"4e432d05-f97c-4799-8a7c-d660680b6298\") " pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:15:15.668852 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:15.668809    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 14:15:15.668886 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:15.668868    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs podName:4e432d05-f97c-4799-8a7c-d660680b6298 nodeName:}" failed. No retries permitted until 2026-09-16 14:16:19.668853767 +0000 UTC m=+130.215595677 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs") pod "network-metrics-daemon-5shg9" (UID: "4e432d05-f97c-4799-8a7c-d660680b6298") : secret "metrics-daemon-secret" not found
Sep 16 14:15:16.238503 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:16.238467    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x7sq6_d906b113-d506-4232-8e95-23f47268744c/serve-healthcheck-canary/0.log"
Sep 16 14:15:16.574315 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:16.574222    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:16.574315 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:16.574279    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:16.574498 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:16.574345    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 14:15:16.574498 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:16.574398    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:16.574498 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:16.574410    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm podName:3b2463f9-3054-4d2d-94c5-b07c406a9187 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:20.574393279 +0000 UTC m=+71.121135188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mbq9r" (UID: "3b2463f9-3054-4d2d-94c5-b07c406a9187") : configmap "kube-rbac-proxy" not found
Sep 16 14:15:16.574498 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:16.574436    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls podName:3b2463f9-3054-4d2d-94c5-b07c406a9187 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:20.574426076 +0000 UTC m=+71.121167985 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mbq9r" (UID: "3b2463f9-3054-4d2d-94c5-b07c406a9187") : secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:18.225161 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:18.225126    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:18.225521 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:18.225189    2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:18.225521 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:18.225504    2553 scope.go:117] "RemoveContainer" containerID="1886f74f7e4d112fa3def3db8d552bf24f9c4fdcc762b8c952a5aefbba7d07e4"
Sep 16 14:15:18.225657 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:18.225639    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-s4pkv_openshift-console-operator(ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe)\"" pod="openshift-console-operator/console-operator-696884fb9c-s4pkv" podUID="ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe"
Sep 16 14:15:20.609730 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:20.609686    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:20.610197 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:20.609746    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:20.610197 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:20.609808    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 14:15:20.610197 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:20.609869    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:20.610197 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:20.609872    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm podName:3b2463f9-3054-4d2d-94c5-b07c406a9187 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:28.609856347 +0000 UTC m=+79.156598270 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mbq9r" (UID: "3b2463f9-3054-4d2d-94c5-b07c406a9187") : configmap "kube-rbac-proxy" not found
Sep 16 14:15:20.610197 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:20.609942    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls podName:3b2463f9-3054-4d2d-94c5-b07c406a9187 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:28.609924469 +0000 UTC m=+79.156666379 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mbq9r" (UID: "3b2463f9-3054-4d2d-94c5-b07c406a9187") : secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:23.634226 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:23.634193    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5l2fd\" (UID: \"136f6c0b-4717-4e2b-b788-fa72b68f6eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:23.634574 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:23.634329    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 14:15:23.634574 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:23.634391    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls podName:136f6c0b-4717-4e2b-b788-fa72b68f6eb8 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:39.634375648 +0000 UTC m=+90.181117556 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5l2fd" (UID: "136f6c0b-4717-4e2b-b788-fa72b68f6eb8") : secret "cluster-monitoring-operator-tls" not found
Sep 16 14:15:24.115232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:24.115204    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ttfnh"
Sep 16 14:15:28.672624 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:28.672586    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:28.673113 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:28.672703    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:28.673289 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:28.673268    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b2463f9-3054-4d2d-94c5-b07c406a9187-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:28.674783 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:28.674765    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b2463f9-3054-4d2d-94c5-b07c406a9187-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbq9r\" (UID: \"3b2463f9-3054-4d2d-94c5-b07c406a9187\") " pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:28.784021 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:28.783991    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mbq9r"
Sep 16 14:15:29.059705 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:29.059672    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mbq9r"]
Sep 16 14:15:29.062449 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:15:29.062415    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2463f9_3054_4d2d_94c5_b07c406a9187.slice/crio-756d559cedb4e3d5cc8e596bc20f73c2dcfc28cfaeb776df8d8a9b449ffb3d53 WatchSource:0}: Error finding container 756d559cedb4e3d5cc8e596bc20f73c2dcfc28cfaeb776df8d8a9b449ffb3d53: Status 404 returned error can't find the container with id 756d559cedb4e3d5cc8e596bc20f73c2dcfc28cfaeb776df8d8a9b449ffb3d53
Sep 16 14:15:29.232131 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:29.232098    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mbq9r" event={"ID":"3b2463f9-3054-4d2d-94c5-b07c406a9187","Type":"ContainerStarted","Data":"7fbe34174490a707f178ad680978bc4068abd1b1a67b8072719621db0d169969"}
Sep 16 14:15:29.232269 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:29.232155    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mbq9r" event={"ID":"3b2463f9-3054-4d2d-94c5-b07c406a9187","Type":"ContainerStarted","Data":"756d559cedb4e3d5cc8e596bc20f73c2dcfc28cfaeb776df8d8a9b449ffb3d53"}
Sep 16 14:15:30.237119 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:30.237083    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mbq9r" event={"ID":"3b2463f9-3054-4d2d-94c5-b07c406a9187","Type":"ContainerStarted","Data":"5f9650f00750130268efcde0565e7dca50e8f69733da4411beacba5b1d4a8628"}
Sep 16 14:15:30.911985 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:30.911956    2553 scope.go:117] "RemoveContainer" containerID="1886f74f7e4d112fa3def3db8d552bf24f9c4fdcc762b8c952a5aefbba7d07e4"
Sep 16 14:15:31.242196 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:31.242172    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:15:31.242731 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:31.242218    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-s4pkv" event={"ID":"ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe","Type":"ContainerStarted","Data":"46182f109415a2e78bfea11b31a0126b33d936eb1382c6cc8661e115add36fb3"}
Sep 16 14:15:31.242731 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:31.242488    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:31.260290 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:31.260243    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-s4pkv" podStartSLOduration=21.58456034 podStartE2EDuration="24.260229121s" totalImagesPullingTime="2.675668781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:31.258998613 +0000 UTC m=+81.805740545" watchObservedRunningTime="2026-09-16 14:15:31.260229121 +0000 UTC m=+81.806971051"
Sep 16 14:15:32.203382 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:32.203278    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-s4pkv"
Sep 16 14:15:32.248550 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:32.248492    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mbq9r" event={"ID":"3b2463f9-3054-4d2d-94c5-b07c406a9187","Type":"ContainerStarted","Data":"b38445b4aeafb312917c7c2a739c2022ea2c733303ac9ef40374311eb945c4dc"}
Sep 16 14:15:32.268839 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:32.268789    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mbq9r" podStartSLOduration=17.810787481 podStartE2EDuration="20.268773192s" totalImagesPullingTime="2.457985711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:15:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:32.267657571 +0000 UTC m=+82.814399502" watchObservedRunningTime="2026-09-16 14:15:32.268773192 +0000 UTC m=+82.815515124"
Sep 16 14:15:35.694996 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.694961    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw"]
Sep 16 14:15:35.697116 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.697096    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw"
Sep 16 14:15:35.699839 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.699815    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 14:15:35.701215 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.701195    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 14:15:35.701328 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.701195    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 14:15:35.701328 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.701258    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 14:15:35.714815 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.714751    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw"]
Sep 16 14:15:35.792905 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.792868    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-phq9c"]
Sep 16 14:15:35.794622 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.794605    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-phq9c"
Sep 16 14:15:35.797606 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.797569    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-l5zv8\""
Sep 16 14:15:35.797739 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.797683    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 14:15:35.798011 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.797994    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 14:15:35.802199 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.802179    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-csd2f"]
Sep 16 14:15:35.803716 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.803702    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-csd2f"
Sep 16 14:15:35.806378 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.806361    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-74t6p\""
Sep 16 14:15:35.806604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.806583    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 14:15:35.806683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.806642    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 14:15:35.812303 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.812284    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-phq9c"]
Sep 16 14:15:35.821799 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.821768    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-csd2f"]
Sep 16 14:15:35.831340 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.831320    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d091897-2f8d-4422-9f82-b7882d9cc02b-tmp\") pod \"klusterlet-addon-workmgr-5b978cdb5f-8gxfw\" (UID: \"6d091897-2f8d-4422-9f82-b7882d9cc02b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw"
Sep 16 14:15:35.831418 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.831347    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6d091897-2f8d-4422-9f82-b7882d9cc02b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b978cdb5f-8gxfw\" (UID: \"6d091897-2f8d-4422-9f82-b7882d9cc02b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw"
Sep 16 14:15:35.831462 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.831423    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcfk4\" (UniqueName: \"kubernetes.io/projected/6d091897-2f8d-4422-9f82-b7882d9cc02b-kube-api-access-fcfk4\") pod \"klusterlet-addon-workmgr-5b978cdb5f-8gxfw\" (UID: \"6d091897-2f8d-4422-9f82-b7882d9cc02b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw"
Sep 16 14:15:35.932537 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.932509    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d091897-2f8d-4422-9f82-b7882d9cc02b-tmp\") pod \"klusterlet-addon-workmgr-5b978cdb5f-8gxfw\" (UID: \"6d091897-2f8d-4422-9f82-b7882d9cc02b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw"
Sep 16 14:15:35.932537 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.932539    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6d091897-2f8d-4422-9f82-b7882d9cc02b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b978cdb5f-8gxfw\" (UID: \"6d091897-2f8d-4422-9f82-b7882d9cc02b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw"
Sep 16 14:15:35.932756 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.932740    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5f306dc-d337-4d52-8d54-f115dc90fcc5-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-csd2f\" (UID: \"a5f306dc-d337-4d52-8d54-f115dc90fcc5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-csd2f"
Sep 16 14:15:35.932826 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.932769    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcfk4\" (UniqueName: \"kubernetes.io/projected/6d091897-2f8d-4422-9f82-b7882d9cc02b-kube-api-access-fcfk4\") pod \"klusterlet-addon-workmgr-5b978cdb5f-8gxfw\" (UID: \"6d091897-2f8d-4422-9f82-b7882d9cc02b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw"
Sep 16 14:15:35.932826 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.932800    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5f306dc-d337-4d52-8d54-f115dc90fcc5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-csd2f\" (UID: \"a5f306dc-d337-4d52-8d54-f115dc90fcc5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-csd2f"
Sep 16 14:15:35.932826 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.932823    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc5tt\" (UniqueName: \"kubernetes.io/projected/4f3af190-2324-42b9-8ade-3ddd9189df6e-kube-api-access-fc5tt\") pod \"downloads-76f99584f4-phq9c\" (UID: \"4f3af190-2324-42b9-8ade-3ddd9189df6e\") " pod="openshift-console/downloads-76f99584f4-phq9c"
Sep 16 14:15:35.933001 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.932885    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d091897-2f8d-4422-9f82-b7882d9cc02b-tmp\") pod \"klusterlet-addon-workmgr-5b978cdb5f-8gxfw\" (UID: \"6d091897-2f8d-4422-9f82-b7882d9cc02b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw"
Sep 16 14:15:35.935095 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.935076    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6d091897-2f8d-4422-9f82-b7882d9cc02b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b978cdb5f-8gxfw\" (UID: \"6d091897-2f8d-4422-9f82-b7882d9cc02b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw"
Sep 16 14:15:35.946266 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:35.946215    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcfk4\" (UniqueName: \"kubernetes.io/projected/6d091897-2f8d-4422-9f82-b7882d9cc02b-kube-api-access-fcfk4\") pod \"klusterlet-addon-workmgr-5b978cdb5f-8gxfw\" (UID: \"6d091897-2f8d-4422-9f82-b7882d9cc02b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw"
Sep 16 14:15:36.006861 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:36.006828    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw"
Sep 16 14:15:36.033980 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:36.033952    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5f306dc-d337-4d52-8d54-f115dc90fcc5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-csd2f\" (UID: \"a5f306dc-d337-4d52-8d54-f115dc90fcc5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-csd2f"
Sep 16 14:15:36.034096 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:36.033982    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc5tt\" (UniqueName: \"kubernetes.io/projected/4f3af190-2324-42b9-8ade-3ddd9189df6e-kube-api-access-fc5tt\") pod \"downloads-76f99584f4-phq9c\" (UID: \"4f3af190-2324-42b9-8ade-3ddd9189df6e\") " pod="openshift-console/downloads-76f99584f4-phq9c"
Sep 16 14:15:36.034096 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:36.034056    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5f306dc-d337-4d52-8d54-f115dc90fcc5-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-csd2f\" (UID: \"a5f306dc-d337-4d52-8d54-f115dc90fcc5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-csd2f"
Sep 16 14:15:36.034626 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:36.034607    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5f306dc-d337-4d52-8d54-f115dc90fcc5-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-csd2f\" (UID: \"a5f306dc-d337-4d52-8d54-f115dc90fcc5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-csd2f"
Sep 16 14:15:36.036205 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:36.036187    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5f306dc-d337-4d52-8d54-f115dc90fcc5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-csd2f\" (UID: \"a5f306dc-d337-4d52-8d54-f115dc90fcc5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-csd2f"
Sep 16 14:15:36.046955 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:36.046924    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc5tt\" (UniqueName: \"kubernetes.io/projected/4f3af190-2324-42b9-8ade-3ddd9189df6e-kube-api-access-fc5tt\") pod \"downloads-76f99584f4-phq9c\" (UID: \"4f3af190-2324-42b9-8ade-3ddd9189df6e\") " pod="openshift-console/downloads-76f99584f4-phq9c"
Sep 16 14:15:36.103179 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:36.103148    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-phq9c"
Sep 16 14:15:36.111931 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:36.111914    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-csd2f"
Sep 16 14:15:36.283290 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:36.283262    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw"]
Sep 16 14:15:36.287402 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:15:36.287377    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d091897_2f8d_4422_9f82_b7882d9cc02b.slice/crio-cfa90e9262ef73e7454e9bcec4214c7234a4397c0f0703854d98d2747578aafc WatchSource:0}: Error finding container cfa90e9262ef73e7454e9bcec4214c7234a4397c0f0703854d98d2747578aafc: Status 404 returned error can't find the container with id cfa90e9262ef73e7454e9bcec4214c7234a4397c0f0703854d98d2747578aafc
Sep 16 14:15:36.377899 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:36.377864    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-phq9c"]
Sep 16 14:15:36.380723 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:36.380702    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-csd2f"]
Sep 16 14:15:36.380847 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:15:36.380819    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3af190_2324_42b9_8ade_3ddd9189df6e.slice/crio-e91f015b85d8a705d904033087684cda542bfa6081c6630bc0a4fe6ed607b56a WatchSource:0}: Error finding container e91f015b85d8a705d904033087684cda542bfa6081c6630bc0a4fe6ed607b56a: Status 404 returned error can't find the container with id e91f015b85d8a705d904033087684cda542bfa6081c6630bc0a4fe6ed607b56a
Sep 16 14:15:36.383577 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:15:36.383554    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f306dc_d337_4d52_8d54_f115dc90fcc5.slice/crio-26dab018b8399b7cfffb31ab6a92c887b7aa7267f0399ec02c101b26b5746796 WatchSource:0}: Error finding container 26dab018b8399b7cfffb31ab6a92c887b7aa7267f0399ec02c101b26b5746796: Status 404 returned error can't find the container with id 26dab018b8399b7cfffb31ab6a92c887b7aa7267f0399ec02c101b26b5746796
Sep 16 14:15:37.270207 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:37.270164    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-phq9c" event={"ID":"4f3af190-2324-42b9-8ade-3ddd9189df6e","Type":"ContainerStarted","Data":"e91f015b85d8a705d904033087684cda542bfa6081c6630bc0a4fe6ed607b56a"}
Sep 16 14:15:37.271561 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:37.271528    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw" event={"ID":"6d091897-2f8d-4422-9f82-b7882d9cc02b","Type":"ContainerStarted","Data":"cfa90e9262ef73e7454e9bcec4214c7234a4397c0f0703854d98d2747578aafc"}
Sep 16 14:15:37.272788 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:37.272756    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-csd2f" event={"ID":"a5f306dc-d337-4d52-8d54-f115dc90fcc5","Type":"ContainerStarted","Data":"26dab018b8399b7cfffb31ab6a92c887b7aa7267f0399ec02c101b26b5746796"}
Sep 16 14:15:38.278047 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:38.278005    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-csd2f" event={"ID":"a5f306dc-d337-4d52-8d54-f115dc90fcc5","Type":"ContainerStarted","Data":"b7eae823363d0af9e8d59d458180125e4a7a01b9bdd987fcd0c4c0e46cba0e7a"}
Sep 16 14:15:38.314084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:38.314031    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-csd2f" podStartSLOduration=2.24324078 podStartE2EDuration="3.314010835s" totalImagesPullingTime="1.070770055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:38.313010558 +0000 UTC m=+88.859752489" watchObservedRunningTime="2026-09-16 14:15:38.314010835 +0000 UTC m=+88.860752765"
Sep 16 14:15:39.662250 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:39.662201    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5l2fd\" (UID: \"136f6c0b-4717-4e2b-b788-fa72b68f6eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:39.665231 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:39.665204    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/136f6c0b-4717-4e2b-b788-fa72b68f6eb8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5l2fd\" (UID: \"136f6c0b-4717-4e2b-b788-fa72b68f6eb8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:39.721400 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:39.721277    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m7tsr\""
Sep 16 14:15:39.728834 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:39.728808    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"
Sep 16 14:15:40.553194 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:40.553103    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd"]
Sep 16 14:15:41.853609 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:15:41.853572    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136f6c0b_4717_4e2b_b788_fa72b68f6eb8.slice/crio-914bc4f06f2ca29792356892ea1d63e35ffab6aa261651ca1a72bbf43e95814d WatchSource:0}: Error finding container 914bc4f06f2ca29792356892ea1d63e35ffab6aa261651ca1a72bbf43e95814d: Status 404 returned error can't find the container with id 914bc4f06f2ca29792356892ea1d63e35ffab6aa261651ca1a72bbf43e95814d
Sep 16 14:15:42.295772 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:42.295730    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd" event={"ID":"136f6c0b-4717-4e2b-b788-fa72b68f6eb8","Type":"ContainerStarted","Data":"914bc4f06f2ca29792356892ea1d63e35ffab6aa261651ca1a72bbf43e95814d"}
Sep 16 14:15:42.297595 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:42.297565    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw" event={"ID":"6d091897-2f8d-4422-9f82-b7882d9cc02b","Type":"ContainerStarted","Data":"7cdf3266c885a4c7d45ed64cc1007eeb4f329ff5b6dde39896d21fd8ea963f85"}
Sep 16 14:15:42.297859 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:42.297808    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw"
Sep 16 14:15:42.299583 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:42.299563    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw"
Sep 16 14:15:42.338659 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:42.338602    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b978cdb5f-8gxfw" podStartSLOduration=1.736426303 podStartE2EDuration="7.338583275s" totalImagesPullingTime="5.602156972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:42.337508501 +0000 UTC m=+92.884250435" watchObservedRunningTime="2026-09-16 14:15:42.338583275 +0000 UTC m=+92.885325204"
Sep 16 14:15:43.002089 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.001602    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-897f6dc4-w774t"]
Sep 16 14:15:43.004040 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.003922    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.009279 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.009057    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gqfcf\""
Sep 16 14:15:43.009279 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.009107    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 14:15:43.009279 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.009057    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 14:15:43.009279 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.009222    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 14:15:43.009279 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.009256    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 14:15:43.009611 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.009343    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 14:15:43.015236 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.015214    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-897f6dc4-w774t"]
Sep 16 14:15:43.091368 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.091330    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-oauth-config\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.091368 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.091384    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-oauth-serving-cert\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.091781 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.091431    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-serving-cert\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.091781 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.091477    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-service-ca\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.091781 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.091556    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2mt6\" (UniqueName: \"kubernetes.io/projected/63f865cf-7b11-4a79-bfa7-36a02e00a180-kube-api-access-f2mt6\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.091781 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.091620    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-config\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.192020 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.191983    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-oauth-serving-cert\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.192186 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.192044    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-serving-cert\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.192186 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.192069    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-service-ca\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.192186 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.192107    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2mt6\" (UniqueName: \"kubernetes.io/projected/63f865cf-7b11-4a79-bfa7-36a02e00a180-kube-api-access-f2mt6\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.192186 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.192156    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-config\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.192403 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.192220    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-oauth-config\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.192811 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.192787    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-oauth-serving-cert\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.192927 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.192821    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-service-ca\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.193171 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.193126    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-config\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.195464 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.195419    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-serving-cert\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.195951 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.195929    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-oauth-config\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.201066 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.201044    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2mt6\" (UniqueName: \"kubernetes.io/projected/63f865cf-7b11-4a79-bfa7-36a02e00a180-kube-api-access-f2mt6\") pod \"console-897f6dc4-w774t\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") " pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.316712 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.316616    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:43.597514 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:43.597432    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-897f6dc4-w774t"]
Sep 16 14:15:44.077380 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:15:44.077324    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f865cf_7b11_4a79_bfa7_36a02e00a180.slice/crio-901d696745b471c139e0fc020515883aee00b7912b7d022aa1b7c6d04e2d6896 WatchSource:0}: Error finding container 901d696745b471c139e0fc020515883aee00b7912b7d022aa1b7c6d04e2d6896: Status 404 returned error can't find the container with id 901d696745b471c139e0fc020515883aee00b7912b7d022aa1b7c6d04e2d6896
Sep 16 14:15:44.306165 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:44.306119    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd" event={"ID":"136f6c0b-4717-4e2b-b788-fa72b68f6eb8","Type":"ContainerStarted","Data":"2335682226e9adaa470e3ac6bdd45822ee07f59479b5b8bd8438acde932dca92"}
Sep 16 14:15:44.307415 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:44.307388    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-897f6dc4-w774t" event={"ID":"63f865cf-7b11-4a79-bfa7-36a02e00a180","Type":"ContainerStarted","Data":"901d696745b471c139e0fc020515883aee00b7912b7d022aa1b7c6d04e2d6896"}
Sep 16 14:15:44.325167 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:44.325120    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5l2fd" podStartSLOduration=35.078332545 podStartE2EDuration="37.325106832s" totalImagesPullingTime="2.246774287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:44.324330973 +0000 UTC m=+94.871072904" watchObservedRunningTime="2026-09-16 14:15:44.325106832 +0000 UTC m=+94.871848763"
Sep 16 14:15:44.607365 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:44.607333    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-54q4d"]
Sep 16 14:15:44.609295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:44.609275    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-54q4d"
Sep 16 14:15:44.612544 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:44.612152    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 14:15:44.612544 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:44.612402    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9qs6g\""
Sep 16 14:15:44.619733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:44.619677    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-54q4d"]
Sep 16 14:15:44.705541 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:44.705504    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dd7160a7-b3ca-4104-a986-2ad932a3874d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-54q4d\" (UID: \"dd7160a7-b3ca-4104-a986-2ad932a3874d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-54q4d"
Sep 16 14:15:44.806317 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:44.806287    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dd7160a7-b3ca-4104-a986-2ad932a3874d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-54q4d\" (UID: \"dd7160a7-b3ca-4104-a986-2ad932a3874d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-54q4d"
Sep 16 14:15:44.809165 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:44.809133    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dd7160a7-b3ca-4104-a986-2ad932a3874d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-54q4d\" (UID: \"dd7160a7-b3ca-4104-a986-2ad932a3874d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-54q4d"
Sep 16 14:15:44.922809 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:44.922734    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-54q4d"
Sep 16 14:15:45.109514 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:45.109482    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:15:45.109998 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:45.109590    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:15:45.112168 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:45.112136    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68b65eb9-84ac-48c6-8ffc-49c9eee408b3-metrics-tls\") pod \"dns-default-9x4k6\" (UID: \"68b65eb9-84ac-48c6-8ffc-49c9eee408b3\") " pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:15:45.112321 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:45.112303    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls\") pod \"image-registry-598474997c-5r45z\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") " pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:15:45.127299 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:45.127241    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9f4mh\""
Sep 16 14:15:45.134849 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:45.134814    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:15:45.206703 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:45.204222    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-54q4d"]
Sep 16 14:15:45.209385 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:15:45.209342    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7160a7_b3ca_4104_a986_2ad932a3874d.slice/crio-5bc5da6629cc3974088906a7cb174e936670c0d34a041b1e3cb48d05123525c2 WatchSource:0}: Error finding container 5bc5da6629cc3974088906a7cb174e936670c0d34a041b1e3cb48d05123525c2: Status 404 returned error can't find the container with id 5bc5da6629cc3974088906a7cb174e936670c0d34a041b1e3cb48d05123525c2
Sep 16 14:15:45.312594 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:45.312556    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-54q4d" event={"ID":"dd7160a7-b3ca-4104-a986-2ad932a3874d","Type":"ContainerStarted","Data":"5bc5da6629cc3974088906a7cb174e936670c0d34a041b1e3cb48d05123525c2"}
Sep 16 14:15:45.403287 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:45.403255    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-566s2\""
Sep 16 14:15:45.409541 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:45.409387    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9x4k6"]
Sep 16 14:15:45.412398 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:45.412369    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:15:45.413139 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:15:45.413110    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b65eb9_84ac_48c6_8ffc_49c9eee408b3.slice/crio-938748964d166e6e99b18d03750a58aa5b9977524b511b9e12fb6b8ca10e86f3 WatchSource:0}: Error finding container 938748964d166e6e99b18d03750a58aa5b9977524b511b9e12fb6b8ca10e86f3: Status 404 returned error can't find the container with id 938748964d166e6e99b18d03750a58aa5b9977524b511b9e12fb6b8ca10e86f3
Sep 16 14:15:45.700600 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:45.700559    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-598474997c-5r45z"]
Sep 16 14:15:45.703917 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:15:45.703875    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb095cf19_aa32_4e5c_af3c_7bd969dd0895.slice/crio-9741551f07f0edff73ea6dbaf3f4e3d3cf2f11dc09ab44cc1123c69edf269190 WatchSource:0}: Error finding container 9741551f07f0edff73ea6dbaf3f4e3d3cf2f11dc09ab44cc1123c69edf269190: Status 404 returned error can't find the container with id 9741551f07f0edff73ea6dbaf3f4e3d3cf2f11dc09ab44cc1123c69edf269190
Sep 16 14:15:46.319681 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:46.319649    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-598474997c-5r45z" event={"ID":"b095cf19-aa32-4e5c-af3c-7bd969dd0895","Type":"ContainerStarted","Data":"55f90726c0dd43760fe93e636e6411463d160298ebfa4f5220df57d6df5c734f"}
Sep 16 14:15:46.320140 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:46.319709    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-598474997c-5r45z" event={"ID":"b095cf19-aa32-4e5c-af3c-7bd969dd0895","Type":"ContainerStarted","Data":"9741551f07f0edff73ea6dbaf3f4e3d3cf2f11dc09ab44cc1123c69edf269190"}
Sep 16 14:15:46.320436 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:46.320389    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:15:46.323952 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:46.323881    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9x4k6" event={"ID":"68b65eb9-84ac-48c6-8ffc-49c9eee408b3","Type":"ContainerStarted","Data":"938748964d166e6e99b18d03750a58aa5b9977524b511b9e12fb6b8ca10e86f3"}
Sep 16 14:15:46.344066 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:46.343173    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-598474997c-5r45z" podStartSLOduration=96.343154685 podStartE2EDuration="1m36.343154685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:46.342795857 +0000 UTC m=+96.889537792" watchObservedRunningTime="2026-09-16 14:15:46.343154685 +0000 UTC m=+96.889896619"
Sep 16 14:15:49.338141 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:49.338099    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-897f6dc4-w774t" event={"ID":"63f865cf-7b11-4a79-bfa7-36a02e00a180","Type":"ContainerStarted","Data":"e820b5bcdc31992325e5490dcac73c5c3f0c984c0651aaf12c40920708b995ed"}
Sep 16 14:15:49.339854 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:49.339827    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-54q4d" event={"ID":"dd7160a7-b3ca-4104-a986-2ad932a3874d","Type":"ContainerStarted","Data":"8b49f4f3c9ff464dbfe35e3843067eb24fe24a3d10985334ab94d4e638db1c02"}
Sep 16 14:15:49.340003 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:49.339970    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-54q4d"
Sep 16 14:15:49.342088 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:49.342049    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9x4k6" event={"ID":"68b65eb9-84ac-48c6-8ffc-49c9eee408b3","Type":"ContainerStarted","Data":"3a6758fd9f95176762f848dfc35814944108295b23c016c1a0d1f1f8cdb0cce8"}
Sep 16 14:15:49.342198 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:49.342114    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:15:49.342198 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:49.342143    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9x4k6" event={"ID":"68b65eb9-84ac-48c6-8ffc-49c9eee408b3","Type":"ContainerStarted","Data":"6516cb59587eb01e3a0a70272c14782526dda1529b68ea5f0cfb2001ede9c81e"}
Sep 16 14:15:49.346020 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:49.345977    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-54q4d"
Sep 16 14:15:49.357926 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:49.357870    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-897f6dc4-w774t" podStartSLOduration=2.970141949 podStartE2EDuration="7.357856613s" totalImagesPullingTime="4.387714664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:49.356824581 +0000 UTC m=+99.903566608" watchObservedRunningTime="2026-09-16 14:15:49.357856613 +0000 UTC m=+99.904598543"
Sep 16 14:15:49.373745 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:49.373654    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-54q4d" podStartSLOduration=2.124416003 podStartE2EDuration="5.37364089s" totalImagesPullingTime="3.249224887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:49.37293188 +0000 UTC m=+99.919673813" watchObservedRunningTime="2026-09-16 14:15:49.37364089 +0000 UTC m=+99.920382822"
Sep 16 14:15:49.391190 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:49.391137    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9x4k6" podStartSLOduration=65.338965782 podStartE2EDuration="1m8.39111984s" totalImagesPullingTime="3.052154058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:49.390344252 +0000 UTC m=+99.937086217" watchObservedRunningTime="2026-09-16 14:15:49.39111984 +0000 UTC m=+99.937861763"
Sep 16 14:15:53.316885 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:53.316844    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:53.317318 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:53.316975    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:53.322347 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:53.322319    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:53.362701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:53.362678    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:15:55.091413 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.091381    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"]
Sep 16 14:15:55.096794 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.096771    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dkdrg"]
Sep 16 14:15:55.096969 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.096934    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.100038 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.099943    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-j9hjj\""
Sep 16 14:15:55.100264 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.100220    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 14:15:55.100264 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.100251    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.101085 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.101030    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 14:15:55.101085 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.101058    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 14:15:55.101085 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.101036    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 14:15:55.103900 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.103865    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 14:15:55.104019 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.103875    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 14:15:55.104261 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.104235    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-njn57\""
Sep 16 14:15:55.104364 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.104330    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 14:15:55.120305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.120281    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"]
Sep 16 14:15:55.197560 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.197499    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62ac91a1-838a-4136-b993-c630a0fde35a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.197560 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.197546    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62ac91a1-838a-4136-b993-c630a0fde35a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.197560 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.197575    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-root\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.197922 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.197599    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.197922 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.197628    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/62ac91a1-838a-4136-b993-c630a0fde35a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.197922 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.197670    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-node-exporter-accelerators-collector-config\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.197922 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.197738    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/62ac91a1-838a-4136-b993-c630a0fde35a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.197922 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.197779    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-metrics-client-ca\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.197922 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.197805    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh4jt\" (UniqueName: \"kubernetes.io/projected/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-kube-api-access-rh4jt\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.198198 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.197934    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9r4x\" (UniqueName: \"kubernetes.io/projected/62ac91a1-838a-4136-b993-c630a0fde35a-kube-api-access-p9r4x\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.198198 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.197968    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-node-exporter-textfile\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.198198 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.198000    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-node-exporter-wtmp\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.198198 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.198025    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-sys\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.198198 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.198118    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-node-exporter-tls\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.198198 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.198148    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/62ac91a1-838a-4136-b993-c630a0fde35a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.299108 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299070    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-node-exporter-textfile\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.299293 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299126    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-node-exporter-wtmp\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.299293 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299157    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-sys\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.299293 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299207    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-node-exporter-tls\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.299293 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299235    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/62ac91a1-838a-4136-b993-c630a0fde35a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.299293 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299266    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62ac91a1-838a-4136-b993-c630a0fde35a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.299293 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299290    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62ac91a1-838a-4136-b993-c630a0fde35a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.299600 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299317    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-root\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.299600 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299329    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-node-exporter-wtmp\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.299600 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299354    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.299600 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:55.299460    2553 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 16 14:15:55.299600 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299504    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-node-exporter-textfile\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.299600 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:15:55.299535    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62ac91a1-838a-4136-b993-c630a0fde35a-kube-state-metrics-tls podName:62ac91a1-838a-4136-b993-c630a0fde35a nodeName:}" failed. No retries permitted until 2026-09-16 14:15:55.799514714 +0000 UTC m=+106.346256646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/62ac91a1-838a-4136-b993-c630a0fde35a-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-5pxl9" (UID: "62ac91a1-838a-4136-b993-c630a0fde35a") : secret "kube-state-metrics-tls" not found
Sep 16 14:15:55.299600 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299561    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/62ac91a1-838a-4136-b993-c630a0fde35a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.299990 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299625    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-node-exporter-accelerators-collector-config\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.299990 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299663    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/62ac91a1-838a-4136-b993-c630a0fde35a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.299990 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299694    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-metrics-client-ca\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.299990 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299718    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh4jt\" (UniqueName: \"kubernetes.io/projected/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-kube-api-access-rh4jt\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.299990 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299779    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9r4x\" (UniqueName: \"kubernetes.io/projected/62ac91a1-838a-4136-b993-c630a0fde35a-kube-api-access-p9r4x\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.299990 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.299847    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-root\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.300280 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.300124    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62ac91a1-838a-4136-b993-c630a0fde35a-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.300280 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.300186    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-sys\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.300443 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.300416    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-node-exporter-accelerators-collector-config\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.300792 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.300728    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/62ac91a1-838a-4136-b993-c630a0fde35a-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.300919 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.300903    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/62ac91a1-838a-4136-b993-c630a0fde35a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.301103 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.301078    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-metrics-client-ca\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.302294 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.302122    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-node-exporter-tls\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.302464 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.302439    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62ac91a1-838a-4136-b993-c630a0fde35a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.302983 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.302964    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.319583 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.319556    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9r4x\" (UniqueName: \"kubernetes.io/projected/62ac91a1-838a-4136-b993-c630a0fde35a-kube-api-access-p9r4x\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.319693 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.319606    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh4jt\" (UniqueName: \"kubernetes.io/projected/a2de9462-1851-4e5e-90c6-3fe7fb19b91c-kube-api-access-rh4jt\") pod \"node-exporter-dkdrg\" (UID: \"a2de9462-1851-4e5e-90c6-3fe7fb19b91c\") " pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.415647 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.415572    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dkdrg"
Sep 16 14:15:55.805660 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.805393    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/62ac91a1-838a-4136-b993-c630a0fde35a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.808348 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:55.808324    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/62ac91a1-838a-4136-b993-c630a0fde35a-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-5pxl9\" (UID: \"62ac91a1-838a-4136-b993-c630a0fde35a\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:55.882852 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:15:55.882822    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2de9462_1851_4e5e_90c6_3fe7fb19b91c.slice/crio-439cb78186b7eeb561e511c63d9ade9d5f50c93ea55c5960c36db1a2c2d29d68 WatchSource:0}: Error finding container 439cb78186b7eeb561e511c63d9ade9d5f50c93ea55c5960c36db1a2c2d29d68: Status 404 returned error can't find the container with id 439cb78186b7eeb561e511c63d9ade9d5f50c93ea55c5960c36db1a2c2d29d68
Sep 16 14:15:56.009815 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.009786    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"
Sep 16 14:15:56.208589 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.208410    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 14:15:56.216589 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.216511    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.219864 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.219836    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 14:15:56.220629 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.220454    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 14:15:56.220629 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.220485    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 14:15:56.220629 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.220503    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 14:15:56.220948 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.220909    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 14:15:56.221017 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.220955    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 14:15:56.221066 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.220965    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 14:15:56.221217 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.221131    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hgsv5\""
Sep 16 14:15:56.221289 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.221264    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 14:15:56.222185 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.222045    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 14:15:56.232837 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.232817    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 14:15:56.304208 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.304169    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9"]
Sep 16 14:15:56.310966 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.310373    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/984dce34-c05d-48c1-9686-055dabd6364e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.310966 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.310431    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-web-config\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.310966 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.310459    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-config-volume\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.310966 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.310487    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.310966 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.310516    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/984dce34-c05d-48c1-9686-055dabd6364e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.310966 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.310578    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/984dce34-c05d-48c1-9686-055dabd6364e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.310966 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.310613    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/984dce34-c05d-48c1-9686-055dabd6364e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.310966 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.310644    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.310966 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.310679    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xpp6\" (UniqueName: \"kubernetes.io/projected/984dce34-c05d-48c1-9686-055dabd6364e-kube-api-access-4xpp6\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.310966 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.310715    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.310966 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.310750    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.310966 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.310787    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/984dce34-c05d-48c1-9686-055dabd6364e-config-out\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.310966 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.310811    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.372588 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.372551    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dkdrg" event={"ID":"a2de9462-1851-4e5e-90c6-3fe7fb19b91c","Type":"ContainerStarted","Data":"439cb78186b7eeb561e511c63d9ade9d5f50c93ea55c5960c36db1a2c2d29d68"}
Sep 16 14:15:56.374216 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.374181    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9" event={"ID":"62ac91a1-838a-4136-b993-c630a0fde35a","Type":"ContainerStarted","Data":"1366b5ca5cc55221d994ff6fa6513a77c80d5cf170ec62c69fcb889da1d56f3a"}
Sep 16 14:15:56.376129 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.376101    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-phq9c" event={"ID":"4f3af190-2324-42b9-8ade-3ddd9189df6e","Type":"ContainerStarted","Data":"4bcc322d2d489e44c46b3a44b3136005b5f684f9ff0cc020e4d22afcf200f9f8"}
Sep 16 14:15:56.376356 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.376338    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-phq9c"
Sep 16 14:15:56.393861 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.393833    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-phq9c"
Sep 16 14:15:56.395391 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.395345    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-phq9c" podStartSLOduration=1.816569698 podStartE2EDuration="21.39532804s" totalImagesPullingTime="19.578758342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:56.393656708 +0000 UTC m=+106.940398661" watchObservedRunningTime="2026-09-16 14:15:56.39532804 +0000 UTC m=+106.942069972"
Sep 16 14:15:56.411299 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.411270    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/984dce34-c05d-48c1-9686-055dabd6364e-config-out\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.411458 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.411306    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.411458 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.411332    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/984dce34-c05d-48c1-9686-055dabd6364e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.411458 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.411370    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-web-config\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.411458 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.411394    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-config-volume\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.411458 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.411421    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.411458 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.411448    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/984dce34-c05d-48c1-9686-055dabd6364e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.411820 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.411506    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/984dce34-c05d-48c1-9686-055dabd6364e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.411820 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.411541    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/984dce34-c05d-48c1-9686-055dabd6364e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.411820 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.411570    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.411820 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.411601    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpp6\" (UniqueName: \"kubernetes.io/projected/984dce34-c05d-48c1-9686-055dabd6364e-kube-api-access-4xpp6\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.411820 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.411640    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.411820 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.411675    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.412546 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.412436    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/984dce34-c05d-48c1-9686-055dabd6364e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.413901 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.413415    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/984dce34-c05d-48c1-9686-055dabd6364e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.413901 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.413426    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/984dce34-c05d-48c1-9686-055dabd6364e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.415348 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.415318    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.417023 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.416974    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-web-config\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.418654 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.418630    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/984dce34-c05d-48c1-9686-055dabd6364e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.419926 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.419869    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-config-volume\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.420739 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.420692    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/984dce34-c05d-48c1-9686-055dabd6364e-config-out\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.421020 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.420976    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.422640 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.422258    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.422640 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.422474    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.422640 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.422600    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/984dce34-c05d-48c1-9686-055dabd6364e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.444026 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.443991    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xpp6\" (UniqueName: \"kubernetes.io/projected/984dce34-c05d-48c1-9686-055dabd6364e-kube-api-access-4xpp6\") pod \"alertmanager-main-0\" (UID: \"984dce34-c05d-48c1-9686-055dabd6364e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.528905 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.528858    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:15:56.831106 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:56.831007    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 14:15:56.833300 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:15:56.833251    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984dce34_c05d_48c1_9686_055dabd6364e.slice/crio-0dab0a88cbc3501b6930861bbfd1018546920851af6a744c84280b17ca19b8e4 WatchSource:0}: Error finding container 0dab0a88cbc3501b6930861bbfd1018546920851af6a744c84280b17ca19b8e4: Status 404 returned error can't find the container with id 0dab0a88cbc3501b6930861bbfd1018546920851af6a744c84280b17ca19b8e4
Sep 16 14:15:57.382681 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:57.382633    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984dce34-c05d-48c1-9686-055dabd6364e","Type":"ContainerStarted","Data":"0dab0a88cbc3501b6930861bbfd1018546920851af6a744c84280b17ca19b8e4"}
Sep 16 14:15:58.355618 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:58.355582    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-598474997c-5r45z"]
Sep 16 14:15:58.389841 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:58.389794    2553 generic.go:356] "Generic (PLEG): container finished" podID="a2de9462-1851-4e5e-90c6-3fe7fb19b91c" containerID="bd7e8f61be8d653fdd66e74e000fd18622edf056a1551b2e5ce08453a80bd681" exitCode=0
Sep 16 14:15:58.390322 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:58.390018    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dkdrg" event={"ID":"a2de9462-1851-4e5e-90c6-3fe7fb19b91c","Type":"ContainerDied","Data":"bd7e8f61be8d653fdd66e74e000fd18622edf056a1551b2e5ce08453a80bd681"}
Sep 16 14:15:59.348210 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:15:59.348172    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9x4k6"
Sep 16 14:16:00.402440 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:00.402400    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dkdrg" event={"ID":"a2de9462-1851-4e5e-90c6-3fe7fb19b91c","Type":"ContainerStarted","Data":"91c801caf90168140d5ff539904ce3be1e2e1b18ad3d346a6b66ca5296a894df"}
Sep 16 14:16:00.402924 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:00.402467    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dkdrg" event={"ID":"a2de9462-1851-4e5e-90c6-3fe7fb19b91c","Type":"ContainerStarted","Data":"93ab7cdab94bb051abe7cdfb1e6a11fedd41664aa13d29176372ff272ca1f9ef"}
Sep 16 14:16:00.404305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:00.404279    2553 generic.go:356] "Generic (PLEG): container finished" podID="984dce34-c05d-48c1-9686-055dabd6364e" containerID="ef9e40df99c9ba68de68f951e04306ef4db0eab3fe2c49d8fc97891c0d5ea615" exitCode=0
Sep 16 14:16:00.404421 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:00.404371    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984dce34-c05d-48c1-9686-055dabd6364e","Type":"ContainerDied","Data":"ef9e40df99c9ba68de68f951e04306ef4db0eab3fe2c49d8fc97891c0d5ea615"}
Sep 16 14:16:00.407062 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:00.407036    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9" event={"ID":"62ac91a1-838a-4136-b993-c630a0fde35a","Type":"ContainerStarted","Data":"ba39c9e42f639c86927ec671c4ccb31ba139cd8f8b86eaa0112437c766a019f4"}
Sep 16 14:16:00.407155 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:00.407097    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9" event={"ID":"62ac91a1-838a-4136-b993-c630a0fde35a","Type":"ContainerStarted","Data":"e52b6201fbce573615af5a3ad8b29be9fd9a680d575d0b96881e01580770fe82"}
Sep 16 14:16:00.407155 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:00.407128    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9" event={"ID":"62ac91a1-838a-4136-b993-c630a0fde35a","Type":"ContainerStarted","Data":"4ea3e4d899a0c24bdd1865450e7d0bf245c053d1df362a9324674b1e799133c1"}
Sep 16 14:16:00.425777 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:00.425731    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dkdrg" podStartSLOduration=3.595818495 podStartE2EDuration="5.42571907s" totalImagesPullingTime="1.728918851s" totalInitContainerRuntime="100.981724ms" isStatefulPod=true podCreationTimestamp="2026-09-16 14:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:16:00.425510055 +0000 UTC m=+110.972251987" watchObservedRunningTime="2026-09-16 14:16:00.42571907 +0000 UTC m=+110.972461340"
Sep 16 14:16:00.456361 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:00.456306    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-5pxl9" podStartSLOduration=2.402535151 podStartE2EDuration="5.456288815s" totalImagesPullingTime="3.053753664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:16:00.455974096 +0000 UTC m=+111.002716025" watchObservedRunningTime="2026-09-16 14:16:00.456288815 +0000 UTC m=+111.003030749"
Sep 16 14:16:01.365606 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.365569    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 14:16:01.396611 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.396574    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 14:16:01.396804 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.396749    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.399736 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.399709    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 14:16:01.399865 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.399773    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-54rec2ebqgesm\""
Sep 16 14:16:01.400220 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.400200    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 14:16:01.400323 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.400295    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 14:16:01.400382 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.400358    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 14:16:01.400721 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.400695    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 14:16:01.400817 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.400791    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nhxrs\""
Sep 16 14:16:01.401237 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.401217    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 14:16:01.401370 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.401308    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 14:16:01.401370 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.401364    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 14:16:01.401501 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.401432    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 14:16:01.402837 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.402816    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 14:16:01.404151 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.404130    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 14:16:01.404254 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.404135    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 14:16:01.404408 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.404389    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 14:16:01.405523 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.405501    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 14:16:01.408978 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.408957    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 14:16:01.466236 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466197    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.466236 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466237    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.466477 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466269    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.466477 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466299    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-946xp\" (UniqueName: \"kubernetes.io/projected/fbc3a51b-ec10-43c1-acf9-4ccd064500be-kube-api-access-946xp\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.466477 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466382    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.466621 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466490    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fbc3a51b-ec10-43c1-acf9-4ccd064500be-config-out\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.466680 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466605    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.466680 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466666    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fbc3a51b-ec10-43c1-acf9-4ccd064500be-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.466780 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466696    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.466780 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466729    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-config\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.466780 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466764    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.466960 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466824    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.466960 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466848    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.466960 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466916    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.466960 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466941    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.467152 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466963    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fbc3a51b-ec10-43c1-acf9-4ccd064500be-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.467152 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.466988    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.467152 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.467045    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.467152 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.467112    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-web-config\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.467348 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.467229    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568110 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568069    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568110 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568115    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fbc3a51b-ec10-43c1-acf9-4ccd064500be-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568364 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568140    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568364 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568184    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-config\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568364 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568207    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568364 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568245    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568364 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568270    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568364 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568299    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568364 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568327    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568364 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568351    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fbc3a51b-ec10-43c1-acf9-4ccd064500be-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568742 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568376    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568742 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568408    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568742 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568435    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-web-config\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568742 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568492    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568742 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568525    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568742 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568548    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568742 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568577    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568742 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568607    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-946xp\" (UniqueName: \"kubernetes.io/projected/fbc3a51b-ec10-43c1-acf9-4ccd064500be-kube-api-access-946xp\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568742 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568657    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.568742 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.568701    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fbc3a51b-ec10-43c1-acf9-4ccd064500be-config-out\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.569241 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.569080    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.569241 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.569118    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.569241 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.569234    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.569659 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.569635    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.571078 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.571048    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fbc3a51b-ec10-43c1-acf9-4ccd064500be-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.571970 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.571941    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.572075 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.572041    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.575040 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.574938    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fbc3a51b-ec10-43c1-acf9-4ccd064500be-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.576077 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.575876    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fbc3a51b-ec10-43c1-acf9-4ccd064500be-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.576347 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.576231    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fbc3a51b-ec10-43c1-acf9-4ccd064500be-config-out\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.576801 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.576470    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.576801 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.576664    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-config\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.576801 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.576763    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-web-config\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.577010 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.576822    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.577010 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.576847    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.577160 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.577115    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.577655 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.577611    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.578038 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.578012    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.579641 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.579610    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fbc3a51b-ec10-43c1-acf9-4ccd064500be-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.580557 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.580530    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-946xp\" (UniqueName: \"kubernetes.io/projected/fbc3a51b-ec10-43c1-acf9-4ccd064500be-kube-api-access-946xp\") pod \"prometheus-k8s-0\" (UID: \"fbc3a51b-ec10-43c1-acf9-4ccd064500be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:01.710278 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:01.710194    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:02.031112 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:02.031081    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 14:16:02.036115 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:16:02.036055    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc3a51b_ec10_43c1_acf9_4ccd064500be.slice/crio-b08a4045b2d7930b1482d7fdac913f22b35a900d2a84bf5ac0d4b6e9d5c43484 WatchSource:0}: Error finding container b08a4045b2d7930b1482d7fdac913f22b35a900d2a84bf5ac0d4b6e9d5c43484: Status 404 returned error can't find the container with id b08a4045b2d7930b1482d7fdac913f22b35a900d2a84bf5ac0d4b6e9d5c43484
Sep 16 14:16:02.419827 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:02.419744    2553 generic.go:356] "Generic (PLEG): container finished" podID="fbc3a51b-ec10-43c1-acf9-4ccd064500be" containerID="7888d747580cdc076d57f0e43dc1a401cae8b538125bae63b41c68794ead83b0" exitCode=0
Sep 16 14:16:02.420274 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:02.419870    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbc3a51b-ec10-43c1-acf9-4ccd064500be","Type":"ContainerDied","Data":"7888d747580cdc076d57f0e43dc1a401cae8b538125bae63b41c68794ead83b0"}
Sep 16 14:16:02.420274 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:02.419970    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbc3a51b-ec10-43c1-acf9-4ccd064500be","Type":"ContainerStarted","Data":"b08a4045b2d7930b1482d7fdac913f22b35a900d2a84bf5ac0d4b6e9d5c43484"}
Sep 16 14:16:04.436071 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:04.435983    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984dce34-c05d-48c1-9686-055dabd6364e","Type":"ContainerStarted","Data":"57105355f1dde534f647e6a1207c3d54bc280595983ffa3fcf4b84837c0e9b61"}
Sep 16 14:16:04.436507 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:04.436074    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984dce34-c05d-48c1-9686-055dabd6364e","Type":"ContainerStarted","Data":"cae9f6f7d9818dfa5e943e11017caf55006321a7f9ef0fd71847ac857ca49a67"}
Sep 16 14:16:04.436507 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:04.436151    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984dce34-c05d-48c1-9686-055dabd6364e","Type":"ContainerStarted","Data":"73ae3efd7670afcca1d286d9ac9d1aaadcaac05efe4936711b5e8b6c3d0bcec9"}
Sep 16 14:16:04.436507 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:04.436197    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984dce34-c05d-48c1-9686-055dabd6364e","Type":"ContainerStarted","Data":"97938e7327995fd427e32b6eedc65a1d1b60d898108dcd1a936765a9ff7f6956"}
Sep 16 14:16:04.436507 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:04.436240    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984dce34-c05d-48c1-9686-055dabd6364e","Type":"ContainerStarted","Data":"8142f83fc8b3a82a7ffbd3534567c986b3a95e72885253961dc8b3777122432e"}
Sep 16 14:16:07.451676 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:07.451584    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbc3a51b-ec10-43c1-acf9-4ccd064500be","Type":"ContainerStarted","Data":"c6b5644f77f63ee30dc89f104986efa3241563e55599ca9b49d7e29d0c6dce41"}
Sep 16 14:16:07.451676 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:07.451655    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbc3a51b-ec10-43c1-acf9-4ccd064500be","Type":"ContainerStarted","Data":"af0d39e411eec1c250e9688a2e74604ae2f01d1c702dce2fab3bed4938358bfa"}
Sep 16 14:16:07.455131 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:07.455103    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984dce34-c05d-48c1-9686-055dabd6364e","Type":"ContainerStarted","Data":"4bde42beb9da685b91131946fb4b68a51f19279978ec067b283827a2031f549b"}
Sep 16 14:16:07.491988 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:07.491494    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.189442099 podStartE2EDuration="11.491471703s" totalImagesPullingTime="8.282737364s" totalInitContainerRuntime="19.29224ms" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:16:07.485267473 +0000 UTC m=+118.032009403" watchObservedRunningTime="2026-09-16 14:16:07.491471703 +0000 UTC m=+118.038213637"
Sep 16 14:16:08.361273 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:08.361246    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:16:10.471806 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:10.471766    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbc3a51b-ec10-43c1-acf9-4ccd064500be","Type":"ContainerStarted","Data":"9c7689657f4fc23d60e7ada4508c2b8d56a1c709e30f4c9cf6102292abe67868"}
Sep 16 14:16:10.472272 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:10.471862    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbc3a51b-ec10-43c1-acf9-4ccd064500be","Type":"ContainerStarted","Data":"06dc653b7c7b83d4c787bc63fa368d6fd154959c8dd6a567119a815b7c1f9cad"}
Sep 16 14:16:10.472272 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:10.471907    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbc3a51b-ec10-43c1-acf9-4ccd064500be","Type":"ContainerStarted","Data":"c542dabd3adfe74cb049ebdc40946fa02a8e52ca4cee426b4e56947846c09baf"}
Sep 16 14:16:10.472272 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:10.471937    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbc3a51b-ec10-43c1-acf9-4ccd064500be","Type":"ContainerStarted","Data":"4c791dd05bbbfacad27df3a9c792c8b8deeb42eecf923786c96993a419fea221"}
Sep 16 14:16:10.502693 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:10.502643    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.785629686 podStartE2EDuration="9.502623813s" totalImagesPullingTime="6.696597775s" totalInitContainerRuntime="20.396352ms" isStatefulPod=false podCreationTimestamp="2026-09-16 14:16:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:16:10.50231082 +0000 UTC m=+121.049052753" watchObservedRunningTime="2026-09-16 14:16:10.502623813 +0000 UTC m=+121.049365746"
Sep 16 14:16:11.673974 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:11.673935    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-897f6dc4-w774t"]
Sep 16 14:16:11.710282 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:11.710252    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:16:19.509821 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:19.509787    2553 generic.go:356] "Generic (PLEG): container finished" podID="556cb41f-d181-44fd-84e4-2a86e3b35965" containerID="a64d1f8dbf9abea6a4aa8e71a136d867d32c08915c1931519b24bdc77acc331d" exitCode=0
Sep 16 14:16:19.510253 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:19.509865    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh" event={"ID":"556cb41f-d181-44fd-84e4-2a86e3b35965","Type":"ContainerDied","Data":"a64d1f8dbf9abea6a4aa8e71a136d867d32c08915c1931519b24bdc77acc331d"}
Sep 16 14:16:19.510253 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:19.510210    2553 scope.go:117] "RemoveContainer" containerID="a64d1f8dbf9abea6a4aa8e71a136d867d32c08915c1931519b24bdc77acc331d"
Sep 16 14:16:19.738402 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:19.738372    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs\") pod \"network-metrics-daemon-5shg9\" (UID: \"4e432d05-f97c-4799-8a7c-d660680b6298\") " pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:16:19.740737 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:19.740710    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e432d05-f97c-4799-8a7c-d660680b6298-metrics-certs\") pod \"network-metrics-daemon-5shg9\" (UID: \"4e432d05-f97c-4799-8a7c-d660680b6298\") " pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:16:20.035813 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:20.035782    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-999gw\""
Sep 16 14:16:20.043250 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:20.043226    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5shg9"
Sep 16 14:16:20.329972 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:20.329949    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5shg9"]
Sep 16 14:16:20.332379 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:16:20.332351    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e432d05_f97c_4799_8a7c_d660680b6298.slice/crio-27e5309f017648f5816345b5a1d49c8befb2d5744c101a3b48f462ea06b15fdb WatchSource:0}: Error finding container 27e5309f017648f5816345b5a1d49c8befb2d5744c101a3b48f462ea06b15fdb: Status 404 returned error can't find the container with id 27e5309f017648f5816345b5a1d49c8befb2d5744c101a3b48f462ea06b15fdb
Sep 16 14:16:20.515610 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:20.515571    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5shg9" event={"ID":"4e432d05-f97c-4799-8a7c-d660680b6298","Type":"ContainerStarted","Data":"27e5309f017648f5816345b5a1d49c8befb2d5744c101a3b48f462ea06b15fdb"}
Sep 16 14:16:20.517286 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:20.517262    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xbfkh" event={"ID":"556cb41f-d181-44fd-84e4-2a86e3b35965","Type":"ContainerStarted","Data":"704004ab863fe239eab93d8f48034794e40d27ca2a9e73a6f4a8af625ea43f29"}
Sep 16 14:16:22.527557 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:22.527510    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5shg9" event={"ID":"4e432d05-f97c-4799-8a7c-d660680b6298","Type":"ContainerStarted","Data":"fef2b0921ed0e3333c44f0cf9ce57a8fca7e94b75003e49b802564b16179fb60"}
Sep 16 14:16:22.527973 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:22.527628    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5shg9" event={"ID":"4e432d05-f97c-4799-8a7c-d660680b6298","Type":"ContainerStarted","Data":"436e608b69a44992e6d2f3cf74f1eb343145d7b8d80722fa3e4a71ed25ad5b16"}
Sep 16 14:16:22.552138 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:22.552088    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5shg9" podStartSLOduration=130.824676244 podStartE2EDuration="2m12.55207083s" totalImagesPullingTime="1.727394586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:16:22.550036442 +0000 UTC m=+133.096778373" watchObservedRunningTime="2026-09-16 14:16:22.55207083 +0000 UTC m=+133.098812757"
Sep 16 14:16:23.380778 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.380724    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-598474997c-5r45z" podUID="b095cf19-aa32-4e5c-af3c-7bd969dd0895" containerName="registry" containerID="cri-o://55f90726c0dd43760fe93e636e6411463d160298ebfa4f5220df57d6df5c734f" gracePeriod=30
Sep 16 14:16:23.534473 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.534441    2553 generic.go:356] "Generic (PLEG): container finished" podID="b095cf19-aa32-4e5c-af3c-7bd969dd0895" containerID="55f90726c0dd43760fe93e636e6411463d160298ebfa4f5220df57d6df5c734f" exitCode=0
Sep 16 14:16:23.534911 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.534511    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-598474997c-5r45z" event={"ID":"b095cf19-aa32-4e5c-af3c-7bd969dd0895","Type":"ContainerDied","Data":"55f90726c0dd43760fe93e636e6411463d160298ebfa4f5220df57d6df5c734f"}
Sep 16 14:16:23.659738 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.659710    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:16:23.772299 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.772271    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-certificates\") pod \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") "
Sep 16 14:16:23.772299 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.772304    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-bound-sa-token\") pod \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") "
Sep 16 14:16:23.772535 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.772359    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b095cf19-aa32-4e5c-af3c-7bd969dd0895-trusted-ca\") pod \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") "
Sep 16 14:16:23.772535 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.772398    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b095cf19-aa32-4e5c-af3c-7bd969dd0895-image-registry-private-configuration\") pod \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") "
Sep 16 14:16:23.772535 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.772437    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b095cf19-aa32-4e5c-af3c-7bd969dd0895-installation-pull-secrets\") pod \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") "
Sep 16 14:16:23.772535 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.772465    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls\") pod \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") "
Sep 16 14:16:23.772535 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.772498    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b095cf19-aa32-4e5c-af3c-7bd969dd0895-ca-trust-extracted\") pod \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") "
Sep 16 14:16:23.772790 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.772540    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm5w8\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-kube-api-access-rm5w8\") pod \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\" (UID: \"b095cf19-aa32-4e5c-af3c-7bd969dd0895\") "
Sep 16 14:16:23.772790 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.772768    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b095cf19-aa32-4e5c-af3c-7bd969dd0895" (UID: "b095cf19-aa32-4e5c-af3c-7bd969dd0895"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:16:23.772945 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.772870    2553 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-certificates\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:16:23.772997 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.772880    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b095cf19-aa32-4e5c-af3c-7bd969dd0895-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b095cf19-aa32-4e5c-af3c-7bd969dd0895" (UID: "b095cf19-aa32-4e5c-af3c-7bd969dd0895"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:16:23.774972 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.774936    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b095cf19-aa32-4e5c-af3c-7bd969dd0895" (UID: "b095cf19-aa32-4e5c-af3c-7bd969dd0895"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:16:23.775103 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.775010    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b095cf19-aa32-4e5c-af3c-7bd969dd0895-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b095cf19-aa32-4e5c-af3c-7bd969dd0895" (UID: "b095cf19-aa32-4e5c-af3c-7bd969dd0895"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:16:23.775167 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.775120    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b095cf19-aa32-4e5c-af3c-7bd969dd0895" (UID: "b095cf19-aa32-4e5c-af3c-7bd969dd0895"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:16:23.775167 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.775150    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b095cf19-aa32-4e5c-af3c-7bd969dd0895-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b095cf19-aa32-4e5c-af3c-7bd969dd0895" (UID: "b095cf19-aa32-4e5c-af3c-7bd969dd0895"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:16:23.775326 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.775268    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-kube-api-access-rm5w8" (OuterVolumeSpecName: "kube-api-access-rm5w8") pod "b095cf19-aa32-4e5c-af3c-7bd969dd0895" (UID: "b095cf19-aa32-4e5c-af3c-7bd969dd0895"). InnerVolumeSpecName "kube-api-access-rm5w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:16:23.781362 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.781335    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b095cf19-aa32-4e5c-af3c-7bd969dd0895-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b095cf19-aa32-4e5c-af3c-7bd969dd0895" (UID: "b095cf19-aa32-4e5c-af3c-7bd969dd0895"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:16:23.873558 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.873523    2553 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b095cf19-aa32-4e5c-af3c-7bd969dd0895-installation-pull-secrets\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:16:23.873558 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.873552    2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-registry-tls\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:16:23.873558 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.873562    2553 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b095cf19-aa32-4e5c-af3c-7bd969dd0895-ca-trust-extracted\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:16:23.873757 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.873570    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm5w8\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-kube-api-access-rm5w8\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:16:23.873757 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.873579    2553 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b095cf19-aa32-4e5c-af3c-7bd969dd0895-bound-sa-token\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:16:23.873757 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.873586    2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b095cf19-aa32-4e5c-af3c-7bd969dd0895-trusted-ca\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:16:23.873757 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:23.873594    2553 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b095cf19-aa32-4e5c-af3c-7bd969dd0895-image-registry-private-configuration\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:16:24.539749 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:24.539718    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-598474997c-5r45z"
Sep 16 14:16:24.539749 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:24.539735    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-598474997c-5r45z" event={"ID":"b095cf19-aa32-4e5c-af3c-7bd969dd0895","Type":"ContainerDied","Data":"9741551f07f0edff73ea6dbaf3f4e3d3cf2f11dc09ab44cc1123c69edf269190"}
Sep 16 14:16:24.540201 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:24.539789    2553 scope.go:117] "RemoveContainer" containerID="55f90726c0dd43760fe93e636e6411463d160298ebfa4f5220df57d6df5c734f"
Sep 16 14:16:24.565500 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:24.565474    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-598474997c-5r45z"]
Sep 16 14:16:24.569833 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:24.569812    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-598474997c-5r45z"]
Sep 16 14:16:25.916304 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:25.916264    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b095cf19-aa32-4e5c-af3c-7bd969dd0895" path="/var/lib/kubelet/pods/b095cf19-aa32-4e5c-af3c-7bd969dd0895/volumes"
Sep 16 14:16:36.694024 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:36.693961    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-897f6dc4-w774t" podUID="63f865cf-7b11-4a79-bfa7-36a02e00a180" containerName="console" containerID="cri-o://e820b5bcdc31992325e5490dcac73c5c3f0c984c0651aaf12c40920708b995ed" gracePeriod=15
Sep 16 14:16:36.993315 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:36.993289    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:16:37.088420 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.088389    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2mt6\" (UniqueName: \"kubernetes.io/projected/63f865cf-7b11-4a79-bfa7-36a02e00a180-kube-api-access-f2mt6\") pod \"63f865cf-7b11-4a79-bfa7-36a02e00a180\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") "
Sep 16 14:16:37.088603 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.088486    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-config\") pod \"63f865cf-7b11-4a79-bfa7-36a02e00a180\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") "
Sep 16 14:16:37.088603 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.088514    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-oauth-serving-cert\") pod \"63f865cf-7b11-4a79-bfa7-36a02e00a180\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") "
Sep 16 14:16:37.088603 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.088553    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-oauth-config\") pod \"63f865cf-7b11-4a79-bfa7-36a02e00a180\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") "
Sep 16 14:16:37.088603 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.088576    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-serving-cert\") pod \"63f865cf-7b11-4a79-bfa7-36a02e00a180\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") "
Sep 16 14:16:37.088603 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.088600    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-service-ca\") pod \"63f865cf-7b11-4a79-bfa7-36a02e00a180\" (UID: \"63f865cf-7b11-4a79-bfa7-36a02e00a180\") "
Sep 16 14:16:37.089182 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.088880    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-config" (OuterVolumeSpecName: "console-config") pod "63f865cf-7b11-4a79-bfa7-36a02e00a180" (UID: "63f865cf-7b11-4a79-bfa7-36a02e00a180"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:16:37.089182 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.088903    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "63f865cf-7b11-4a79-bfa7-36a02e00a180" (UID: "63f865cf-7b11-4a79-bfa7-36a02e00a180"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:16:37.089182 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.089176    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-service-ca" (OuterVolumeSpecName: "service-ca") pod "63f865cf-7b11-4a79-bfa7-36a02e00a180" (UID: "63f865cf-7b11-4a79-bfa7-36a02e00a180"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:16:37.090731 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.090703    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f865cf-7b11-4a79-bfa7-36a02e00a180-kube-api-access-f2mt6" (OuterVolumeSpecName: "kube-api-access-f2mt6") pod "63f865cf-7b11-4a79-bfa7-36a02e00a180" (UID: "63f865cf-7b11-4a79-bfa7-36a02e00a180"). InnerVolumeSpecName "kube-api-access-f2mt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:16:37.090858 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.090829    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "63f865cf-7b11-4a79-bfa7-36a02e00a180" (UID: "63f865cf-7b11-4a79-bfa7-36a02e00a180"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:16:37.090940 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.090921    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "63f865cf-7b11-4a79-bfa7-36a02e00a180" (UID: "63f865cf-7b11-4a79-bfa7-36a02e00a180"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:16:37.189913 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.189863    2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-config\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:16:37.190079 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.189920    2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-oauth-serving-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:16:37.190079 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.189937    2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-oauth-config\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:16:37.190079 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.189951    2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63f865cf-7b11-4a79-bfa7-36a02e00a180-console-serving-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:16:37.190079 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.189964    2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63f865cf-7b11-4a79-bfa7-36a02e00a180-service-ca\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:16:37.190079 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.189977    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2mt6\" (UniqueName: \"kubernetes.io/projected/63f865cf-7b11-4a79-bfa7-36a02e00a180-kube-api-access-f2mt6\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:16:37.596835 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.596799    2553 generic.go:356] "Generic (PLEG): container finished" podID="63f865cf-7b11-4a79-bfa7-36a02e00a180" containerID="e820b5bcdc31992325e5490dcac73c5c3f0c984c0651aaf12c40920708b995ed" exitCode=0
Sep 16 14:16:37.597031 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.596851    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-897f6dc4-w774t"
Sep 16 14:16:37.597031 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.596883    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-897f6dc4-w774t" event={"ID":"63f865cf-7b11-4a79-bfa7-36a02e00a180","Type":"ContainerDied","Data":"e820b5bcdc31992325e5490dcac73c5c3f0c984c0651aaf12c40920708b995ed"}
Sep 16 14:16:37.597031 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.596952    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-897f6dc4-w774t" event={"ID":"63f865cf-7b11-4a79-bfa7-36a02e00a180","Type":"ContainerDied","Data":"901d696745b471c139e0fc020515883aee00b7912b7d022aa1b7c6d04e2d6896"}
Sep 16 14:16:37.597031 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.596974    2553 scope.go:117] "RemoveContainer" containerID="e820b5bcdc31992325e5490dcac73c5c3f0c984c0651aaf12c40920708b995ed"
Sep 16 14:16:37.605157 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.605126    2553 scope.go:117] "RemoveContainer" containerID="e820b5bcdc31992325e5490dcac73c5c3f0c984c0651aaf12c40920708b995ed"
Sep 16 14:16:37.605441 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:16:37.605421    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e820b5bcdc31992325e5490dcac73c5c3f0c984c0651aaf12c40920708b995ed\": container with ID starting with e820b5bcdc31992325e5490dcac73c5c3f0c984c0651aaf12c40920708b995ed not found: ID does not exist" containerID="e820b5bcdc31992325e5490dcac73c5c3f0c984c0651aaf12c40920708b995ed"
Sep 16 14:16:37.605510 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.605453    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e820b5bcdc31992325e5490dcac73c5c3f0c984c0651aaf12c40920708b995ed"} err="failed to get container status \"e820b5bcdc31992325e5490dcac73c5c3f0c984c0651aaf12c40920708b995ed\": rpc error: code = NotFound desc = could not find container \"e820b5bcdc31992325e5490dcac73c5c3f0c984c0651aaf12c40920708b995ed\": container with ID starting with e820b5bcdc31992325e5490dcac73c5c3f0c984c0651aaf12c40920708b995ed not found: ID does not exist"
Sep 16 14:16:37.619034 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.619011    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-897f6dc4-w774t"]
Sep 16 14:16:37.622530 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.622510    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-897f6dc4-w774t"]
Sep 16 14:16:37.918107 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:16:37.918035    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63f865cf-7b11-4a79-bfa7-36a02e00a180" path="/var/lib/kubelet/pods/63f865cf-7b11-4a79-bfa7-36a02e00a180/volumes"
Sep 16 14:17:01.710882 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:01.710849    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:17:01.730037 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:01.730011    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:17:02.708278 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:02.708208    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:17:19.450567 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.450529    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-76c585499c-v4r57"]
Sep 16 14:17:19.451026 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.450998    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b095cf19-aa32-4e5c-af3c-7bd969dd0895" containerName="registry"
Sep 16 14:17:19.451026 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.451014    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b095cf19-aa32-4e5c-af3c-7bd969dd0895" containerName="registry"
Sep 16 14:17:19.451099 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.451032    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63f865cf-7b11-4a79-bfa7-36a02e00a180" containerName="console"
Sep 16 14:17:19.451099 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.451038    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="63f865cf-7b11-4a79-bfa7-36a02e00a180" containerName="console"
Sep 16 14:17:19.451099 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.451089    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="63f865cf-7b11-4a79-bfa7-36a02e00a180" containerName="console"
Sep 16 14:17:19.451215 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.451100    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b095cf19-aa32-4e5c-af3c-7bd969dd0895" containerName="registry"
Sep 16 14:17:19.454118 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.454093    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.456967 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.456942    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tj6fg\""
Sep 16 14:17:19.457114 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.457095    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 14:17:19.457204 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.457121    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 14:17:19.457204 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.457112    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 14:17:19.457471 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.457457    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 14:17:19.458788 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.458768    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 14:17:19.462725 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.462704    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 14:17:19.465861 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.465840    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76c585499c-v4r57"]
Sep 16 14:17:19.525805 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.525771    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c4fc912-3823-4df0-81de-668f4708b204-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.525805 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.525804    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3c4fc912-3823-4df0-81de-668f4708b204-secret-telemeter-client\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.526072 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.525823    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3c4fc912-3823-4df0-81de-668f4708b204-telemeter-client-tls\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.526072 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.525964    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3c4fc912-3823-4df0-81de-668f4708b204-federate-client-tls\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.526072 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.526009    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c4fc912-3823-4df0-81de-668f4708b204-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.526194 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.526070    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f727\" (UniqueName: \"kubernetes.io/projected/3c4fc912-3823-4df0-81de-668f4708b204-kube-api-access-5f727\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.526194 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.526103    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c4fc912-3823-4df0-81de-668f4708b204-serving-certs-ca-bundle\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.526194 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.526125    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c4fc912-3823-4df0-81de-668f4708b204-metrics-client-ca\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.627251 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.627210    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3c4fc912-3823-4df0-81de-668f4708b204-federate-client-tls\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.627251 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.627251    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c4fc912-3823-4df0-81de-668f4708b204-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.627511 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.627274    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f727\" (UniqueName: \"kubernetes.io/projected/3c4fc912-3823-4df0-81de-668f4708b204-kube-api-access-5f727\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.627511 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.627296    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c4fc912-3823-4df0-81de-668f4708b204-serving-certs-ca-bundle\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.627511 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.627319    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c4fc912-3823-4df0-81de-668f4708b204-metrics-client-ca\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.627511 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.627365    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c4fc912-3823-4df0-81de-668f4708b204-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.627511 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.627389    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3c4fc912-3823-4df0-81de-668f4708b204-secret-telemeter-client\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.627511 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.627413    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3c4fc912-3823-4df0-81de-668f4708b204-telemeter-client-tls\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.628092 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.628068    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c4fc912-3823-4df0-81de-668f4708b204-serving-certs-ca-bundle\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.628221 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.628159    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c4fc912-3823-4df0-81de-668f4708b204-metrics-client-ca\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.628547 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.628521    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c4fc912-3823-4df0-81de-668f4708b204-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.629880 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.629849    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c4fc912-3823-4df0-81de-668f4708b204-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.630076 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.630054    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3c4fc912-3823-4df0-81de-668f4708b204-secret-telemeter-client\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.630229 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.630213    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3c4fc912-3823-4df0-81de-668f4708b204-telemeter-client-tls\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.630266 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.630254    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3c4fc912-3823-4df0-81de-668f4708b204-federate-client-tls\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.636136 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.636112    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f727\" (UniqueName: \"kubernetes.io/projected/3c4fc912-3823-4df0-81de-668f4708b204-kube-api-access-5f727\") pod \"telemeter-client-76c585499c-v4r57\" (UID: \"3c4fc912-3823-4df0-81de-668f4708b204\") " pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:19.764908 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:19.764869    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76c585499c-v4r57"
Sep 16 14:17:20.041627 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:20.041602    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76c585499c-v4r57"]
Sep 16 14:17:20.044453 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:17:20.044429    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4fc912_3823_4df0_81de_668f4708b204.slice/crio-29eedb14e7c7da740b567724a715fceac391ffc4b785ab7545f7ba2bfaa23495 WatchSource:0}: Error finding container 29eedb14e7c7da740b567724a715fceac391ffc4b785ab7545f7ba2bfaa23495: Status 404 returned error can't find the container with id 29eedb14e7c7da740b567724a715fceac391ffc4b785ab7545f7ba2bfaa23495
Sep 16 14:17:20.765410 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:20.765373    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76c585499c-v4r57" event={"ID":"3c4fc912-3823-4df0-81de-668f4708b204","Type":"ContainerStarted","Data":"29eedb14e7c7da740b567724a715fceac391ffc4b785ab7545f7ba2bfaa23495"}
Sep 16 14:17:22.777275 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:22.777240    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76c585499c-v4r57" event={"ID":"3c4fc912-3823-4df0-81de-668f4708b204","Type":"ContainerStarted","Data":"73d3434f42153a9d1f144fc96701aa712b6eda4c900da72c5a1d98c58cf2ab48"}
Sep 16 14:17:22.777641 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:22.777298    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76c585499c-v4r57" event={"ID":"3c4fc912-3823-4df0-81de-668f4708b204","Type":"ContainerStarted","Data":"dd001f4a05bd61bb8efeb637369a1dc35ca71828594959a170ee5a7af14132d2"}
Sep 16 14:17:22.777641 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:22.777319    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76c585499c-v4r57" event={"ID":"3c4fc912-3823-4df0-81de-668f4708b204","Type":"ContainerStarted","Data":"6c764c8e339ce7ec62a7ef3d4901a3a1ad8ae529dca520b190c4f7c0c91415fa"}
Sep 16 14:17:22.804042 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:17:22.803990    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-76c585499c-v4r57" podStartSLOduration=1.9225813779999998 podStartE2EDuration="3.803975144s" totalImagesPullingTime="1.881393766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:17:22.801119802 +0000 UTC m=+193.347861732" watchObservedRunningTime="2026-09-16 14:17:22.803975144 +0000 UTC m=+193.350717074"
Sep 16 14:19:09.784736 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:09.784706    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:19:09.789922 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:09.789885    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:19:09.796455 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:09.796436    2553 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 14:19:14.610473 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:14.610422    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2wq7l"]
Sep 16 14:19:14.613733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:14.613710    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2wq7l"
Sep 16 14:19:14.617755 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:14.617547    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 14:19:14.617755 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:14.617597    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 14:19:14.619203 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:14.619178    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-k6rbb\""
Sep 16 14:19:14.623026 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:14.622997    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2wq7l"]
Sep 16 14:19:14.674636 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:14.674460    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66c9f5c5-c461-46e1-86ac-b6429afe83ff-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2wq7l\" (UID: \"66c9f5c5-c461-46e1-86ac-b6429afe83ff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2wq7l"
Sep 16 14:19:14.674636 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:14.674544    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhzp8\" (UniqueName: \"kubernetes.io/projected/66c9f5c5-c461-46e1-86ac-b6429afe83ff-kube-api-access-lhzp8\") pod \"openshift-lws-operator-fd8ccff4c-2wq7l\" (UID: \"66c9f5c5-c461-46e1-86ac-b6429afe83ff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2wq7l"
Sep 16 14:19:14.775359 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:14.775328    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66c9f5c5-c461-46e1-86ac-b6429afe83ff-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2wq7l\" (UID: \"66c9f5c5-c461-46e1-86ac-b6429afe83ff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2wq7l"
Sep 16 14:19:14.775501 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:14.775389    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhzp8\" (UniqueName: \"kubernetes.io/projected/66c9f5c5-c461-46e1-86ac-b6429afe83ff-kube-api-access-lhzp8\") pod \"openshift-lws-operator-fd8ccff4c-2wq7l\" (UID: \"66c9f5c5-c461-46e1-86ac-b6429afe83ff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2wq7l"
Sep 16 14:19:14.775704 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:14.775686    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66c9f5c5-c461-46e1-86ac-b6429afe83ff-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2wq7l\" (UID: \"66c9f5c5-c461-46e1-86ac-b6429afe83ff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2wq7l"
Sep 16 14:19:14.783357 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:14.783329    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhzp8\" (UniqueName: \"kubernetes.io/projected/66c9f5c5-c461-46e1-86ac-b6429afe83ff-kube-api-access-lhzp8\") pod \"openshift-lws-operator-fd8ccff4c-2wq7l\" (UID: \"66c9f5c5-c461-46e1-86ac-b6429afe83ff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2wq7l"
Sep 16 14:19:14.933701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:14.933621    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2wq7l"
Sep 16 14:19:15.210299 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:15.210279    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2wq7l"]
Sep 16 14:19:15.212858 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:19:15.212833    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66c9f5c5_c461_46e1_86ac_b6429afe83ff.slice/crio-51c26ce13d669bb60d1adbe5c54bab310ff780ce04dce89bf7c7d7229faeff87 WatchSource:0}: Error finding container 51c26ce13d669bb60d1adbe5c54bab310ff780ce04dce89bf7c7d7229faeff87: Status 404 returned error can't find the container with id 51c26ce13d669bb60d1adbe5c54bab310ff780ce04dce89bf7c7d7229faeff87
Sep 16 14:19:15.214513 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:15.214490    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:19:16.219512 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:16.219452    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2wq7l" event={"ID":"66c9f5c5-c461-46e1-86ac-b6429afe83ff","Type":"ContainerStarted","Data":"51c26ce13d669bb60d1adbe5c54bab310ff780ce04dce89bf7c7d7229faeff87"}
Sep 16 14:19:18.230155 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:18.230124    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2wq7l" event={"ID":"66c9f5c5-c461-46e1-86ac-b6429afe83ff","Type":"ContainerStarted","Data":"5670298c06ccb9ba5b13998f978a0938c5e9ebad7068b405cc42b213f514553e"}
Sep 16 14:19:18.247967 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:18.247919    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2wq7l" podStartSLOduration=1.81029573 podStartE2EDuration="4.247905983s" totalImagesPullingTime="2.437610253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:19:18.246907756 +0000 UTC m=+308.793649684" watchObservedRunningTime="2026-09-16 14:19:18.247905983 +0000 UTC m=+308.794647906"
Sep 16 14:19:38.037884 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.037847    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"]
Sep 16 14:19:38.041783 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.041756    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.045496 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.045475    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 14:19:38.045629 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.045478    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 14:19:38.045629 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.045527    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k7vhv\""
Sep 16 14:19:38.045812 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.045797    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 14:19:38.046002 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.045981    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 14:19:38.063907 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.063871    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"]
Sep 16 14:19:38.071054 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.071030    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7c930415-76ce-4fc9-b21e-c3e7dee47a44-manifests\") pod \"opendatahub-operator-controller-manager-68b85944db-lvhcd\" (UID: \"7c930415-76ce-4fc9-b21e-c3e7dee47a44\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.071153 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.071087    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7c930415-76ce-4fc9-b21e-c3e7dee47a44-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b85944db-lvhcd\" (UID: \"7c930415-76ce-4fc9-b21e-c3e7dee47a44\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.071204 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.071149    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c930415-76ce-4fc9-b21e-c3e7dee47a44-tmp\") pod \"opendatahub-operator-controller-manager-68b85944db-lvhcd\" (UID: \"7c930415-76ce-4fc9-b21e-c3e7dee47a44\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.071239 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.071207    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c930415-76ce-4fc9-b21e-c3e7dee47a44-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b85944db-lvhcd\" (UID: \"7c930415-76ce-4fc9-b21e-c3e7dee47a44\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.071273 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.071239    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcfpj\" (UniqueName: \"kubernetes.io/projected/7c930415-76ce-4fc9-b21e-c3e7dee47a44-kube-api-access-rcfpj\") pod \"opendatahub-operator-controller-manager-68b85944db-lvhcd\" (UID: \"7c930415-76ce-4fc9-b21e-c3e7dee47a44\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.172073 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.172043    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c930415-76ce-4fc9-b21e-c3e7dee47a44-tmp\") pod \"opendatahub-operator-controller-manager-68b85944db-lvhcd\" (UID: \"7c930415-76ce-4fc9-b21e-c3e7dee47a44\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.172209 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.172078    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c930415-76ce-4fc9-b21e-c3e7dee47a44-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b85944db-lvhcd\" (UID: \"7c930415-76ce-4fc9-b21e-c3e7dee47a44\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.172209 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.172110    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcfpj\" (UniqueName: \"kubernetes.io/projected/7c930415-76ce-4fc9-b21e-c3e7dee47a44-kube-api-access-rcfpj\") pod \"opendatahub-operator-controller-manager-68b85944db-lvhcd\" (UID: \"7c930415-76ce-4fc9-b21e-c3e7dee47a44\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.172209 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.172164    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7c930415-76ce-4fc9-b21e-c3e7dee47a44-manifests\") pod \"opendatahub-operator-controller-manager-68b85944db-lvhcd\" (UID: \"7c930415-76ce-4fc9-b21e-c3e7dee47a44\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.172308 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.172214    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7c930415-76ce-4fc9-b21e-c3e7dee47a44-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b85944db-lvhcd\" (UID: \"7c930415-76ce-4fc9-b21e-c3e7dee47a44\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.172587 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.172557    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7c930415-76ce-4fc9-b21e-c3e7dee47a44-manifests\") pod \"opendatahub-operator-controller-manager-68b85944db-lvhcd\" (UID: \"7c930415-76ce-4fc9-b21e-c3e7dee47a44\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.174328 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.174308    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c930415-76ce-4fc9-b21e-c3e7dee47a44-tmp\") pod \"opendatahub-operator-controller-manager-68b85944db-lvhcd\" (UID: \"7c930415-76ce-4fc9-b21e-c3e7dee47a44\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.174558 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.174542    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7c930415-76ce-4fc9-b21e-c3e7dee47a44-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b85944db-lvhcd\" (UID: \"7c930415-76ce-4fc9-b21e-c3e7dee47a44\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.174658 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.174640    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c930415-76ce-4fc9-b21e-c3e7dee47a44-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b85944db-lvhcd\" (UID: \"7c930415-76ce-4fc9-b21e-c3e7dee47a44\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.185589 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.185571    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcfpj\" (UniqueName: \"kubernetes.io/projected/7c930415-76ce-4fc9-b21e-c3e7dee47a44-kube-api-access-rcfpj\") pod \"opendatahub-operator-controller-manager-68b85944db-lvhcd\" (UID: \"7c930415-76ce-4fc9-b21e-c3e7dee47a44\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.352632 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.352558    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:38.633157 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:38.633078    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"]
Sep 16 14:19:38.636727 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:19:38.636700    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c930415_76ce_4fc9_b21e_c3e7dee47a44.slice/crio-af8190db0a296131922bb42f7113149cc786eb0bd513d4fb907b4ad35a793faf WatchSource:0}: Error finding container af8190db0a296131922bb42f7113149cc786eb0bd513d4fb907b4ad35a793faf: Status 404 returned error can't find the container with id af8190db0a296131922bb42f7113149cc786eb0bd513d4fb907b4ad35a793faf
Sep 16 14:19:39.320292 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:39.320256    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd" event={"ID":"7c930415-76ce-4fc9-b21e-c3e7dee47a44","Type":"ContainerStarted","Data":"af8190db0a296131922bb42f7113149cc786eb0bd513d4fb907b4ad35a793faf"}
Sep 16 14:19:42.337575 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:42.337489    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd" event={"ID":"7c930415-76ce-4fc9-b21e-c3e7dee47a44","Type":"ContainerStarted","Data":"1f62de8a25d885ef2501a6750b0ff5650962c9f901d340880fc286394b1041f1"}
Sep 16 14:19:43.342870 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:43.342837    2553 generic.go:356] "Generic (PLEG): container finished" podID="7c930415-76ce-4fc9-b21e-c3e7dee47a44" containerID="1f62de8a25d885ef2501a6750b0ff5650962c9f901d340880fc286394b1041f1" exitCode=0
Sep 16 14:19:43.343270 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:43.342922    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd" event={"ID":"7c930415-76ce-4fc9-b21e-c3e7dee47a44","Type":"ContainerDied","Data":"1f62de8a25d885ef2501a6750b0ff5650962c9f901d340880fc286394b1041f1"}
Sep 16 14:19:45.353939 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:45.353878    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd" event={"ID":"7c930415-76ce-4fc9-b21e-c3e7dee47a44","Type":"ContainerStarted","Data":"071db8df488b292c8a6d9e46aa66c5de0b8e739a1a85ec34fd9dc04de634371e"}
Sep 16 14:19:45.354296 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:45.353980    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:19:45.392620 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:45.392574    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd" podStartSLOduration=1.7214659509999999 podStartE2EDuration="7.392560351s" totalImagesPullingTime="4.615840845s" totalInitContainerRuntime="1.055253555s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:19:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:19:45.390815511 +0000 UTC m=+335.937557443" watchObservedRunningTime="2026-09-16 14:19:45.392560351 +0000 UTC m=+335.939302281"
Sep 16 14:19:56.360292 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:19:56.360256    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-lvhcd"
Sep 16 14:20:08.326280 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.326206    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"]
Sep 16 14:20:08.329150 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.329130    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.332204 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.332181    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 14:20:08.332449 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.332430    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-42c95\""
Sep 16 14:20:08.332528 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.332451    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 14:20:08.332714 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.332694    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 14:20:08.345109 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.345089    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"]
Sep 16 14:20:08.435047 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.435012    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ddtg\" (UniqueName: \"kubernetes.io/projected/63adf8da-3abd-458d-aae0-1a70b362c4f4-kube-api-access-4ddtg\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.435189 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.435055    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63adf8da-3abd-458d-aae0-1a70b362c4f4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.435189 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.435089    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63adf8da-3abd-458d-aae0-1a70b362c4f4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.435189 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.435119    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63adf8da-3abd-458d-aae0-1a70b362c4f4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.435189 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.435135    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63adf8da-3abd-458d-aae0-1a70b362c4f4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.435189 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.435152    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63adf8da-3abd-458d-aae0-1a70b362c4f4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.435189 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.435173    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63adf8da-3abd-458d-aae0-1a70b362c4f4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.435384 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.435238    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63adf8da-3abd-458d-aae0-1a70b362c4f4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.435384 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.435276    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63adf8da-3abd-458d-aae0-1a70b362c4f4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.536511 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.536474    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ddtg\" (UniqueName: \"kubernetes.io/projected/63adf8da-3abd-458d-aae0-1a70b362c4f4-kube-api-access-4ddtg\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.536668 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.536524    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63adf8da-3abd-458d-aae0-1a70b362c4f4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.536668 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.536574    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63adf8da-3abd-458d-aae0-1a70b362c4f4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.536668 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.536607    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63adf8da-3abd-458d-aae0-1a70b362c4f4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.536668 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.536630    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63adf8da-3abd-458d-aae0-1a70b362c4f4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.536668 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.536655    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63adf8da-3abd-458d-aae0-1a70b362c4f4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.536936 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.536683    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63adf8da-3abd-458d-aae0-1a70b362c4f4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.536936 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.536720    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63adf8da-3abd-458d-aae0-1a70b362c4f4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.536936 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.536744    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63adf8da-3abd-458d-aae0-1a70b362c4f4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.537060 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.536986    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63adf8da-3abd-458d-aae0-1a70b362c4f4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.537113 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.537050    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63adf8da-3abd-458d-aae0-1a70b362c4f4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.537173 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.537139    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63adf8da-3abd-458d-aae0-1a70b362c4f4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.537173 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.537159    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63adf8da-3abd-458d-aae0-1a70b362c4f4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.537559 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.537529    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63adf8da-3abd-458d-aae0-1a70b362c4f4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.538795 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.538778    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63adf8da-3abd-458d-aae0-1a70b362c4f4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.539207 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.539188    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63adf8da-3abd-458d-aae0-1a70b362c4f4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.545473 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.545444    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63adf8da-3abd-458d-aae0-1a70b362c4f4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.545592 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.545569    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ddtg\" (UniqueName: \"kubernetes.io/projected/63adf8da-3abd-458d-aae0-1a70b362c4f4-kube-api-access-4ddtg\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5\" (UID: \"63adf8da-3abd-458d-aae0-1a70b362c4f4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.640324 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.640248    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:08.920947 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:08.920917    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"]
Sep 16 14:20:08.923691 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:20:08.923664    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63adf8da_3abd_458d_aae0_1a70b362c4f4.slice/crio-d7f70efe46e464a060b937f77b095e9341798d1994441ecdeb0cbfb7ae76b252 WatchSource:0}: Error finding container d7f70efe46e464a060b937f77b095e9341798d1994441ecdeb0cbfb7ae76b252: Status 404 returned error can't find the container with id d7f70efe46e464a060b937f77b095e9341798d1994441ecdeb0cbfb7ae76b252
Sep 16 14:20:09.459815 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:09.459780    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5" event={"ID":"63adf8da-3abd-458d-aae0-1a70b362c4f4","Type":"ContainerStarted","Data":"d7f70efe46e464a060b937f77b095e9341798d1994441ecdeb0cbfb7ae76b252"}
Sep 16 14:20:18.420464 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.420427    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"]
Sep 16 14:20:18.423348 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.423327    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"
Sep 16 14:20:18.427098 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.427078    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 14:20:18.428727 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.428702    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 14:20:18.428843 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.428703    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7mzc6\""
Sep 16 14:20:18.428938 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.428712    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 14:20:18.436593 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.436535    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"]
Sep 16 14:20:18.446625 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.446585    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:20:18.446715 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.446667    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:20:18.446715 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.446694    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:20:18.525250 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.525225    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a683d72c-2060-4e6e-b3cb-050dd34e530a-manager-config\") pod \"lws-controller-manager-5cbd5cd6f8-4kxhb\" (UID: \"a683d72c-2060-4e6e-b3cb-050dd34e530a\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"
Sep 16 14:20:18.525355 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.525277    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a683d72c-2060-4e6e-b3cb-050dd34e530a-metrics-cert\") pod \"lws-controller-manager-5cbd5cd6f8-4kxhb\" (UID: \"a683d72c-2060-4e6e-b3cb-050dd34e530a\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"
Sep 16 14:20:18.525355 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.525323    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a683d72c-2060-4e6e-b3cb-050dd34e530a-cert\") pod \"lws-controller-manager-5cbd5cd6f8-4kxhb\" (UID: \"a683d72c-2060-4e6e-b3cb-050dd34e530a\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"
Sep 16 14:20:18.525355 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.525343    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lmtl\" (UniqueName: \"kubernetes.io/projected/a683d72c-2060-4e6e-b3cb-050dd34e530a-kube-api-access-9lmtl\") pod \"lws-controller-manager-5cbd5cd6f8-4kxhb\" (UID: \"a683d72c-2060-4e6e-b3cb-050dd34e530a\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"
Sep 16 14:20:18.626529 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.626493    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a683d72c-2060-4e6e-b3cb-050dd34e530a-manager-config\") pod \"lws-controller-manager-5cbd5cd6f8-4kxhb\" (UID: \"a683d72c-2060-4e6e-b3cb-050dd34e530a\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"
Sep 16 14:20:18.626702 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.626660    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a683d72c-2060-4e6e-b3cb-050dd34e530a-metrics-cert\") pod \"lws-controller-manager-5cbd5cd6f8-4kxhb\" (UID: \"a683d72c-2060-4e6e-b3cb-050dd34e530a\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"
Sep 16 14:20:18.626769 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.626705    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a683d72c-2060-4e6e-b3cb-050dd34e530a-cert\") pod \"lws-controller-manager-5cbd5cd6f8-4kxhb\" (UID: \"a683d72c-2060-4e6e-b3cb-050dd34e530a\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"
Sep 16 14:20:18.626769 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.626737    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lmtl\" (UniqueName: \"kubernetes.io/projected/a683d72c-2060-4e6e-b3cb-050dd34e530a-kube-api-access-9lmtl\") pod \"lws-controller-manager-5cbd5cd6f8-4kxhb\" (UID: \"a683d72c-2060-4e6e-b3cb-050dd34e530a\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"
Sep 16 14:20:18.627320 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.627290    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a683d72c-2060-4e6e-b3cb-050dd34e530a-manager-config\") pod \"lws-controller-manager-5cbd5cd6f8-4kxhb\" (UID: \"a683d72c-2060-4e6e-b3cb-050dd34e530a\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"
Sep 16 14:20:18.629204 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.629176    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a683d72c-2060-4e6e-b3cb-050dd34e530a-metrics-cert\") pod \"lws-controller-manager-5cbd5cd6f8-4kxhb\" (UID: \"a683d72c-2060-4e6e-b3cb-050dd34e530a\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"
Sep 16 14:20:18.629204 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.629179    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a683d72c-2060-4e6e-b3cb-050dd34e530a-cert\") pod \"lws-controller-manager-5cbd5cd6f8-4kxhb\" (UID: \"a683d72c-2060-4e6e-b3cb-050dd34e530a\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"
Sep 16 14:20:18.637001 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.636981    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lmtl\" (UniqueName: \"kubernetes.io/projected/a683d72c-2060-4e6e-b3cb-050dd34e530a-kube-api-access-9lmtl\") pod \"lws-controller-manager-5cbd5cd6f8-4kxhb\" (UID: \"a683d72c-2060-4e6e-b3cb-050dd34e530a\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"
Sep 16 14:20:18.733123 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:18.733090    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"
Sep 16 14:20:19.031846 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:19.031822    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"]
Sep 16 14:20:19.034336 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:20:19.034310    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda683d72c_2060_4e6e_b3cb_050dd34e530a.slice/crio-31b135f2b10917d25e5b2e875d510e3c7983cb3e465fd1a3aa8a9cea2b0c3c33 WatchSource:0}: Error finding container 31b135f2b10917d25e5b2e875d510e3c7983cb3e465fd1a3aa8a9cea2b0c3c33: Status 404 returned error can't find the container with id 31b135f2b10917d25e5b2e875d510e3c7983cb3e465fd1a3aa8a9cea2b0c3c33
Sep 16 14:20:19.508090 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:19.508059    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5" event={"ID":"63adf8da-3abd-458d-aae0-1a70b362c4f4","Type":"ContainerStarted","Data":"4e571fa46086eab151f1fbac345bdadbfa6d1423d702aa8415b6d618207a7fdc"}
Sep 16 14:20:19.509283 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:19.509258    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb" event={"ID":"a683d72c-2060-4e6e-b3cb-050dd34e530a","Type":"ContainerStarted","Data":"31b135f2b10917d25e5b2e875d510e3c7983cb3e465fd1a3aa8a9cea2b0c3c33"}
Sep 16 14:20:19.545709 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:19.545642    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5" podStartSLOduration=2.024978011 podStartE2EDuration="11.545619275s" totalImagesPullingTime="9.520641264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:20:19.545388969 +0000 UTC m=+370.092130903" watchObservedRunningTime="2026-09-16 14:20:19.545619275 +0000 UTC m=+370.092361207"
Sep 16 14:20:19.641043 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:19.641014    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:19.646366 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:19.646340    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:20.514326 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:20.514296    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:20.515243 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:20.515225    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5"
Sep 16 14:20:21.180232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.180197    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z9p4t"]
Sep 16 14:20:21.186181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.186144    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z9p4t"
Sep 16 14:20:21.205643 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.205618    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-pqr6k\""
Sep 16 14:20:21.206148 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.206126    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 16 14:20:21.208318 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.208293    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mqkwd"]
Sep 16 14:20:21.212662 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.212640    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mqkwd"
Sep 16 14:20:21.216035 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.216015    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-5zgwf\""
Sep 16 14:20:21.217231 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.217212    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 16 14:20:21.244993 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.244953    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z9p4t"]
Sep 16 14:20:21.247040 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.247017    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mqkwd"]
Sep 16 14:20:21.353384 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.353350    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fs89\" (UniqueName: \"kubernetes.io/projected/8bba3386-b1f4-4638-b6e0-c052a04f207a-kube-api-access-2fs89\") pod \"llmisvc-controller-manager-777c6bd7d5-mqkwd\" (UID: \"8bba3386-b1f4-4638-b6e0-c052a04f207a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mqkwd"
Sep 16 14:20:21.353541 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.353410    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20018af2-d3a0-46a0-bafb-c88d059786d0-cert\") pod \"kserve-controller-manager-d6f4b65b8-z9p4t\" (UID: \"20018af2-d3a0-46a0-bafb-c88d059786d0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z9p4t"
Sep 16 14:20:21.353541 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.353450    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh75l\" (UniqueName: \"kubernetes.io/projected/20018af2-d3a0-46a0-bafb-c88d059786d0-kube-api-access-nh75l\") pod \"kserve-controller-manager-d6f4b65b8-z9p4t\" (UID: \"20018af2-d3a0-46a0-bafb-c88d059786d0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z9p4t"
Sep 16 14:20:21.353541 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.353486    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bba3386-b1f4-4638-b6e0-c052a04f207a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mqkwd\" (UID: \"8bba3386-b1f4-4638-b6e0-c052a04f207a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mqkwd"
Sep 16 14:20:21.454113 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.454024    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fs89\" (UniqueName: \"kubernetes.io/projected/8bba3386-b1f4-4638-b6e0-c052a04f207a-kube-api-access-2fs89\") pod \"llmisvc-controller-manager-777c6bd7d5-mqkwd\" (UID: \"8bba3386-b1f4-4638-b6e0-c052a04f207a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mqkwd"
Sep 16 14:20:21.454295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.454121    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20018af2-d3a0-46a0-bafb-c88d059786d0-cert\") pod \"kserve-controller-manager-d6f4b65b8-z9p4t\" (UID: \"20018af2-d3a0-46a0-bafb-c88d059786d0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z9p4t"
Sep 16 14:20:21.454295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.454173    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh75l\" (UniqueName: \"kubernetes.io/projected/20018af2-d3a0-46a0-bafb-c88d059786d0-kube-api-access-nh75l\") pod \"kserve-controller-manager-d6f4b65b8-z9p4t\" (UID: \"20018af2-d3a0-46a0-bafb-c88d059786d0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z9p4t"
Sep 16 14:20:21.454295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.454214    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bba3386-b1f4-4638-b6e0-c052a04f207a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mqkwd\" (UID: \"8bba3386-b1f4-4638-b6e0-c052a04f207a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mqkwd"
Sep 16 14:20:21.456899 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.456869    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20018af2-d3a0-46a0-bafb-c88d059786d0-cert\") pod \"kserve-controller-manager-d6f4b65b8-z9p4t\" (UID: \"20018af2-d3a0-46a0-bafb-c88d059786d0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z9p4t"
Sep 16 14:20:21.457029 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.456886    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bba3386-b1f4-4638-b6e0-c052a04f207a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mqkwd\" (UID: \"8bba3386-b1f4-4638-b6e0-c052a04f207a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mqkwd"
Sep 16 14:20:21.466286 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.466259    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh75l\" (UniqueName: \"kubernetes.io/projected/20018af2-d3a0-46a0-bafb-c88d059786d0-kube-api-access-nh75l\") pod \"kserve-controller-manager-d6f4b65b8-z9p4t\" (UID: \"20018af2-d3a0-46a0-bafb-c88d059786d0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z9p4t"
Sep 16 14:20:21.468615 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.468569    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fs89\" (UniqueName: \"kubernetes.io/projected/8bba3386-b1f4-4638-b6e0-c052a04f207a-kube-api-access-2fs89\") pod \"llmisvc-controller-manager-777c6bd7d5-mqkwd\" (UID: \"8bba3386-b1f4-4638-b6e0-c052a04f207a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mqkwd"
Sep 16 14:20:21.499260 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.499229    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z9p4t"
Sep 16 14:20:21.524675 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.524638    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mqkwd"
Sep 16 14:20:21.922277 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.922252    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z9p4t"]
Sep 16 14:20:21.924544 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:20:21.924518    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20018af2_d3a0_46a0_bafb_c88d059786d0.slice/crio-15fff8f3c087a5edd9a5dc170b0320c006aca9af00af81185e7e0fadbac101a3 WatchSource:0}: Error finding container 15fff8f3c087a5edd9a5dc170b0320c006aca9af00af81185e7e0fadbac101a3: Status 404 returned error can't find the container with id 15fff8f3c087a5edd9a5dc170b0320c006aca9af00af81185e7e0fadbac101a3
Sep 16 14:20:21.934466 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:21.934444    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mqkwd"]
Sep 16 14:20:21.935137 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:20:21.935115    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8bba3386_b1f4_4638_b6e0_c052a04f207a.slice/crio-9a5884e7915a338371002adc7dd2f0398ff1b7a42e344795e21dbf5cc798ae4e WatchSource:0}: Error finding container 9a5884e7915a338371002adc7dd2f0398ff1b7a42e344795e21dbf5cc798ae4e: Status 404 returned error can't find the container with id 9a5884e7915a338371002adc7dd2f0398ff1b7a42e344795e21dbf5cc798ae4e
Sep 16 14:20:22.525455 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:22.525423    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb" event={"ID":"a683d72c-2060-4e6e-b3cb-050dd34e530a","Type":"ContainerStarted","Data":"5436df09180e0d99b748713dec599b75e88c08627c2cc4dda75fd9f8cdc62496"}
Sep 16 14:20:22.525953 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:22.525538    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"
Sep 16 14:20:22.529038 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:22.529005    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mqkwd" event={"ID":"8bba3386-b1f4-4638-b6e0-c052a04f207a","Type":"ContainerStarted","Data":"9a5884e7915a338371002adc7dd2f0398ff1b7a42e344795e21dbf5cc798ae4e"}
Sep 16 14:20:22.530460 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:22.530436    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z9p4t" event={"ID":"20018af2-d3a0-46a0-bafb-c88d059786d0","Type":"ContainerStarted","Data":"15fff8f3c087a5edd9a5dc170b0320c006aca9af00af81185e7e0fadbac101a3"}
Sep 16 14:20:22.550635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:22.550584    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb" podStartSLOduration=1.900517041 podStartE2EDuration="4.550568626s" totalImagesPullingTime="2.650051585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:20:22.549272971 +0000 UTC m=+373.096014904" watchObservedRunningTime="2026-09-16 14:20:22.550568626 +0000 UTC m=+373.097310556"
Sep 16 14:20:26.556967 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:26.556912    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mqkwd" event={"ID":"8bba3386-b1f4-4638-b6e0-c052a04f207a","Type":"ContainerStarted","Data":"f630e6eab5e1ed58f67eabe0b8573f32b2f2287990898ad81dcb187c21b7019a"}
Sep 16 14:20:26.557430 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:26.557118    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mqkwd"
Sep 16 14:20:26.558442 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:26.558415    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z9p4t" event={"ID":"20018af2-d3a0-46a0-bafb-c88d059786d0","Type":"ContainerStarted","Data":"018905b4bdf3a94bab8bd38114b63d10b92ce6d4164fe168ab992a85004bba08"}
Sep 16 14:20:26.558604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:26.558587    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z9p4t"
Sep 16 14:20:26.581110 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:26.581061    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mqkwd" podStartSLOduration=1.9751419129999999 podStartE2EDuration="5.581047033s" totalImagesPullingTime="3.60590512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:20:26.579866372 +0000 UTC m=+377.126608304" watchObservedRunningTime="2026-09-16 14:20:26.581047033 +0000 UTC m=+377.127789008"
Sep 16 14:20:26.604014 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:26.603974    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z9p4t" podStartSLOduration=2.038551298 podStartE2EDuration="5.603960461s" totalImagesPullingTime="3.565409163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:20:26.60293971 +0000 UTC m=+377.149681642" watchObservedRunningTime="2026-09-16 14:20:26.603960461 +0000 UTC m=+377.150702391"
Sep 16 14:20:33.541270 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:33.541239    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-4kxhb"
Sep 16 14:20:57.565188 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:57.565152    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mqkwd"
Sep 16 14:20:57.568322 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:20:57.568300    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z9p4t"
Sep 16 14:21:29.270683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:29.270651    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-7np9h"]
Sep 16 14:21:29.272940 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:29.272922    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-7np9h"
Sep 16 14:21:29.277185 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:29.277162    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 14:21:29.277974 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:29.277955    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 14:21:29.280513 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:29.280498    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8kshc\""
Sep 16 14:21:29.286225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:29.286197    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-7np9h"]
Sep 16 14:21:29.308027 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:29.308002    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mftvw\" (UniqueName: \"kubernetes.io/projected/436a2c5e-3287-408c-8111-0bf29d3777f7-kube-api-access-mftvw\") pod \"authorino-operator-76b478bd78-7np9h\" (UID: \"436a2c5e-3287-408c-8111-0bf29d3777f7\") " pod="kuadrant-system/authorino-operator-76b478bd78-7np9h"
Sep 16 14:21:29.409251 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:29.409219    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mftvw\" (UniqueName: \"kubernetes.io/projected/436a2c5e-3287-408c-8111-0bf29d3777f7-kube-api-access-mftvw\") pod \"authorino-operator-76b478bd78-7np9h\" (UID: \"436a2c5e-3287-408c-8111-0bf29d3777f7\") " pod="kuadrant-system/authorino-operator-76b478bd78-7np9h"
Sep 16 14:21:29.431663 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:29.431634    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mftvw\" (UniqueName: \"kubernetes.io/projected/436a2c5e-3287-408c-8111-0bf29d3777f7-kube-api-access-mftvw\") pod \"authorino-operator-76b478bd78-7np9h\" (UID: \"436a2c5e-3287-408c-8111-0bf29d3777f7\") " pod="kuadrant-system/authorino-operator-76b478bd78-7np9h"
Sep 16 14:21:29.583321 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:29.583243    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-7np9h"
Sep 16 14:21:29.878129 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:29.878048    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-7np9h"]
Sep 16 14:21:29.881700 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:21:29.881673    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436a2c5e_3287_408c_8111_0bf29d3777f7.slice/crio-01652ceee72d5daad3c7130e19b70f4497bb6cce4c38a59cbdfe84cac13fbc44 WatchSource:0}: Error finding container 01652ceee72d5daad3c7130e19b70f4497bb6cce4c38a59cbdfe84cac13fbc44: Status 404 returned error can't find the container with id 01652ceee72d5daad3c7130e19b70f4497bb6cce4c38a59cbdfe84cac13fbc44
Sep 16 14:21:30.860148 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:30.860108    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-7np9h" event={"ID":"436a2c5e-3287-408c-8111-0bf29d3777f7","Type":"ContainerStarted","Data":"01652ceee72d5daad3c7130e19b70f4497bb6cce4c38a59cbdfe84cac13fbc44"}
Sep 16 14:21:32.875039 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:32.875004    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-7np9h" event={"ID":"436a2c5e-3287-408c-8111-0bf29d3777f7","Type":"ContainerStarted","Data":"f2c378bb81caf8161f3e3501504e2c973c11603d08e1083f7400c82068e98c0e"}
Sep 16 14:21:32.875437 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:32.875151    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-7np9h"
Sep 16 14:21:32.979836 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:32.979770    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-7np9h" podStartSLOduration=1.792570006 podStartE2EDuration="3.979753223s" totalImagesPullingTime="2.187183217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:21:32.965631153 +0000 UTC m=+443.512373096" watchObservedRunningTime="2026-09-16 14:21:32.979753223 +0000 UTC m=+443.526495154"
Sep 16 14:21:43.883130 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:43.883098    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-7np9h"
Sep 16 14:21:46.405248 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:46.405215    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"]
Sep 16 14:21:46.408824 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:46.408808    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"
Sep 16 14:21:46.424766 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:46.424745    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wt2h7\""
Sep 16 14:21:46.444092 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:46.444063    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"]
Sep 16 14:21:46.448145 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:46.448119    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5df2edc4-d6c5-4c5c-95b7-399f90a35586-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nr88p\" (UID: \"5df2edc4-d6c5-4c5c-95b7-399f90a35586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"
Sep 16 14:21:46.448258 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:46.448185    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl9jl\" (UniqueName: \"kubernetes.io/projected/5df2edc4-d6c5-4c5c-95b7-399f90a35586-kube-api-access-cl9jl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nr88p\" (UID: \"5df2edc4-d6c5-4c5c-95b7-399f90a35586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"
Sep 16 14:21:46.548599 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:46.548564    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5df2edc4-d6c5-4c5c-95b7-399f90a35586-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nr88p\" (UID: \"5df2edc4-d6c5-4c5c-95b7-399f90a35586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"
Sep 16 14:21:46.548752 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:46.548626    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl9jl\" (UniqueName: \"kubernetes.io/projected/5df2edc4-d6c5-4c5c-95b7-399f90a35586-kube-api-access-cl9jl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nr88p\" (UID: \"5df2edc4-d6c5-4c5c-95b7-399f90a35586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"
Sep 16 14:21:46.548956 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:46.548936    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5df2edc4-d6c5-4c5c-95b7-399f90a35586-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nr88p\" (UID: \"5df2edc4-d6c5-4c5c-95b7-399f90a35586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"
Sep 16 14:21:46.582738 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:46.582706    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl9jl\" (UniqueName: \"kubernetes.io/projected/5df2edc4-d6c5-4c5c-95b7-399f90a35586-kube-api-access-cl9jl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-nr88p\" (UID: \"5df2edc4-d6c5-4c5c-95b7-399f90a35586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"
Sep 16 14:21:46.718698 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:46.718617    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"
Sep 16 14:21:46.997056 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:46.997031    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"]
Sep 16 14:21:46.999175 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:21:46.999144    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df2edc4_d6c5_4c5c_95b7_399f90a35586.slice/crio-4dac47b79eaee8e7559b7daa176aad505be7642621dd3318d10072fcf970f30c WatchSource:0}: Error finding container 4dac47b79eaee8e7559b7daa176aad505be7642621dd3318d10072fcf970f30c: Status 404 returned error can't find the container with id 4dac47b79eaee8e7559b7daa176aad505be7642621dd3318d10072fcf970f30c
Sep 16 14:21:47.955680 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:47.955641    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p" event={"ID":"5df2edc4-d6c5-4c5c-95b7-399f90a35586","Type":"ContainerStarted","Data":"4dac47b79eaee8e7559b7daa176aad505be7642621dd3318d10072fcf970f30c"}
Sep 16 14:21:50.972853 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:50.972808    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p" event={"ID":"5df2edc4-d6c5-4c5c-95b7-399f90a35586","Type":"ContainerStarted","Data":"9556fcca9065bc06eff744bf265ed3f80f3fd437b3f4ea134c23dedcd6f7652c"}
Sep 16 14:21:50.973269 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:50.972873    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"
Sep 16 14:21:50.999567 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:21:50.999520    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p" podStartSLOduration=1.610154426 podStartE2EDuration="4.999506747s" totalImagesPullingTime="3.389352321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:21:50.998631112 +0000 UTC m=+461.545373044" watchObservedRunningTime="2026-09-16 14:21:50.999506747 +0000 UTC m=+461.546248678"
Sep 16 14:22:01.980583 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:01.980555    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"
Sep 16 14:22:42.026316 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:42.026285    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qxpfl"]
Sep 16 14:22:42.028920 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:42.028878    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qxpfl"
Sep 16 14:22:42.031860 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:42.031837    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:22:42.031994 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:42.031928    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-2w5c8\""
Sep 16 14:22:42.031994 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:42.031960    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 14:22:42.032970 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:42.032954    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 16 14:22:42.041784 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:42.041764    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qxpfl"]
Sep 16 14:22:42.128631 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:42.128601    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f8bf5d21-2659-4472-a72f-da2f20e6428a-data\") pod \"postgres-69fb4594cf-qxpfl\" (UID: \"f8bf5d21-2659-4472-a72f-da2f20e6428a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qxpfl"
Sep 16 14:22:42.128795 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:42.128644    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqmbt\" (UniqueName: \"kubernetes.io/projected/f8bf5d21-2659-4472-a72f-da2f20e6428a-kube-api-access-bqmbt\") pod \"postgres-69fb4594cf-qxpfl\" (UID: \"f8bf5d21-2659-4472-a72f-da2f20e6428a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qxpfl"
Sep 16 14:22:42.229762 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:42.229723    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f8bf5d21-2659-4472-a72f-da2f20e6428a-data\") pod \"postgres-69fb4594cf-qxpfl\" (UID: \"f8bf5d21-2659-4472-a72f-da2f20e6428a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qxpfl"
Sep 16 14:22:42.229947 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:42.229780    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqmbt\" (UniqueName: \"kubernetes.io/projected/f8bf5d21-2659-4472-a72f-da2f20e6428a-kube-api-access-bqmbt\") pod \"postgres-69fb4594cf-qxpfl\" (UID: \"f8bf5d21-2659-4472-a72f-da2f20e6428a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qxpfl"
Sep 16 14:22:42.230123 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:42.230106    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f8bf5d21-2659-4472-a72f-da2f20e6428a-data\") pod \"postgres-69fb4594cf-qxpfl\" (UID: \"f8bf5d21-2659-4472-a72f-da2f20e6428a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qxpfl"
Sep 16 14:22:42.239948 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:42.239925    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqmbt\" (UniqueName: \"kubernetes.io/projected/f8bf5d21-2659-4472-a72f-da2f20e6428a-kube-api-access-bqmbt\") pod \"postgres-69fb4594cf-qxpfl\" (UID: \"f8bf5d21-2659-4472-a72f-da2f20e6428a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qxpfl"
Sep 16 14:22:42.339932 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:42.339830    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qxpfl"
Sep 16 14:22:42.616009 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:42.615985    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qxpfl"]
Sep 16 14:22:42.618029 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:22:42.618005    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bf5d21_2659_4472_a72f_da2f20e6428a.slice/crio-fffd2370be3a1a5cf4fe411c2af6200ca4147c5afafea73d439326350ca46757 WatchSource:0}: Error finding container fffd2370be3a1a5cf4fe411c2af6200ca4147c5afafea73d439326350ca46757: Status 404 returned error can't find the container with id fffd2370be3a1a5cf4fe411c2af6200ca4147c5afafea73d439326350ca46757
Sep 16 14:22:43.220507 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:43.220475    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qxpfl" event={"ID":"f8bf5d21-2659-4472-a72f-da2f20e6428a","Type":"ContainerStarted","Data":"fffd2370be3a1a5cf4fe411c2af6200ca4147c5afafea73d439326350ca46757"}
Sep 16 14:22:48.247444 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:48.247407    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qxpfl" event={"ID":"f8bf5d21-2659-4472-a72f-da2f20e6428a","Type":"ContainerStarted","Data":"2887793289a79138083ab0bac691f6c112249aa579ed9c7bed6207c576918f70"}
Sep 16 14:22:48.247822 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:48.247566    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qxpfl"
Sep 16 14:22:54.280682 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:54.280655    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qxpfl"
Sep 16 14:22:54.300342 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:22:54.300296    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qxpfl" podStartSLOduration=8.606442586 podStartE2EDuration="13.300281677s" totalImagesPullingTime="4.693839091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:22:48.279738859 +0000 UTC m=+518.826480790" watchObservedRunningTime="2026-09-16 14:22:54.300281677 +0000 UTC m=+524.847023608"
Sep 16 14:23:39.944142 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:39.944105    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-g9lgv"]
Sep 16 14:23:39.947798 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:39.947779    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-g9lgv"
Sep 16 14:23:39.951614 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:39.951591    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 14:23:39.952173 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:39.952150    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-mktph\""
Sep 16 14:23:39.965387 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:39.965361    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-g9lgv"]
Sep 16 14:23:40.040824 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:40.040793    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lz5r\" (UniqueName: \"kubernetes.io/projected/2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b-kube-api-access-6lz5r\") pod \"payload-processing-5b54b7fbb7-g9lgv\" (UID: \"2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-g9lgv"
Sep 16 14:23:40.040993 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:40.040855    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-g9lgv\" (UID: \"2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-g9lgv"
Sep 16 14:23:40.141860 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:40.141829    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-g9lgv\" (UID: \"2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-g9lgv"
Sep 16 14:23:40.142044 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:40.141920    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lz5r\" (UniqueName: \"kubernetes.io/projected/2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b-kube-api-access-6lz5r\") pod \"payload-processing-5b54b7fbb7-g9lgv\" (UID: \"2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-g9lgv"
Sep 16 14:23:40.142610 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:40.142584    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-g9lgv\" (UID: \"2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-g9lgv"
Sep 16 14:23:40.150398 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:40.150374    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lz5r\" (UniqueName: \"kubernetes.io/projected/2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b-kube-api-access-6lz5r\") pod \"payload-processing-5b54b7fbb7-g9lgv\" (UID: \"2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-g9lgv"
Sep 16 14:23:40.257788 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:40.257755    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-g9lgv"
Sep 16 14:23:40.534593 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:40.534573    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-g9lgv"]
Sep 16 14:23:40.536992 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:23:40.536963    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9d6dec_6bfa_483b_92a3_f1bc5e6c624b.slice/crio-409f30158bfc4734314a19aee2a2519831c91a5c5fb45d002b6a8185335f7b76 WatchSource:0}: Error finding container 409f30158bfc4734314a19aee2a2519831c91a5c5fb45d002b6a8185335f7b76: Status 404 returned error can't find the container with id 409f30158bfc4734314a19aee2a2519831c91a5c5fb45d002b6a8185335f7b76
Sep 16 14:23:41.503712 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:41.503672    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-g9lgv" event={"ID":"2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b","Type":"ContainerStarted","Data":"409f30158bfc4734314a19aee2a2519831c91a5c5fb45d002b6a8185335f7b76"}
Sep 16 14:23:43.515756 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:43.515725    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-g9lgv" event={"ID":"2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b","Type":"ContainerStarted","Data":"e240a912ea8f332e130dd98a3556df6ef7e9812c7c621dbd524765f0ab4e3588"}
Sep 16 14:23:43.516135 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:43.515850    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-g9lgv"
Sep 16 14:23:54.522552 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:54.522518    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-g9lgv"
Sep 16 14:23:54.544222 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:23:54.544173    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-g9lgv" podStartSLOduration=13.068390206 podStartE2EDuration="15.54415765s" totalImagesPullingTime="2.475767444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:23:43.542062955 +0000 UTC m=+574.088804888" watchObservedRunningTime="2026-09-16 14:23:54.54415765 +0000 UTC m=+585.090899580"
Sep 16 14:24:09.818012 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:09.817975    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:24:09.819799 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:09.819777    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:24:54.223884 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.223851    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"]
Sep 16 14:24:54.227693 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.227674    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.231624 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.231564    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 14:24:54.231812 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.231792    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bm6t2\""
Sep 16 14:24:54.231923 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.231838    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 14:24:54.231982 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.231950    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 16 14:24:54.239211 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.239192    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"]
Sep 16 14:24:54.286683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.286647    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ef12922-dcf0-4d90-8c04-0f63d2c36375-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.286836 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.286699    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ef12922-dcf0-4d90-8c04-0f63d2c36375-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.286836 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.286750    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ef12922-dcf0-4d90-8c04-0f63d2c36375-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.286836 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.286810    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ef12922-dcf0-4d90-8c04-0f63d2c36375-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.286969 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.286841    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ef12922-dcf0-4d90-8c04-0f63d2c36375-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.286969 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.286911    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kptvx\" (UniqueName: \"kubernetes.io/projected/4ef12922-dcf0-4d90-8c04-0f63d2c36375-kube-api-access-kptvx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.388036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.388000    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ef12922-dcf0-4d90-8c04-0f63d2c36375-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.388200 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.388060    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ef12922-dcf0-4d90-8c04-0f63d2c36375-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.388200 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.388102    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ef12922-dcf0-4d90-8c04-0f63d2c36375-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.388300 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.388225    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ef12922-dcf0-4d90-8c04-0f63d2c36375-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.388300 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.388280    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kptvx\" (UniqueName: \"kubernetes.io/projected/4ef12922-dcf0-4d90-8c04-0f63d2c36375-kube-api-access-kptvx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.388400 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.388368    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ef12922-dcf0-4d90-8c04-0f63d2c36375-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.388452 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.388395    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ef12922-dcf0-4d90-8c04-0f63d2c36375-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.388452 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.388411    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ef12922-dcf0-4d90-8c04-0f63d2c36375-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.388671 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.388651    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ef12922-dcf0-4d90-8c04-0f63d2c36375-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.390495 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.390463    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ef12922-dcf0-4d90-8c04-0f63d2c36375-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.390706 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.390684    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ef12922-dcf0-4d90-8c04-0f63d2c36375-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.397169 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.397149    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kptvx\" (UniqueName: \"kubernetes.io/projected/4ef12922-dcf0-4d90-8c04-0f63d2c36375-kube-api-access-kptvx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z\" (UID: \"4ef12922-dcf0-4d90-8c04-0f63d2c36375\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.537861 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.537782    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:24:54.812423 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.812397    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"]
Sep 16 14:24:54.814625 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:24:54.814589    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef12922_dcf0_4d90_8c04_0f63d2c36375.slice/crio-dc31848cc612b3896bd42045c4054e3cddc3f97e98c585e349dbeab15761b631 WatchSource:0}: Error finding container dc31848cc612b3896bd42045c4054e3cddc3f97e98c585e349dbeab15761b631: Status 404 returned error can't find the container with id dc31848cc612b3896bd42045c4054e3cddc3f97e98c585e349dbeab15761b631
Sep 16 14:24:54.815919 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.815904    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:24:54.875000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:54.874960    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z" event={"ID":"4ef12922-dcf0-4d90-8c04-0f63d2c36375","Type":"ContainerStarted","Data":"dc31848cc612b3896bd42045c4054e3cddc3f97e98c585e349dbeab15761b631"}
Sep 16 14:24:56.886564 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:56.886529    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z" event={"ID":"4ef12922-dcf0-4d90-8c04-0f63d2c36375","Type":"ContainerStarted","Data":"ebe91d01ac1e4bc06bf7b35fe37177073458d3ff832c8729b4c76832ba0fa7e3"}
Sep 16 14:24:56.906571 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:24:56.906519    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z" podStartSLOduration=1.882252062 podStartE2EDuration="2.906476718s" totalImagesPullingTime="1.024224656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:24:56.90503674 +0000 UTC m=+647.451778672" watchObservedRunningTime="2026-09-16 14:24:56.906476718 +0000 UTC m=+647.453218666"
Sep 16 14:25:04.538666 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:04.538620    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:25:04.539078 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:04.538691    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:25:04.551409 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:04.551382    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:25:04.938029 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:04.937953    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z"
Sep 16 14:25:21.472335 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.472297    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"]
Sep 16 14:25:21.477157 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.477119    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.483264 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.483234    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 16 14:25:21.496309 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.496282    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"]
Sep 16 14:25:21.646445 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.646407    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a009633-059b-4702-bbdd-ad7d8c083a04-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.646608 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.646452    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a009633-059b-4702-bbdd-ad7d8c083a04-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.646608 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.646591    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dpcc\" (UniqueName: \"kubernetes.io/projected/4a009633-059b-4702-bbdd-ad7d8c083a04-kube-api-access-2dpcc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.646689 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.646654    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a009633-059b-4702-bbdd-ad7d8c083a04-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.646689 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.646676    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a009633-059b-4702-bbdd-ad7d8c083a04-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.646752 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.646705    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a009633-059b-4702-bbdd-ad7d8c083a04-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.747230 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.747156    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a009633-059b-4702-bbdd-ad7d8c083a04-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.747230 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.747195    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a009633-059b-4702-bbdd-ad7d8c083a04-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.747406 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.747238    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dpcc\" (UniqueName: \"kubernetes.io/projected/4a009633-059b-4702-bbdd-ad7d8c083a04-kube-api-access-2dpcc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.747406 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.747273    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a009633-059b-4702-bbdd-ad7d8c083a04-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.747406 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.747290    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a009633-059b-4702-bbdd-ad7d8c083a04-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.747406 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.747321    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a009633-059b-4702-bbdd-ad7d8c083a04-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.747581 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.747561    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a009633-059b-4702-bbdd-ad7d8c083a04-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.747635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.747613    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a009633-059b-4702-bbdd-ad7d8c083a04-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.747702 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.747687    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a009633-059b-4702-bbdd-ad7d8c083a04-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.749307 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.749286    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a009633-059b-4702-bbdd-ad7d8c083a04-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.749622 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.749606    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a009633-059b-4702-bbdd-ad7d8c083a04-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.763928 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.763876    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dpcc\" (UniqueName: \"kubernetes.io/projected/4a009633-059b-4702-bbdd-ad7d8c083a04-kube-api-access-2dpcc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b\" (UID: \"4a009633-059b-4702-bbdd-ad7d8c083a04\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:21.788515 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:21.788494    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:22.071178 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:22.071152    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"]
Sep 16 14:25:22.072825 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:25:22.072799    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a009633_059b_4702_bbdd_ad7d8c083a04.slice/crio-2d5415672cb2e990d7f96edb4fc795167d1cf67f8a6e7f5fb52c73bfb2b311bd WatchSource:0}: Error finding container 2d5415672cb2e990d7f96edb4fc795167d1cf67f8a6e7f5fb52c73bfb2b311bd: Status 404 returned error can't find the container with id 2d5415672cb2e990d7f96edb4fc795167d1cf67f8a6e7f5fb52c73bfb2b311bd
Sep 16 14:25:23.023745 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:23.023701    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b" event={"ID":"4a009633-059b-4702-bbdd-ad7d8c083a04","Type":"ContainerStarted","Data":"464d93000201bc217f0fd856d187061ed02f63065e82feec0ef4cb6e2abfe29a"}
Sep 16 14:25:23.024213 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:23.023757    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b" event={"ID":"4a009633-059b-4702-bbdd-ad7d8c083a04","Type":"ContainerStarted","Data":"2d5415672cb2e990d7f96edb4fc795167d1cf67f8a6e7f5fb52c73bfb2b311bd"}
Sep 16 14:25:23.078684 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:23.078636    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b" podStartSLOduration=1.78971745 podStartE2EDuration="2.078622934s" totalImagesPullingTime="288.905484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:25:23.078419999 +0000 UTC m=+673.625161931" watchObservedRunningTime="2026-09-16 14:25:23.078622934 +0000 UTC m=+673.625364864"
Sep 16 14:25:31.789652 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:31.789619    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:31.790033 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:31.789685    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:31.802169 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:31.802146    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:25:32.086169 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:25:32.086088    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b"
Sep 16 14:27:18.916105 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:18.916073    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"]
Sep 16 14:27:18.918855 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:18.918831    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:18.921335 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:18.921317    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-094dab-openshift-default-dockercfg-xvkx4\""
Sep 16 14:27:18.934181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:18.934156    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"]
Sep 16 14:27:18.978615 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:18.978582    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-token\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:18.978615 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:18.978614    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf55h\" (UniqueName: \"kubernetes.io/projected/e15beb83-9f62-4765-9efe-74a0be02cbfe-kube-api-access-cf55h\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:18.978789 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:18.978638    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-workload-certs\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:18.978789 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:18.978686    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-workload-socket\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:18.978789 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:18.978708    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-data\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:18.978789 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:18.978763    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-credential-socket\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:18.978940 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:18.978795    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15beb83-9f62-4765-9efe-74a0be02cbfe-istiod-ca-cert\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:18.978940 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:18.978830    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-podinfo\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:18.978940 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:18.978875    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-envoy\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.079377 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.079340    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-envoy\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.079555 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.079389    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-token\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.079555 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.079413    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf55h\" (UniqueName: \"kubernetes.io/projected/e15beb83-9f62-4765-9efe-74a0be02cbfe-kube-api-access-cf55h\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.079555 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.079440    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-workload-certs\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.079555 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.079467    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-workload-socket\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.079555 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.079490    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-data\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.079555 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.079539    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-credential-socket\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.079867 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.079598    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15beb83-9f62-4765-9efe-74a0be02cbfe-istiod-ca-cert\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.079867 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.079676    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-podinfo\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.080003 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.079922    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-workload-certs\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.080003 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.079955    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-workload-socket\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.080092 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.080048    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-data\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.080199 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.080180    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-credential-socket\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.080369 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.080350    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15beb83-9f62-4765-9efe-74a0be02cbfe-istiod-ca-cert\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.081794 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.081770    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-envoy\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.082096 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.082080    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-podinfo\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.091878 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.091849    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf55h\" (UniqueName: \"kubernetes.io/projected/e15beb83-9f62-4765-9efe-74a0be02cbfe-kube-api-access-cf55h\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.092262 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.092241    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-token\") pod \"e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") " pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.230050 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.230017    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:19.507816 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.507777    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"]
Sep 16 14:27:19.509776 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:27:19.509747    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15beb83_9f62_4765_9efe_74a0be02cbfe.slice/crio-31317e1486cbf8f6e5842f6bc269ab8144bcecff2ff41af9f09e65fafc61b8e3 WatchSource:0}: Error finding container 31317e1486cbf8f6e5842f6bc269ab8144bcecff2ff41af9f09e65fafc61b8e3: Status 404 returned error can't find the container with id 31317e1486cbf8f6e5842f6bc269ab8144bcecff2ff41af9f09e65fafc61b8e3
Sep 16 14:27:19.511977 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.511944    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:19.512073 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.512010    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:19.512073 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.512062    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:19.615569 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.615538    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4" event={"ID":"e15beb83-9f62-4765-9efe-74a0be02cbfe","Type":"ContainerStarted","Data":"44ed5742430b80f8a168537b51fd601584769d8ed7bf271313de6a0a9fc20a23"}
Sep 16 14:27:19.615732 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.615596    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4" event={"ID":"e15beb83-9f62-4765-9efe-74a0be02cbfe","Type":"ContainerStarted","Data":"31317e1486cbf8f6e5842f6bc269ab8144bcecff2ff41af9f09e65fafc61b8e3"}
Sep 16 14:27:19.638795 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:19.638742    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4" podStartSLOduration=1.638726812 podStartE2EDuration="1.638726812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:19.636423656 +0000 UTC m=+790.183165589" watchObservedRunningTime="2026-09-16 14:27:19.638726812 +0000 UTC m=+790.185468742"
Sep 16 14:27:20.230973 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:20.230933    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:20.235585 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:20.235559    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:20.621905 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:20.621812    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:20.622850 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:20.622831    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:26.277193 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.277156    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"]
Sep 16 14:27:26.280562 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.280542    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"
Sep 16 14:27:26.283214 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.283195    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-924916\""
Sep 16 14:27:26.283340 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.283234    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-924916-dockercfg-68zh5\""
Sep 16 14:27:26.293698 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.293676    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"]
Sep 16 14:27:26.353243 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.353213    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmmdn\" (UniqueName: \"kubernetes.io/projected/fb3dc797-c169-49f0-a387-b6760e6a1f80-kube-api-access-qmmdn\") pod \"payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6\" (UID: \"fb3dc797-c169-49f0-a387-b6760e6a1f80\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"
Sep 16 14:27:26.353585 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.353546    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3dc797-c169-49f0-a387-b6760e6a1f80-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6\" (UID: \"fb3dc797-c169-49f0-a387-b6760e6a1f80\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"
Sep 16 14:27:26.454845 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.454801    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmmdn\" (UniqueName: \"kubernetes.io/projected/fb3dc797-c169-49f0-a387-b6760e6a1f80-kube-api-access-qmmdn\") pod \"payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6\" (UID: \"fb3dc797-c169-49f0-a387-b6760e6a1f80\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"
Sep 16 14:27:26.454845 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.454842    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3dc797-c169-49f0-a387-b6760e6a1f80-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6\" (UID: \"fb3dc797-c169-49f0-a387-b6760e6a1f80\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"
Sep 16 14:27:26.455559 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.455536    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3dc797-c169-49f0-a387-b6760e6a1f80-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6\" (UID: \"fb3dc797-c169-49f0-a387-b6760e6a1f80\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"
Sep 16 14:27:26.463380 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.463348    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmmdn\" (UniqueName: \"kubernetes.io/projected/fb3dc797-c169-49f0-a387-b6760e6a1f80-kube-api-access-qmmdn\") pod \"payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6\" (UID: \"fb3dc797-c169-49f0-a387-b6760e6a1f80\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"
Sep 16 14:27:26.591834 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.591749    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"
Sep 16 14:27:26.835638 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.835606    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"]
Sep 16 14:27:26.838144 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.838122    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:26.842286 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.842232    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-84fdff-openshift-default-dockercfg-s5hvr\""
Sep 16 14:27:26.855170 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.855146    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"]
Sep 16 14:27:26.870204 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.870186    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"]
Sep 16 14:27:26.872621 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:27:26.872589    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3dc797_c169_49f0_a387_b6760e6a1f80.slice/crio-648333514baded8bda6208e074e14ebc32a625bc2f6a4f4d590c04e633c5b5e9 WatchSource:0}: Error finding container 648333514baded8bda6208e074e14ebc32a625bc2f6a4f4d590c04e633c5b5e9: Status 404 returned error can't find the container with id 648333514baded8bda6208e074e14ebc32a625bc2f6a4f4d590c04e633c5b5e9
Sep 16 14:27:26.960921 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.960875    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-token\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:26.961050 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.960944    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-data\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:26.961050 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.960991    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-envoy\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:26.961050 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.961027    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istiod-ca-cert\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:26.961193 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.961092    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-podinfo\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:26.961193 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.961121    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8vsm\" (UniqueName: \"kubernetes.io/projected/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-kube-api-access-c8vsm\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:26.961193 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.961146    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-workload-certs\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:26.961193 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.961167    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-credential-socket\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:26.961193 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:26.961185    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-workload-socket\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.062492 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.062452    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-token\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.062658 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.062510    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-data\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.062658 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.062548    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-envoy\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.062658 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.062588    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istiod-ca-cert\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.062658 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.062652    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-podinfo\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.062883 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.062681    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8vsm\" (UniqueName: \"kubernetes.io/projected/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-kube-api-access-c8vsm\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.062883 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.062735    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-workload-certs\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.062883 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.062784    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-credential-socket\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.062883 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.062816    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-workload-socket\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.063239 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.063015    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-data\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.063239 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.063160    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-workload-socket\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.063342 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.063241    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-workload-certs\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.063342 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.063281    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-credential-socket\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.063445 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.063385    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istiod-ca-cert\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.065540 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.065516    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-envoy\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.065705 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.065689    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-podinfo\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.070718 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.070699    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-token\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.070947 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.070926    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8vsm\" (UniqueName: \"kubernetes.io/projected/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-kube-api-access-c8vsm\") pod \"e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") " pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.149880 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.149808    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:27.429648 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.429620    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"]
Sep 16 14:27:27.432111 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:27:27.432079    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d64442e_52ea_46fe_a65e_6cbfeda11c7f.slice/crio-2882944ed518074278c183f076417cef258f0c3290759d41d95638e5b20203bd WatchSource:0}: Error finding container 2882944ed518074278c183f076417cef258f0c3290759d41d95638e5b20203bd: Status 404 returned error can't find the container with id 2882944ed518074278c183f076417cef258f0c3290759d41d95638e5b20203bd
Sep 16 14:27:27.434500 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.434466    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:27.434568 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.434531    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:27.434568 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.434563    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:27.668623 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.668581    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq" event={"ID":"9d64442e-52ea-46fe-a65e-6cbfeda11c7f","Type":"ContainerStarted","Data":"a1f6397dafe5d11e42e899ae9c5a95d7ee60f882184f77f35ac9c76af0f65c27"}
Sep 16 14:27:27.668787 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.668654    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq" event={"ID":"9d64442e-52ea-46fe-a65e-6cbfeda11c7f","Type":"ContainerStarted","Data":"2882944ed518074278c183f076417cef258f0c3290759d41d95638e5b20203bd"}
Sep 16 14:27:27.670563 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.670527    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6" event={"ID":"fb3dc797-c169-49f0-a387-b6760e6a1f80","Type":"ContainerStarted","Data":"c699ec3265b52db13c0125baff66ff92e196a95c2d1ce6628c2421e50c73d3d3"}
Sep 16 14:27:27.670694 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.670585    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6" event={"ID":"fb3dc797-c169-49f0-a387-b6760e6a1f80","Type":"ContainerStarted","Data":"648333514baded8bda6208e074e14ebc32a625bc2f6a4f4d590c04e633c5b5e9"}
Sep 16 14:27:27.670694 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.670629    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"
Sep 16 14:27:27.692352 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.692258    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq" podStartSLOduration=1.692237466 podStartE2EDuration="1.692237466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:27.690094976 +0000 UTC m=+798.236836946" watchObservedRunningTime="2026-09-16 14:27:27.692237466 +0000 UTC m=+798.238979398"
Sep 16 14:27:27.716726 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:27.716677    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6" podStartSLOduration=1.716661492 podStartE2EDuration="1.716661492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:27.714939167 +0000 UTC m=+798.261681098" watchObservedRunningTime="2026-09-16 14:27:27.716661492 +0000 UTC m=+798.263403422"
Sep 16 14:27:28.150991 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:28.150953    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:28.158710 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:28.158685    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:28.676327 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:28.676292    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:28.677141 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:28.677119    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:29.205196 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.205160    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"]
Sep 16 14:27:29.208330 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.208311    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.211870 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.211849    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1e4514fd-openshift-default-dockercfg-dp4v2\""
Sep 16 14:27:29.224927 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.224904    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"]
Sep 16 14:27:29.284591 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.284557    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-credential-socket\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.284768 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.284608    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istiod-ca-cert\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.284768 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.284728    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-workload-socket\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.284921 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.284773    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-envoy\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.284921 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.284804    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6n99\" (UniqueName: \"kubernetes.io/projected/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-kube-api-access-m6n99\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.284921 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.284857    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-token\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.284921 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.284883    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-workload-certs\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.285104 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.284949    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-data\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.285104 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.284991    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-podinfo\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.385682 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.385645    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-token\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.385682 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.385682    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-workload-certs\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.385963 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.385703    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-data\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.385963 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.385730    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-podinfo\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.385963 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.385779    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-credential-socket\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.385963 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.385823    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istiod-ca-cert\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.385963 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.385923    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-workload-socket\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.386219 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.385967    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-envoy\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.386219 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.385993    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6n99\" (UniqueName: \"kubernetes.io/projected/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-kube-api-access-m6n99\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.386219 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.386193    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-workload-certs\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.386353 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.386237    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-credential-socket\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.386353 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.386294    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-data\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.386353 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.386340    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-workload-socket\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.386648 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.386628    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istiod-ca-cert\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.388188 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.388168    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-envoy\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.388557 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.388541    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-podinfo\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.394368 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.394344    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-token\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.394524 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.394507    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6n99\" (UniqueName: \"kubernetes.io/projected/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-kube-api-access-m6n99\") pod \"e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") " pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.519620 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.519578    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:29.827417 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.827390    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"]
Sep 16 14:27:29.829370 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:27:29.829348    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1f1ccf_ca52_4e62_93a9_40534b5e678d.slice/crio-cf9bad0faf79cded371e84d4ca93e1604d20aeddff850381fe4c3450dc297097 WatchSource:0}: Error finding container cf9bad0faf79cded371e84d4ca93e1604d20aeddff850381fe4c3450dc297097: Status 404 returned error can't find the container with id cf9bad0faf79cded371e84d4ca93e1604d20aeddff850381fe4c3450dc297097
Sep 16 14:27:29.832344 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.832295    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:29.832461 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.832391    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:29.832461 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:29.832430    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:30.693485 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:30.693438    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw" event={"ID":"ec1f1ccf-ca52-4e62-93a9-40534b5e678d","Type":"ContainerStarted","Data":"7a22735f46c89527db2bc2b0405b3093a4d0ad7ebf0ca749163c994bbf0e8de9"}
Sep 16 14:27:30.693672 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:30.693497    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw" event={"ID":"ec1f1ccf-ca52-4e62-93a9-40534b5e678d","Type":"ContainerStarted","Data":"cf9bad0faf79cded371e84d4ca93e1604d20aeddff850381fe4c3450dc297097"}
Sep 16 14:27:30.720233 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:30.720152    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw" podStartSLOduration=1.7200967729999999 podStartE2EDuration="1.720096773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:30.719972313 +0000 UTC m=+801.266714270" watchObservedRunningTime="2026-09-16 14:27:30.720096773 +0000 UTC m=+801.266838726"
Sep 16 14:27:31.519973 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:31.519937    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:31.524718 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:31.524694    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:31.698423 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:31.698389    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:31.699439 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:31.699420    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:27:36.663089 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:36.663048    2553 patch_prober.go:28] interesting pod/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 14:27:36.663450 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:36.663125    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4" podUID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 14:27:38.679002 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:38.678914    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"
Sep 16 14:27:38.739719 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:38.739682    2553 generic.go:356] "Generic (PLEG): container finished" podID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerID="44ed5742430b80f8a168537b51fd601584769d8ed7bf271313de6a0a9fc20a23" exitCode=0
Sep 16 14:27:38.739863 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:38.739735    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4" event={"ID":"e15beb83-9f62-4765-9efe-74a0be02cbfe","Type":"ContainerDied","Data":"44ed5742430b80f8a168537b51fd601584769d8ed7bf271313de6a0a9fc20a23"}
Sep 16 14:27:38.740325 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:38.740303    2553 scope.go:117] "RemoveContainer" containerID="44ed5742430b80f8a168537b51fd601584769d8ed7bf271313de6a0a9fc20a23"
Sep 16 14:27:38.741622 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:38.741591    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:38.741691 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:38.741656    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:38.741739 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:38.741691    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:39.747076 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:39.747042    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4" event={"ID":"e15beb83-9f62-4765-9efe-74a0be02cbfe","Type":"ContainerStarted","Data":"8619d11cfcc69701967d9edec253c07fae7819c5f4e72b17214ec9c49d3c6ee0"}
Sep 16 14:27:40.230454 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:40.230423    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:40.230659 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:40.230640    2553 patch_prober.go:28] interesting pod/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 16 14:27:40.230712 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:40.230683    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4" podUID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 16 14:27:41.230955 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:41.230918    2553 patch_prober.go:28] interesting pod/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 16 14:27:41.231332 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:41.230972    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4" podUID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 16 14:27:42.019833 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.019800    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"]
Sep 16 14:27:42.022298 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.022274    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.025319 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.025300    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d0b89a-openshift-default-dockercfg-fbnvk\""
Sep 16 14:27:42.040914 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.040876    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"]
Sep 16 14:27:42.127970 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.127932    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-istio-envoy\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.128156 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.128001    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-credential-socket\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.128156 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.128093    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr9fq\" (UniqueName: \"kubernetes.io/projected/37282c70-7f7e-43f4-9bbd-448859955468-kube-api-access-vr9fq\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.128241 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.128166    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-workload-certs\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.128241 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.128229    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-istio-data\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.128303 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.128284    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37282c70-7f7e-43f4-9bbd-448859955468-istio-token\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.128378 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.128356    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37282c70-7f7e-43f4-9bbd-448859955468-istio-podinfo\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.128378 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.128382    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-workload-socket\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.128555 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.128420    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37282c70-7f7e-43f4-9bbd-448859955468-istiod-ca-cert\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.229668 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.229631    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-credential-socket\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.229861 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.229685    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr9fq\" (UniqueName: \"kubernetes.io/projected/37282c70-7f7e-43f4-9bbd-448859955468-kube-api-access-vr9fq\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.229861 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.229809    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-workload-certs\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.229861 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.229851    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-istio-data\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.230080 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.229886    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37282c70-7f7e-43f4-9bbd-448859955468-istio-token\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.230080 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.229963    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37282c70-7f7e-43f4-9bbd-448859955468-istio-podinfo\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.230080 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.230012    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-workload-socket\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.230080 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.230040    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-credential-socket\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.230080 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.230047    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37282c70-7f7e-43f4-9bbd-448859955468-istiod-ca-cert\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.230334 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.230184    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-istio-data\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.230334 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.230197    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-istio-envoy\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.230410 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.230355    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-workload-socket\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.230538 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.230512    2553 patch_prober.go:28] interesting pod/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 16 14:27:42.230538 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.230532    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-workload-certs\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.230714 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.230563    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4" podUID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 16 14:27:42.231085 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.231059    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37282c70-7f7e-43f4-9bbd-448859955468-istiod-ca-cert\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.232596 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.232571    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-istio-envoy\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.232790 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.232775    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37282c70-7f7e-43f4-9bbd-448859955468-istio-podinfo\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.255565 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.255533    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr9fq\" (UniqueName: \"kubernetes.io/projected/37282c70-7f7e-43f4-9bbd-448859955468-kube-api-access-vr9fq\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.265835 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.265804    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37282c70-7f7e-43f4-9bbd-448859955468-istio-token\") pod \"e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") " pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.334246 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.334156    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:42.657076 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.656982    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"]
Sep 16 14:27:42.686940 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.686232    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:42.686940 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.686320    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:42.686940 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.686360    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:42.774645 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.774603    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r" event={"ID":"37282c70-7f7e-43f4-9bbd-448859955468","Type":"ContainerStarted","Data":"a5011758fb67b36fb471f59fca185d2c77d0b4c235f689a0c7729f8cc9dd8fd5"}
Sep 16 14:27:42.774771 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.774678    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r" event={"ID":"37282c70-7f7e-43f4-9bbd-448859955468","Type":"ContainerStarted","Data":"55068d86656a3bc30a223e47b422382de508999ef5919c3257ab0e00cba0dc8c"}
Sep 16 14:27:42.800304 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:42.800243    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r" podStartSLOduration=1.800220193 podStartE2EDuration="1.800220193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:42.797770657 +0000 UTC m=+813.344512612" watchObservedRunningTime="2026-09-16 14:27:42.800220193 +0000 UTC m=+813.346962126"
Sep 16 14:27:43.230675 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:43.230617    2553 patch_prober.go:28] interesting pod/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 16 14:27:43.230830 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:43.230700    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4" podUID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 16 14:27:43.335001 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:43.334942    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:43.340402 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:43.340375    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:43.783124 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:43.783088    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:43.784865 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:43.784846    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:27:44.230471 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:44.230436    2553 patch_prober.go:28] interesting pod/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 16 14:27:44.230643 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:44.230503    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4" podUID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 16 14:27:44.719084 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:44.719001    2553 patch_prober.go:28] interesting pod/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 14:27:44.719465 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:44.719084    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq" podUID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 14:27:45.230588 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:45.230554    2553 patch_prober.go:28] interesting pod/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 16 14:27:45.230752 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:45.230610    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4" podUID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 16 14:27:46.235524 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:46.235498    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:46.236014 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:46.235921    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:46.236866 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:46.236846    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:27:46.799863 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:46.799833    2553 generic.go:356] "Generic (PLEG): container finished" podID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" containerID="a1f6397dafe5d11e42e899ae9c5a95d7ee60f882184f77f35ac9c76af0f65c27" exitCode=0
Sep 16 14:27:46.800054 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:46.799921    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq" event={"ID":"9d64442e-52ea-46fe-a65e-6cbfeda11c7f","Type":"ContainerDied","Data":"a1f6397dafe5d11e42e899ae9c5a95d7ee60f882184f77f35ac9c76af0f65c27"}
Sep 16 14:27:46.800410 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:46.800391    2553 scope.go:117] "RemoveContainer" containerID="a1f6397dafe5d11e42e899ae9c5a95d7ee60f882184f77f35ac9c76af0f65c27"
Sep 16 14:27:46.801550 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:46.801522    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:46.801658 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:46.801584    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:46.801658 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:46.801620    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:27:47.452051 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:47.452012    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld"]
Sep 16 14:27:47.455977 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:47.455948    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld"
Sep 16 14:27:47.458467 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:47.458441    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:27:47.468686 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:47.468649    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld"]
Sep 16 14:27:47.489773 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:47.489745    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b54e8fe-2494-4d30-a021-6cf0f3f8169b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld\" (UID: \"7b54e8fe-2494-4d30-a021-6cf0f3f8169b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld"
Sep 16 14:27:47.590776 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:47.590738    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b54e8fe-2494-4d30-a021-6cf0f3f8169b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld\" (UID: \"7b54e8fe-2494-4d30-a021-6cf0f3f8169b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld"
Sep 16 14:27:47.591528 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:47.591507    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b54e8fe-2494-4d30-a021-6cf0f3f8169b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld\" (UID: \"7b54e8fe-2494-4d30-a021-6cf0f3f8169b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld"
Sep 16 14:27:47.770431 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:47.770394    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld"
Sep 16 14:27:47.808940 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:47.808904    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq" event={"ID":"9d64442e-52ea-46fe-a65e-6cbfeda11c7f","Type":"ContainerStarted","Data":"2a2b60885e6fa389555cdb05466d283d2ff607cb29cfb2fc0cd81a937f8c4d0a"}
Sep 16 14:27:48.040602 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:48.040573    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld"]
Sep 16 14:27:48.042114 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:27:48.042087    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b54e8fe_2494_4d30_a021_6cf0f3f8169b.slice/crio-e4dc900ab00c12f1da7c02294a0da2ad00a67e71b61b34f39b4c02e4af838e5c WatchSource:0}: Error finding container e4dc900ab00c12f1da7c02294a0da2ad00a67e71b61b34f39b4c02e4af838e5c: Status 404 returned error can't find the container with id e4dc900ab00c12f1da7c02294a0da2ad00a67e71b61b34f39b4c02e4af838e5c
Sep 16 14:27:48.151004 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:48.150970    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:48.151400 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:48.151366    2553 patch_prober.go:28] interesting pod/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 14:27:48.151529 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:48.151486    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq" podUID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 14:27:48.815866 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:48.815823    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld" event={"ID":"7b54e8fe-2494-4d30-a021-6cf0f3f8169b","Type":"ContainerStarted","Data":"e4dc900ab00c12f1da7c02294a0da2ad00a67e71b61b34f39b4c02e4af838e5c"}
Sep 16 14:27:49.150629 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:49.150546    2553 patch_prober.go:28] interesting pod/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 14:27:49.150787 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:49.150621    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq" podUID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 14:27:50.150469 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:50.150431    2553 patch_prober.go:28] interesting pod/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 14:27:50.151034 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:50.150503    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq" podUID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 14:27:51.151211 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:51.151166    2553 patch_prober.go:28] interesting pod/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 14:27:51.151698 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:51.151248    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq" podUID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 14:27:51.837038 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:51.837004    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld" event={"ID":"7b54e8fe-2494-4d30-a021-6cf0f3f8169b","Type":"ContainerStarted","Data":"20b9583d0cd96b18586ea3dc3f252701868eca38c48d4d47ff3c838157044256"}
Sep 16 14:27:51.861195 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:51.861119    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld" podStartSLOduration=1.814867233 podStartE2EDuration="4.861097807s" totalImagesPullingTime="3.046230574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:51.857730244 +0000 UTC m=+822.404472176" watchObservedRunningTime="2026-09-16 14:27:51.861097807 +0000 UTC m=+822.407839744"
Sep 16 14:27:52.150446 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:52.150413    2553 patch_prober.go:28] interesting pod/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 14:27:52.150605 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:52.150465    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq" podUID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 14:27:52.860168 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:52.860127    2553 generic.go:356] "Generic (PLEG): container finished" podID="7b54e8fe-2494-4d30-a021-6cf0f3f8169b" containerID="20b9583d0cd96b18586ea3dc3f252701868eca38c48d4d47ff3c838157044256" exitCode=0
Sep 16 14:27:52.860691 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:52.860211    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld" event={"ID":"7b54e8fe-2494-4d30-a021-6cf0f3f8169b","Type":"ContainerDied","Data":"20b9583d0cd96b18586ea3dc3f252701868eca38c48d4d47ff3c838157044256"}
Sep 16 14:27:53.151261 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:53.151174    2553 patch_prober.go:28] interesting pod/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 16 14:27:53.151559 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:53.151528    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq" podUID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 16 14:27:54.046132 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:54.046107    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld"
Sep 16 14:27:54.175968 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:54.175332    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b54e8fe-2494-4d30-a021-6cf0f3f8169b-maas-api-service-ca\") pod \"7b54e8fe-2494-4d30-a021-6cf0f3f8169b\" (UID: \"7b54e8fe-2494-4d30-a021-6cf0f3f8169b\") "
Sep 16 14:27:54.176139 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:54.176002    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b54e8fe-2494-4d30-a021-6cf0f3f8169b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b54e8fe-2494-4d30-a021-6cf0f3f8169b" (UID: "7b54e8fe-2494-4d30-a021-6cf0f3f8169b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:27:54.179331 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:54.178359    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:54.179527 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:54.179507    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:54.180769 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:54.180724    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:27:54.279687 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:54.279650    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b54e8fe-2494-4d30-a021-6cf0f3f8169b-maas-api-service-ca\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:27:54.879090 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:54.879057    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld"
Sep 16 14:27:54.882016 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:54.881981    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld" event={"ID":"7b54e8fe-2494-4d30-a021-6cf0f3f8169b","Type":"ContainerDied","Data":"e4dc900ab00c12f1da7c02294a0da2ad00a67e71b61b34f39b4c02e4af838e5c"}
Sep 16 14:27:54.882155 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:54.882069    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4dc900ab00c12f1da7c02294a0da2ad00a67e71b61b34f39b4c02e4af838e5c"
Sep 16 14:27:57.755235 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.755199    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"]
Sep 16 14:27:57.755853 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.755830    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b54e8fe-2494-4d30-a021-6cf0f3f8169b" containerName="revoke-keys"
Sep 16 14:27:57.755985 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.755855    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b54e8fe-2494-4d30-a021-6cf0f3f8169b" containerName="revoke-keys"
Sep 16 14:27:57.756044 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.755988    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b54e8fe-2494-4d30-a021-6cf0f3f8169b" containerName="revoke-keys"
Sep 16 14:27:57.758122 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.758103    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.772221 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.772201    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5d484d\"/\"simulator-w4-5d484d-kserve-self-signed-certs\""
Sep 16 14:27:57.772318 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.772303    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5d484d\"/\"openshift-service-ca.crt\""
Sep 16 14:27:57.772514 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.772500    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5d484d\"/\"default-dockercfg-vdgm5\""
Sep 16 14:27:57.772617 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.772601    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5d484d\"/\"kube-root-ca.crt\""
Sep 16 14:27:57.800072 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.800038    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"]
Sep 16 14:27:57.816565 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.816537    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-home\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.816565 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.816570    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-tmp-dir\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.816804 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.816703    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsqkv\" (UniqueName: \"kubernetes.io/projected/d68b8afe-5f69-468c-950e-264b03834708-kube-api-access-hsqkv\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.816804 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.816791    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-model-cache\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.816944 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.816816    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d68b8afe-5f69-468c-950e-264b03834708-tls-certs\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.816944 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.816927    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-dshm\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.921806 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.921767    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-model-cache\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.922166 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.922125    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d68b8afe-5f69-468c-950e-264b03834708-tls-certs\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.922297 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.922219    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-dshm\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.922297 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.922263    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-home\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.922297 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.922281    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-tmp-dir\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.922469 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.922386    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsqkv\" (UniqueName: \"kubernetes.io/projected/d68b8afe-5f69-468c-950e-264b03834708-kube-api-access-hsqkv\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.922540 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.922510    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-model-cache\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.922942 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.922873    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-tmp-dir\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.923117 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.923098    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-home\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.925246 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.925222    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-dshm\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.925642 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.925619    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d68b8afe-5f69-468c-950e-264b03834708-tls-certs\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:57.930768 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:57.930739    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsqkv\" (UniqueName: \"kubernetes.io/projected/d68b8afe-5f69-468c-950e-264b03834708-kube-api-access-hsqkv\") pod \"simulator-w4-5d484d-kserve-6879bf5f4-f2w94\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") " pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:58.070668 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:58.070587    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:27:58.269913 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:58.269846    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld"]
Sep 16 14:27:58.274462 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:58.274432    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1e4514fd-wm2ld"]
Sep 16 14:27:58.363987 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:58.363962    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"]
Sep 16 14:27:58.365819 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:27:58.365793    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd68b8afe_5f69_468c_950e_264b03834708.slice/crio-fafa674fac603a78d123d8ff9810394b34f9f7d0dcc15c4ddf0e7a75b3b9ec3e WatchSource:0}: Error finding container fafa674fac603a78d123d8ff9810394b34f9f7d0dcc15c4ddf0e7a75b3b9ec3e: Status 404 returned error can't find the container with id fafa674fac603a78d123d8ff9810394b34f9f7d0dcc15c4ddf0e7a75b3b9ec3e
Sep 16 14:27:58.422217 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:58.422184    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"]
Sep 16 14:27:58.422841 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:58.422783    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw" podUID="ec1f1ccf-ca52-4e62-93a9-40534b5e678d" containerName="istio-proxy" containerID="cri-o://7a22735f46c89527db2bc2b0405b3093a4d0ad7ebf0ca749163c994bbf0e8de9" gracePeriod=30
Sep 16 14:27:58.903796 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:58.903759    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94" event={"ID":"d68b8afe-5f69-468c-950e-264b03834708","Type":"ContainerStarted","Data":"1c09d736ba8edcea13cc1250e452e1156e346e285c5a6661aec265cad47f4583"}
Sep 16 14:27:58.903796 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:58.903808    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94" event={"ID":"d68b8afe-5f69-468c-950e-264b03834708","Type":"ContainerStarted","Data":"fafa674fac603a78d123d8ff9810394b34f9f7d0dcc15c4ddf0e7a75b3b9ec3e"}
Sep 16 14:27:58.949959 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:58.949904    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94" podStartSLOduration=1.949872141 podStartE2EDuration="1.949872141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:58.948132525 +0000 UTC m=+829.494874448" watchObservedRunningTime="2026-09-16 14:27:58.949872141 +0000 UTC m=+829.496614072"
Sep 16 14:27:59.479850 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.479812    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"]
Sep 16 14:27:59.483547 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.483521    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.486225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.486199    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7cfdb2ad-gw-openshift-default-dockercfg-thfmk\""
Sep 16 14:27:59.497576 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.497544    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"]
Sep 16 14:27:59.538706 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.538673    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-token\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.538706 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.538707    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-podinfo\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.538883 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.538763    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-envoy\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.538883 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.538795    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istiod-ca-cert\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.538883 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.538817    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-workload-certs\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.538883 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.538843    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-credential-socket\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.539051 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.538948    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7x6z\" (UniqueName: \"kubernetes.io/projected/86f61c34-a538-4c7e-aa7b-17dbd81b0421-kube-api-access-m7x6z\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.539092 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.539056    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-data\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.539092 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.539080    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-workload-socket\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.640521 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.640476    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-token\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.640710 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.640540    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-podinfo\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.640710 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.640566    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-envoy\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.640710 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.640595    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istiod-ca-cert\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.640710 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.640626    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-workload-certs\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.640710 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.640667    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-credential-socket\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.640710 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.640701    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7x6z\" (UniqueName: \"kubernetes.io/projected/86f61c34-a538-4c7e-aa7b-17dbd81b0421-kube-api-access-m7x6z\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.641036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.640801    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-data\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.641036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.640833    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-workload-socket\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.641180 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.641151    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-workload-certs\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.641252 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.641174    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-credential-socket\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.641302 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.641253    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-workload-socket\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.641357 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.641346    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-data\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.641567 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.641540    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istiod-ca-cert\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.643272 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.643243    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-envoy\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.643379 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.643364    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-podinfo\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.648389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.648368    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-token\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.648660 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.648640    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7x6z\" (UniqueName: \"kubernetes.io/projected/86f61c34-a538-4c7e-aa7b-17dbd81b0421-kube-api-access-m7x6z\") pod \"e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") " pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.797881 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.797800    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:27:59.917598 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:27:59.917567    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b54e8fe-2494-4d30-a021-6cf0f3f8169b" path="/var/lib/kubelet/pods/7b54e8fe-2494-4d30-a021-6cf0f3f8169b/volumes"
Sep 16 14:28:00.091297 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.091271    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"]
Sep 16 14:28:00.120619 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:28:00.120591    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f61c34_a538_4c7e_aa7b_17dbd81b0421.slice/crio-004777b44ac0be51858b9aee1df0c9c94c20f079a5bde395fb55f5c1b9573222 WatchSource:0}: Error finding container 004777b44ac0be51858b9aee1df0c9c94c20f079a5bde395fb55f5c1b9573222: Status 404 returned error can't find the container with id 004777b44ac0be51858b9aee1df0c9c94c20f079a5bde395fb55f5c1b9573222
Sep 16 14:28:00.128717 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.128680    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:28:00.128817 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.128745    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:28:00.128817 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.128778    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:28:00.307203 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.307138    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:28:00.450137 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.450091    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-credential-socket\") pod \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") "
Sep 16 14:28:00.450298 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.450149    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-data\") pod \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") "
Sep 16 14:28:00.450298 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.450191    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istiod-ca-cert\") pod \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") "
Sep 16 14:28:00.450298 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.450232    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-envoy\") pod \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") "
Sep 16 14:28:00.450298 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.450293    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6n99\" (UniqueName: \"kubernetes.io/projected/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-kube-api-access-m6n99\") pod \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") "
Sep 16 14:28:00.450507 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.450319    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-workload-certs\") pod \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") "
Sep 16 14:28:00.450507 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.450358    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-podinfo\") pod \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") "
Sep 16 14:28:00.450507 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.450397    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec1f1ccf-ca52-4e62-93a9-40534b5e678d" (UID: "ec1f1ccf-ca52-4e62-93a9-40534b5e678d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:28:00.450507 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.450422    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-token\") pod \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") "
Sep 16 14:28:00.450507 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.450456    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec1f1ccf-ca52-4e62-93a9-40534b5e678d" (UID: "ec1f1ccf-ca52-4e62-93a9-40534b5e678d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:28:00.450753 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.450521    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-workload-socket\") pod \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\" (UID: \"ec1f1ccf-ca52-4e62-93a9-40534b5e678d\") "
Sep 16 14:28:00.450753 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.450543    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec1f1ccf-ca52-4e62-93a9-40534b5e678d" (UID: "ec1f1ccf-ca52-4e62-93a9-40534b5e678d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:28:00.450753 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.450612    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec1f1ccf-ca52-4e62-93a9-40534b5e678d" (UID: "ec1f1ccf-ca52-4e62-93a9-40534b5e678d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:28:00.450959 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.450825    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec1f1ccf-ca52-4e62-93a9-40534b5e678d" (UID: "ec1f1ccf-ca52-4e62-93a9-40534b5e678d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:28:00.451043 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.451024    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istiod-ca-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:00.451105 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.451047    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-workload-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:00.451105 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.451060    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-workload-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:00.451105 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.451073    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-credential-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:00.451105 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.451087    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-data\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:00.453549 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.453419    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec1f1ccf-ca52-4e62-93a9-40534b5e678d" (UID: "ec1f1ccf-ca52-4e62-93a9-40534b5e678d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:28:00.453549 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.453452    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-kube-api-access-m6n99" (OuterVolumeSpecName: "kube-api-access-m6n99") pod "ec1f1ccf-ca52-4e62-93a9-40534b5e678d" (UID: "ec1f1ccf-ca52-4e62-93a9-40534b5e678d"). InnerVolumeSpecName "kube-api-access-m6n99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:28:00.453549 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.453514    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec1f1ccf-ca52-4e62-93a9-40534b5e678d" (UID: "ec1f1ccf-ca52-4e62-93a9-40534b5e678d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:28:00.453786 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.453553    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec1f1ccf-ca52-4e62-93a9-40534b5e678d" (UID: "ec1f1ccf-ca52-4e62-93a9-40534b5e678d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:28:00.552351 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.552320    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-envoy\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:00.552351 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.552352    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6n99\" (UniqueName: \"kubernetes.io/projected/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-kube-api-access-m6n99\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:00.552566 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.552367    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-podinfo\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:00.552566 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.552382    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1f1ccf-ca52-4e62-93a9-40534b5e678d-istio-token\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:00.917960 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.917919    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h" event={"ID":"86f61c34-a538-4c7e-aa7b-17dbd81b0421","Type":"ContainerStarted","Data":"b93d3a26fcaa6e9d51bfda78f9101efd934d38b2dccf0ae39e054960c54ee675"}
Sep 16 14:28:00.918371 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.917984    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h" event={"ID":"86f61c34-a538-4c7e-aa7b-17dbd81b0421","Type":"ContainerStarted","Data":"004777b44ac0be51858b9aee1df0c9c94c20f079a5bde395fb55f5c1b9573222"}
Sep 16 14:28:00.919334 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.919311    2553 generic.go:356] "Generic (PLEG): container finished" podID="ec1f1ccf-ca52-4e62-93a9-40534b5e678d" containerID="7a22735f46c89527db2bc2b0405b3093a4d0ad7ebf0ca749163c994bbf0e8de9" exitCode=0
Sep 16 14:28:00.919425 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.919351    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"
Sep 16 14:28:00.919425 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.919356    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw" event={"ID":"ec1f1ccf-ca52-4e62-93a9-40534b5e678d","Type":"ContainerDied","Data":"7a22735f46c89527db2bc2b0405b3093a4d0ad7ebf0ca749163c994bbf0e8de9"}
Sep 16 14:28:00.919425 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.919392    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw" event={"ID":"ec1f1ccf-ca52-4e62-93a9-40534b5e678d","Type":"ContainerDied","Data":"cf9bad0faf79cded371e84d4ca93e1604d20aeddff850381fe4c3450dc297097"}
Sep 16 14:28:00.919425 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.919416    2553 scope.go:117] "RemoveContainer" containerID="7a22735f46c89527db2bc2b0405b3093a4d0ad7ebf0ca749163c994bbf0e8de9"
Sep 16 14:28:00.928817 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.928799    2553 scope.go:117] "RemoveContainer" containerID="7a22735f46c89527db2bc2b0405b3093a4d0ad7ebf0ca749163c994bbf0e8de9"
Sep 16 14:28:00.929086 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:28:00.929067    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a22735f46c89527db2bc2b0405b3093a4d0ad7ebf0ca749163c994bbf0e8de9\": container with ID starting with 7a22735f46c89527db2bc2b0405b3093a4d0ad7ebf0ca749163c994bbf0e8de9 not found: ID does not exist" containerID="7a22735f46c89527db2bc2b0405b3093a4d0ad7ebf0ca749163c994bbf0e8de9"
Sep 16 14:28:00.929151 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.929097    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a22735f46c89527db2bc2b0405b3093a4d0ad7ebf0ca749163c994bbf0e8de9"} err="failed to get container status \"7a22735f46c89527db2bc2b0405b3093a4d0ad7ebf0ca749163c994bbf0e8de9\": rpc error: code = NotFound desc = could not find container \"7a22735f46c89527db2bc2b0405b3093a4d0ad7ebf0ca749163c994bbf0e8de9\": container with ID starting with 7a22735f46c89527db2bc2b0405b3093a4d0ad7ebf0ca749163c994bbf0e8de9 not found: ID does not exist"
Sep 16 14:28:00.947462 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.947418    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h" podStartSLOduration=1.947405309 podStartE2EDuration="1.947405309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:28:00.945588311 +0000 UTC m=+831.492330244" watchObservedRunningTime="2026-09-16 14:28:00.947405309 +0000 UTC m=+831.494147259"
Sep 16 14:28:00.963284 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.963262    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"]
Sep 16 14:28:00.967410 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:00.967377    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1e4514fd-openshift-default-58dd9445d4-5s8sw"]
Sep 16 14:28:01.798211 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:01.798180    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:28:01.804132 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:01.804107    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:28:01.918137 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:01.918101    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec1f1ccf-ca52-4e62-93a9-40534b5e678d" path="/var/lib/kubelet/pods/ec1f1ccf-ca52-4e62-93a9-40534b5e678d/volumes"
Sep 16 14:28:01.925671 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:01.925647    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:28:01.926504 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:01.926488    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:28:08.072606 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:08.072027    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:28:08.073087 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:08.072803    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:28:08.091623 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:08.091590    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:28:08.992111 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:08.992080    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:28:13.784619 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:13.784587    2553 patch_prober.go:28] interesting pod/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 16 14:28:13.784987 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:13.784656    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r" podUID="37282c70-7f7e-43f4-9bbd-448859955468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 16 14:28:14.000832 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:14.000798    2553 generic.go:356] "Generic (PLEG): container finished" podID="37282c70-7f7e-43f4-9bbd-448859955468" containerID="a5011758fb67b36fb471f59fca185d2c77d0b4c235f689a0c7729f8cc9dd8fd5" exitCode=0
Sep 16 14:28:14.001019 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:14.000860    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r" event={"ID":"37282c70-7f7e-43f4-9bbd-448859955468","Type":"ContainerDied","Data":"a5011758fb67b36fb471f59fca185d2c77d0b4c235f689a0c7729f8cc9dd8fd5"}
Sep 16 14:28:14.001323 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:14.001308    2553 scope.go:117] "RemoveContainer" containerID="a5011758fb67b36fb471f59fca185d2c77d0b4c235f689a0c7729f8cc9dd8fd5"
Sep 16 14:28:14.002458 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:14.002430    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:28:14.002510 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:14.002495    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:28:14.002544 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:14.002533    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:28:15.013182 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:15.013142    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r" event={"ID":"37282c70-7f7e-43f4-9bbd-448859955468","Type":"ContainerStarted","Data":"0181beca7ed6ab3a49c807fee282b3075bd75df6bddeaa5ca472ba4041770b1d"}
Sep 16 14:28:15.334430 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:15.334338    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:28:15.334562 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:15.334510    2553 patch_prober.go:28] interesting pod/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 16 14:28:15.334604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:15.334564    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r" podUID="37282c70-7f7e-43f4-9bbd-448859955468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 16 14:28:16.334705 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:16.334670    2553 patch_prober.go:28] interesting pod/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 16 14:28:16.335136 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:16.334721    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r" podUID="37282c70-7f7e-43f4-9bbd-448859955468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 16 14:28:17.331814 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.331781    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"]
Sep 16 14:28:17.332272 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.332258    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec1f1ccf-ca52-4e62-93a9-40534b5e678d" containerName="istio-proxy"
Sep 16 14:28:17.332318 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.332273    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec1f1ccf-ca52-4e62-93a9-40534b5e678d" containerName="istio-proxy"
Sep 16 14:28:17.332353 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.332344    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec1f1ccf-ca52-4e62-93a9-40534b5e678d" containerName="istio-proxy"
Sep 16 14:28:17.335124 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.335099    2553 patch_prober.go:28] interesting pod/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 16 14:28:17.335442 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.335156    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r" podUID="37282c70-7f7e-43f4-9bbd-448859955468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 16 14:28:17.336674 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.336648    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.339440 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.339419    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-924916\"/\"kube-root-ca.crt\""
Sep 16 14:28:17.340677 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.340656    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-924916\"/\"openshift-service-ca.crt\""
Sep 16 14:28:17.340767 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.340696    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-924916\"/\"default-dockercfg-xgs7f\""
Sep 16 14:28:17.340767 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.340719    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-924916\"/\"simulator-w0-924916-kserve-self-signed-certs\""
Sep 16 14:28:17.348622 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.348600    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"]
Sep 16 14:28:17.458524 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.458479    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-home\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.458783 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.458762    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-tmp-dir\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.459014 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.458986    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24zjv\" (UniqueName: \"kubernetes.io/projected/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-kube-api-access-24zjv\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.459193 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.459174    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-model-cache\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.459313 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.459298    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-dshm\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.459438 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.459424    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-tls-certs\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.560440 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.560402    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-home\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.561262 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.561241    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-tmp-dir\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.561724 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.561707    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24zjv\" (UniqueName: \"kubernetes.io/projected/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-kube-api-access-24zjv\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.562190 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.562171    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-model-cache\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.562577 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.562563    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-dshm\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.563033 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.563016    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-tls-certs\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.563217 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.562512    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-model-cache\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.564399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.561163    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-home\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.564522 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.561628    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-tmp-dir\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.566150 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.566130    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-dshm\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.567530 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.567509    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-tls-certs\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.573057 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.573013    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24zjv\" (UniqueName: \"kubernetes.io/projected/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-kube-api-access-24zjv\") pod \"simulator-w0-924916-kserve-7b4b8cc696-mt7q8\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") " pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.647161 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.647066    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:17.969971 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:17.969636    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"]
Sep 16 14:28:18.031525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:18.031500    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8" event={"ID":"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0","Type":"ContainerStarted","Data":"7b8e08e1737bc90228f331a51ca22009fb9473f0a2e9e698f05d4959efc97fcd"}
Sep 16 14:28:18.334927 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:18.334823    2553 patch_prober.go:28] interesting pod/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 16 14:28:18.334927 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:18.334878    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r" podUID="37282c70-7f7e-43f4-9bbd-448859955468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 16 14:28:19.038363 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:19.038329    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8" event={"ID":"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0","Type":"ContainerStarted","Data":"6f063678471b3d8958c482111c200f42af7efaa09054685d20551676f4f8b502"}
Sep 16 14:28:19.059329 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:19.059252    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8" podStartSLOduration=2.059228537 podStartE2EDuration="2.059228537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:28:19.056163905 +0000 UTC m=+849.602905839" watchObservedRunningTime="2026-09-16 14:28:19.059228537 +0000 UTC m=+849.605970471"
Sep 16 14:28:19.335571 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:19.335484    2553 patch_prober.go:28] interesting pod/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 16 14:28:19.335571 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:19.335540    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r" podUID="37282c70-7f7e-43f4-9bbd-448859955468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 16 14:28:20.335222 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:20.335181    2553 patch_prober.go:28] interesting pod/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 16 14:28:20.335745 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:20.335243    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r" podUID="37282c70-7f7e-43f4-9bbd-448859955468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 16 14:28:21.338646 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:21.338614    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:28:21.339026 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:21.338917    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:28:21.339704 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:21.339686    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:28:27.087485 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.087447    2553 generic.go:356] "Generic (PLEG): container finished" podID="86f61c34-a538-4c7e-aa7b-17dbd81b0421" containerID="b93d3a26fcaa6e9d51bfda78f9101efd934d38b2dccf0ae39e054960c54ee675" exitCode=0
Sep 16 14:28:27.087909 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.087528    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h" event={"ID":"86f61c34-a538-4c7e-aa7b-17dbd81b0421","Type":"ContainerDied","Data":"b93d3a26fcaa6e9d51bfda78f9101efd934d38b2dccf0ae39e054960c54ee675"}
Sep 16 14:28:27.088093 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.088076    2553 scope.go:117] "RemoveContainer" containerID="b93d3a26fcaa6e9d51bfda78f9101efd934d38b2dccf0ae39e054960c54ee675"
Sep 16 14:28:27.089280 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.089255    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:28:27.089357 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.089321    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:28:27.089357 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.089352    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:28:27.478279 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.478241    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq"]
Sep 16 14:28:27.482722 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.482694    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq"
Sep 16 14:28:27.487429 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.487379    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:28:27.495395 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.495367    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq"]
Sep 16 14:28:27.585961 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.580058    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq\" (UID: \"02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq"
Sep 16 14:28:27.647903 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.647863    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:27.648040 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.647944    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:27.687313 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.687274    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:27.687500 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.687394    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq\" (UID: \"02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq"
Sep 16 14:28:27.688007 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.687989    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq\" (UID: \"02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq"
Sep 16 14:28:27.796778 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:27.796698    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq"
Sep 16 14:28:28.071932 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:28.071799    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq"]
Sep 16 14:28:28.084102 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:28:28.084060    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ea1a4b_ccf6_4b7d_bdd3_3a424afeb0dd.slice/crio-c1f83fb34a6305adb437aaf2401db4d726e444c987ca201334ee4d6f184da1c5 WatchSource:0}: Error finding container c1f83fb34a6305adb437aaf2401db4d726e444c987ca201334ee4d6f184da1c5: Status 404 returned error can't find the container with id c1f83fb34a6305adb437aaf2401db4d726e444c987ca201334ee4d6f184da1c5
Sep 16 14:28:28.096596 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:28.096569    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h" event={"ID":"86f61c34-a538-4c7e-aa7b-17dbd81b0421","Type":"ContainerStarted","Data":"a1c2f51d258f3354e4c0f2ca2f324fb1d94519908c5f25b7a34837c4ffc30c42"}
Sep 16 14:28:28.097787 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:28.097761    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq" event={"ID":"02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd","Type":"ContainerStarted","Data":"c1f83fb34a6305adb437aaf2401db4d726e444c987ca201334ee4d6f184da1c5"}
Sep 16 14:28:28.109748 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:28.109719    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:28:28.798689 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:28.798659    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:28:28.800109 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:28.800012    2553 patch_prober.go:28] interesting pod/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 16 14:28:28.800109 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:28.800068    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h" podUID="86f61c34-a538-4c7e-aa7b-17dbd81b0421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 16 14:28:29.105619 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:29.105590    2553 generic.go:356] "Generic (PLEG): container finished" podID="02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd" containerID="e178f90e60707bce24099f115c035c5f0eeb922d4cc7c3aefd89f2556925b374" exitCode=0
Sep 16 14:28:29.105996 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:29.105679    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq" event={"ID":"02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd","Type":"ContainerDied","Data":"e178f90e60707bce24099f115c035c5f0eeb922d4cc7c3aefd89f2556925b374"}
Sep 16 14:28:29.799018 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:29.798988    2553 patch_prober.go:28] interesting pod/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 16 14:28:29.799180 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:29.799041    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h" podUID="86f61c34-a538-4c7e-aa7b-17dbd81b0421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 16 14:28:30.300931 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:30.300907    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq"
Sep 16 14:28:30.415031 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:30.414941    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd-maas-api-service-ca\") pod \"02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd\" (UID: \"02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd\") "
Sep 16 14:28:30.415248 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:30.415226    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd" (UID: "02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:28:30.415429 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:30.415323    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd-maas-api-service-ca\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:30.798832 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:30.798798    2553 patch_prober.go:28] interesting pod/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 16 14:28:30.799097 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:30.798853    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h" podUID="86f61c34-a538-4c7e-aa7b-17dbd81b0421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 16 14:28:31.116684 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:31.116590    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq" event={"ID":"02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd","Type":"ContainerDied","Data":"c1f83fb34a6305adb437aaf2401db4d726e444c987ca201334ee4d6f184da1c5"}
Sep 16 14:28:31.116684 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:31.116647    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1f83fb34a6305adb437aaf2401db4d726e444c987ca201334ee4d6f184da1c5"
Sep 16 14:28:31.116684 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:31.116599    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq"
Sep 16 14:28:31.799189 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:31.798843    2553 patch_prober.go:28] interesting pod/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 16 14:28:31.799189 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:31.798914    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h" podUID="86f61c34-a538-4c7e-aa7b-17dbd81b0421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 16 14:28:31.925855 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:31.925830    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:28:32.798693 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:32.798639    2553 patch_prober.go:28] interesting pod/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 16 14:28:32.798863 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:32.798712    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h" podUID="86f61c34-a538-4c7e-aa7b-17dbd81b0421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 16 14:28:33.803540 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:33.803511    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:28:33.804479 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:33.804460    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:28:34.424692 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:34.424661    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq"]
Sep 16 14:28:34.426760 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:34.426741    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7cfdb2ad-g4krq"]
Sep 16 14:28:34.587791 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:34.587761    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"]
Sep 16 14:28:35.137163 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:35.137122    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h" podUID="86f61c34-a538-4c7e-aa7b-17dbd81b0421" containerName="istio-proxy" containerID="cri-o://a1c2f51d258f3354e4c0f2ca2f324fb1d94519908c5f25b7a34837c4ffc30c42" gracePeriod=30
Sep 16 14:28:35.918315 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:35.918282    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd" path="/var/lib/kubelet/pods/02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd/volumes"
Sep 16 14:28:40.413534 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.413466    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:28:40.519609 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.519583    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-token\") pod \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") "
Sep 16 14:28:40.519802 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.519617    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-podinfo\") pod \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") "
Sep 16 14:28:40.519802 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.519645    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-workload-certs\") pod \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") "
Sep 16 14:28:40.519802 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.519663    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-credential-socket\") pod \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") "
Sep 16 14:28:40.519802 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.519714    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-data\") pod \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") "
Sep 16 14:28:40.519802 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.519735    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7x6z\" (UniqueName: \"kubernetes.io/projected/86f61c34-a538-4c7e-aa7b-17dbd81b0421-kube-api-access-m7x6z\") pod \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") "
Sep 16 14:28:40.519802 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.519757    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-envoy\") pod \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") "
Sep 16 14:28:40.520151 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.519809    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-workload-socket\") pod \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") "
Sep 16 14:28:40.520151 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.519836    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istiod-ca-cert\") pod \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\" (UID: \"86f61c34-a538-4c7e-aa7b-17dbd81b0421\") "
Sep 16 14:28:40.520151 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.520006    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "86f61c34-a538-4c7e-aa7b-17dbd81b0421" (UID: "86f61c34-a538-4c7e-aa7b-17dbd81b0421"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:28:40.520151 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.520035    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "86f61c34-a538-4c7e-aa7b-17dbd81b0421" (UID: "86f61c34-a538-4c7e-aa7b-17dbd81b0421"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:28:40.520151 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.520050    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-data" (OuterVolumeSpecName: "istio-data") pod "86f61c34-a538-4c7e-aa7b-17dbd81b0421" (UID: "86f61c34-a538-4c7e-aa7b-17dbd81b0421"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:28:40.520468 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.520263    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "86f61c34-a538-4c7e-aa7b-17dbd81b0421" (UID: "86f61c34-a538-4c7e-aa7b-17dbd81b0421"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:28:40.520468 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.520303    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "86f61c34-a538-4c7e-aa7b-17dbd81b0421" (UID: "86f61c34-a538-4c7e-aa7b-17dbd81b0421"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:28:40.520468 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.520325    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-workload-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:40.520468 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.520345    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-credential-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:40.520468 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.520358    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-data\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:40.521818 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.521790    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-token" (OuterVolumeSpecName: "istio-token") pod "86f61c34-a538-4c7e-aa7b-17dbd81b0421" (UID: "86f61c34-a538-4c7e-aa7b-17dbd81b0421"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:28:40.521953 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.521871    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "86f61c34-a538-4c7e-aa7b-17dbd81b0421" (UID: "86f61c34-a538-4c7e-aa7b-17dbd81b0421"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:28:40.522205 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.522184    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "86f61c34-a538-4c7e-aa7b-17dbd81b0421" (UID: "86f61c34-a538-4c7e-aa7b-17dbd81b0421"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:28:40.522291 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.522200    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f61c34-a538-4c7e-aa7b-17dbd81b0421-kube-api-access-m7x6z" (OuterVolumeSpecName: "kube-api-access-m7x6z") pod "86f61c34-a538-4c7e-aa7b-17dbd81b0421" (UID: "86f61c34-a538-4c7e-aa7b-17dbd81b0421"). InnerVolumeSpecName "kube-api-access-m7x6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:28:40.620878 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.620843    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7x6z\" (UniqueName: \"kubernetes.io/projected/86f61c34-a538-4c7e-aa7b-17dbd81b0421-kube-api-access-m7x6z\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:40.620878 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.620873    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-envoy\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:40.620878 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.620884    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f61c34-a538-4c7e-aa7b-17dbd81b0421-workload-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:40.621117 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.620911    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istiod-ca-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:40.621117 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.620920    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-token\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:40.621117 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:40.620927    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86f61c34-a538-4c7e-aa7b-17dbd81b0421-istio-podinfo\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:28:41.174299 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.174266    2553 generic.go:356] "Generic (PLEG): container finished" podID="86f61c34-a538-4c7e-aa7b-17dbd81b0421" containerID="a1c2f51d258f3354e4c0f2ca2f324fb1d94519908c5f25b7a34837c4ffc30c42" exitCode=0
Sep 16 14:28:41.174467 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.174311    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"
Sep 16 14:28:41.174467 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.174354    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h" event={"ID":"86f61c34-a538-4c7e-aa7b-17dbd81b0421","Type":"ContainerDied","Data":"a1c2f51d258f3354e4c0f2ca2f324fb1d94519908c5f25b7a34837c4ffc30c42"}
Sep 16 14:28:41.174467 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.174415    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h" event={"ID":"86f61c34-a538-4c7e-aa7b-17dbd81b0421","Type":"ContainerDied","Data":"004777b44ac0be51858b9aee1df0c9c94c20f079a5bde395fb55f5c1b9573222"}
Sep 16 14:28:41.174467 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.174439    2553 scope.go:117] "RemoveContainer" containerID="a1c2f51d258f3354e4c0f2ca2f324fb1d94519908c5f25b7a34837c4ffc30c42"
Sep 16 14:28:41.185286 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.185272    2553 scope.go:117] "RemoveContainer" containerID="b93d3a26fcaa6e9d51bfda78f9101efd934d38b2dccf0ae39e054960c54ee675"
Sep 16 14:28:41.193419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.193406    2553 scope.go:117] "RemoveContainer" containerID="a1c2f51d258f3354e4c0f2ca2f324fb1d94519908c5f25b7a34837c4ffc30c42"
Sep 16 14:28:41.193667 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:28:41.193650    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c2f51d258f3354e4c0f2ca2f324fb1d94519908c5f25b7a34837c4ffc30c42\": container with ID starting with a1c2f51d258f3354e4c0f2ca2f324fb1d94519908c5f25b7a34837c4ffc30c42 not found: ID does not exist" containerID="a1c2f51d258f3354e4c0f2ca2f324fb1d94519908c5f25b7a34837c4ffc30c42"
Sep 16 14:28:41.193712 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.193675    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c2f51d258f3354e4c0f2ca2f324fb1d94519908c5f25b7a34837c4ffc30c42"} err="failed to get container status \"a1c2f51d258f3354e4c0f2ca2f324fb1d94519908c5f25b7a34837c4ffc30c42\": rpc error: code = NotFound desc = could not find container \"a1c2f51d258f3354e4c0f2ca2f324fb1d94519908c5f25b7a34837c4ffc30c42\": container with ID starting with a1c2f51d258f3354e4c0f2ca2f324fb1d94519908c5f25b7a34837c4ffc30c42 not found: ID does not exist"
Sep 16 14:28:41.193712 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.193689    2553 scope.go:117] "RemoveContainer" containerID="b93d3a26fcaa6e9d51bfda78f9101efd934d38b2dccf0ae39e054960c54ee675"
Sep 16 14:28:41.193969 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:28:41.193947    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b93d3a26fcaa6e9d51bfda78f9101efd934d38b2dccf0ae39e054960c54ee675\": container with ID starting with b93d3a26fcaa6e9d51bfda78f9101efd934d38b2dccf0ae39e054960c54ee675 not found: ID does not exist" containerID="b93d3a26fcaa6e9d51bfda78f9101efd934d38b2dccf0ae39e054960c54ee675"
Sep 16 14:28:41.194011 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.193979    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b93d3a26fcaa6e9d51bfda78f9101efd934d38b2dccf0ae39e054960c54ee675"} err="failed to get container status \"b93d3a26fcaa6e9d51bfda78f9101efd934d38b2dccf0ae39e054960c54ee675\": rpc error: code = NotFound desc = could not find container \"b93d3a26fcaa6e9d51bfda78f9101efd934d38b2dccf0ae39e054960c54ee675\": container with ID starting with b93d3a26fcaa6e9d51bfda78f9101efd934d38b2dccf0ae39e054960c54ee675 not found: ID does not exist"
Sep 16 14:28:41.199703 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.199682    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"]
Sep 16 14:28:41.202743 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.202721    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7cfdb2ad-gw-openshift-default-6dc44b5bfb-fdb9h"]
Sep 16 14:28:41.382804 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.382775    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"]
Sep 16 14:28:41.383339 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.383319    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f61c34-a538-4c7e-aa7b-17dbd81b0421" containerName="istio-proxy"
Sep 16 14:28:41.383339 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.383340    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f61c34-a538-4c7e-aa7b-17dbd81b0421" containerName="istio-proxy"
Sep 16 14:28:41.383472 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.383355    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f61c34-a538-4c7e-aa7b-17dbd81b0421" containerName="istio-proxy"
Sep 16 14:28:41.383472 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.383361    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f61c34-a538-4c7e-aa7b-17dbd81b0421" containerName="istio-proxy"
Sep 16 14:28:41.383472 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.383368    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd" containerName="revoke-keys"
Sep 16 14:28:41.383472 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.383374    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd" containerName="revoke-keys"
Sep 16 14:28:41.383472 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.383467    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f61c34-a538-4c7e-aa7b-17dbd81b0421" containerName="istio-proxy"
Sep 16 14:28:41.383630 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.383476    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="02ea1a4b-ccf6-4b7d-bdd3-3a424afeb0dd" containerName="revoke-keys"
Sep 16 14:28:41.388302 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.388279    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:28:41.391618 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.391594    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5317692d-serving-cert\""
Sep 16 14:28:41.391618 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.391599    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 14:28:41.391763 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.391706    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qjgdq\""
Sep 16 14:28:41.408373 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.408343    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"]
Sep 16 14:28:41.428374 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.428315    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs2m5\" (UniqueName: \"kubernetes.io/projected/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-kube-api-access-zs2m5\") pod \"maas-api-e2e-ait-5317692d-84d8c7454c-skhkq\" (UID: \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:28:41.428640 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.428393    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-maas-api-tls\") pod \"maas-api-e2e-ait-5317692d-84d8c7454c-skhkq\" (UID: \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:28:41.428640 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.428429    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-openshift-service-ca\") pod \"maas-api-e2e-ait-5317692d-84d8c7454c-skhkq\" (UID: \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:28:41.428640 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.428445    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-metrics-certs\") pod \"maas-api-e2e-ait-5317692d-84d8c7454c-skhkq\" (UID: \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:28:41.529388 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.529358    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-maas-api-tls\") pod \"maas-api-e2e-ait-5317692d-84d8c7454c-skhkq\" (UID: \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:28:41.529521 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.529403    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-openshift-service-ca\") pod \"maas-api-e2e-ait-5317692d-84d8c7454c-skhkq\" (UID: \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:28:41.529521 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.529420    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-metrics-certs\") pod \"maas-api-e2e-ait-5317692d-84d8c7454c-skhkq\" (UID: \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:28:41.529521 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.529470    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs2m5\" (UniqueName: \"kubernetes.io/projected/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-kube-api-access-zs2m5\") pod \"maas-api-e2e-ait-5317692d-84d8c7454c-skhkq\" (UID: \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:28:41.530140 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.530121    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-openshift-service-ca\") pod \"maas-api-e2e-ait-5317692d-84d8c7454c-skhkq\" (UID: \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:28:41.531845 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.531820    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-metrics-certs\") pod \"maas-api-e2e-ait-5317692d-84d8c7454c-skhkq\" (UID: \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:28:41.531957 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.531885    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-maas-api-tls\") pod \"maas-api-e2e-ait-5317692d-84d8c7454c-skhkq\" (UID: \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:28:41.537667 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.537644    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs2m5\" (UniqueName: \"kubernetes.io/projected/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-kube-api-access-zs2m5\") pod \"maas-api-e2e-ait-5317692d-84d8c7454c-skhkq\" (UID: \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:28:41.700328 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.700248    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:28:41.922099 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.920598    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86f61c34-a538-4c7e-aa7b-17dbd81b0421" path="/var/lib/kubelet/pods/86f61c34-a538-4c7e-aa7b-17dbd81b0421/volumes"
Sep 16 14:28:41.988350 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:41.988318    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"]
Sep 16 14:28:41.989537 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:28:41.989513    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe565e3_607b_4e0e_8bbf_1622b5bcdbf4.slice/crio-157a0bcd769213d263f7db0b916ad23373ffa019e7c1c6ca693f2000fd578b38 WatchSource:0}: Error finding container 157a0bcd769213d263f7db0b916ad23373ffa019e7c1c6ca693f2000fd578b38: Status 404 returned error can't find the container with id 157a0bcd769213d263f7db0b916ad23373ffa019e7c1c6ca693f2000fd578b38
Sep 16 14:28:42.180934 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:42.180900    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq" event={"ID":"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4","Type":"ContainerStarted","Data":"157a0bcd769213d263f7db0b916ad23373ffa019e7c1c6ca693f2000fd578b38"}
Sep 16 14:28:45.203411 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:45.203375    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq" event={"ID":"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4","Type":"ContainerStarted","Data":"75a35287538c0e66ffe9ea67c459071f1ddba4d167426d02ef8edfd863c7e342"}
Sep 16 14:28:45.203846 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:45.203470    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:28:45.228429 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:45.228385    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq" podStartSLOduration=1.607401001 podStartE2EDuration="4.22837059s" totalImagesPullingTime="2.620969589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:28:45.22661438 +0000 UTC m=+875.773356312" watchObservedRunningTime="2026-09-16 14:28:45.22837059 +0000 UTC m=+875.775112520"
Sep 16 14:28:51.215070 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:28:51.215038    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:29:09.854435 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:09.854370    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:29:09.855190 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:09.855163    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:29:21.940211 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:21.940177    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"]
Sep 16 14:29:21.940636 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:21.940373    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq" podUID="9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4" containerName="maas-api" containerID="cri-o://75a35287538c0e66ffe9ea67c459071f1ddba4d167426d02ef8edfd863c7e342" gracePeriod=30
Sep 16 14:29:22.195696 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.195620    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:29:22.245295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.245252    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs2m5\" (UniqueName: \"kubernetes.io/projected/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-kube-api-access-zs2m5\") pod \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\" (UID: \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\") "
Sep 16 14:29:22.245480 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.245390    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-metrics-certs\") pod \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\" (UID: \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\") "
Sep 16 14:29:22.245480 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.245442    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-openshift-service-ca\") pod \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\" (UID: \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\") "
Sep 16 14:29:22.245480 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.245468    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-maas-api-tls\") pod \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\" (UID: \"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4\") "
Sep 16 14:29:22.246176 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.246137    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4" (UID: "9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:29:22.247541 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.246587    2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-openshift-service-ca\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:29:22.257193 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.257055    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4" (UID: "9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:29:22.257193 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.257157    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4" (UID: "9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:29:22.265165 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.264999    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-kube-api-access-zs2m5" (OuterVolumeSpecName: "kube-api-access-zs2m5") pod "9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4" (UID: "9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4"). InnerVolumeSpecName "kube-api-access-zs2m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:29:22.347554 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.347520    2553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-metrics-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:29:22.347554 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.347551    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-maas-api-tls\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:29:22.347743 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.347564    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs2m5\" (UniqueName: \"kubernetes.io/projected/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4-kube-api-access-zs2m5\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:29:22.414603 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.414571    2553 generic.go:356] "Generic (PLEG): container finished" podID="9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4" containerID="75a35287538c0e66ffe9ea67c459071f1ddba4d167426d02ef8edfd863c7e342" exitCode=0
Sep 16 14:29:22.414768 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.414614    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"
Sep 16 14:29:22.414768 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.414663    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq" event={"ID":"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4","Type":"ContainerDied","Data":"75a35287538c0e66ffe9ea67c459071f1ddba4d167426d02ef8edfd863c7e342"}
Sep 16 14:29:22.414768 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.414712    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq" event={"ID":"9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4","Type":"ContainerDied","Data":"157a0bcd769213d263f7db0b916ad23373ffa019e7c1c6ca693f2000fd578b38"}
Sep 16 14:29:22.414768 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.414740    2553 scope.go:117] "RemoveContainer" containerID="75a35287538c0e66ffe9ea67c459071f1ddba4d167426d02ef8edfd863c7e342"
Sep 16 14:29:22.425554 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.425533    2553 scope.go:117] "RemoveContainer" containerID="75a35287538c0e66ffe9ea67c459071f1ddba4d167426d02ef8edfd863c7e342"
Sep 16 14:29:22.425919 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:29:22.425883    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a35287538c0e66ffe9ea67c459071f1ddba4d167426d02ef8edfd863c7e342\": container with ID starting with 75a35287538c0e66ffe9ea67c459071f1ddba4d167426d02ef8edfd863c7e342 not found: ID does not exist" containerID="75a35287538c0e66ffe9ea67c459071f1ddba4d167426d02ef8edfd863c7e342"
Sep 16 14:29:22.426028 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.425927    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a35287538c0e66ffe9ea67c459071f1ddba4d167426d02ef8edfd863c7e342"} err="failed to get container status \"75a35287538c0e66ffe9ea67c459071f1ddba4d167426d02ef8edfd863c7e342\": rpc error: code = NotFound desc = could not find container \"75a35287538c0e66ffe9ea67c459071f1ddba4d167426d02ef8edfd863c7e342\": container with ID starting with 75a35287538c0e66ffe9ea67c459071f1ddba4d167426d02ef8edfd863c7e342 not found: ID does not exist"
Sep 16 14:29:22.458306 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.458223    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"]
Sep 16 14:29:22.484174 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:22.484143    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5317692d-84d8c7454c-skhkq"]
Sep 16 14:29:23.442199 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.442163    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"]
Sep 16 14:29:23.442781 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.442762    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4" containerName="maas-api"
Sep 16 14:29:23.442915 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.442784    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4" containerName="maas-api"
Sep 16 14:29:23.442982 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.442938    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f61c34-a538-4c7e-aa7b-17dbd81b0421" containerName="istio-proxy"
Sep 16 14:29:23.442982 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.442956    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4" containerName="maas-api"
Sep 16 14:29:23.445922 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.445901    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.453871 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.453853    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-094dab\"/\"kube-root-ca.crt\""
Sep 16 14:29:23.455108 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.455089    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-094dab\"/\"openshift-service-ca.crt\""
Sep 16 14:29:23.455226 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.455153    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-094dab\"/\"test-model-094dab-kserve-self-signed-certs\""
Sep 16 14:29:23.455226 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.455184    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-094dab\"/\"default-dockercfg-5n8pj\""
Sep 16 14:29:23.474241 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.474213    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"]
Sep 16 14:29:23.559327 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.559289    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8fe988-1786-4c9f-833e-a0960ebd7849-tls-certs\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.559495 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.559382    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-model-cache\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.559495 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.559442    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-home\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.559495 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.559493    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-dshm\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.559631 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.559513    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-tmp-dir\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.559631 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.559558    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvbf\" (UniqueName: \"kubernetes.io/projected/fe8fe988-1786-4c9f-833e-a0960ebd7849-kube-api-access-5cvbf\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.660885 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.660851    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-dshm\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.660885 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.660884    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-tmp-dir\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.661134 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.660937    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvbf\" (UniqueName: \"kubernetes.io/projected/fe8fe988-1786-4c9f-833e-a0960ebd7849-kube-api-access-5cvbf\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.661134 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.660985    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8fe988-1786-4c9f-833e-a0960ebd7849-tls-certs\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.661134 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.661046    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-model-cache\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.661134 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.661071    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-home\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.661400 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.661374    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-tmp-dir\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.661491 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.661466    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-model-cache\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.661491 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.661484    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-home\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.663345 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.663318    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-dshm\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.664055 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.664031    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8fe988-1786-4c9f-833e-a0960ebd7849-tls-certs\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.675255 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.675074    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvbf\" (UniqueName: \"kubernetes.io/projected/fe8fe988-1786-4c9f-833e-a0960ebd7849-kube-api-access-5cvbf\") pod \"test-model-094dab-kserve-7cf6c47449-lkh7c\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") " pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.756324 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.756288    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:23.917538 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:23.917506    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4" path="/var/lib/kubelet/pods/9fe565e3-607b-4e0e-8bbf-1622b5bcdbf4/volumes"
Sep 16 14:29:24.033847 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.033827    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"]
Sep 16 14:29:24.035739 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:29:24.035712    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe8fe988_1786_4c9f_833e_a0960ebd7849.slice/crio-abc23e2b861dab063c90893c59bca8c8e425668a685cfefc33e54eeac67c4204 WatchSource:0}: Error finding container abc23e2b861dab063c90893c59bca8c8e425668a685cfefc33e54eeac67c4204: Status 404 returned error can't find the container with id abc23e2b861dab063c90893c59bca8c8e425668a685cfefc33e54eeac67c4204
Sep 16 14:29:24.429635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.429534    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c" event={"ID":"fe8fe988-1786-4c9f-833e-a0960ebd7849","Type":"ContainerStarted","Data":"df037354ec85934b70685c33d7eaf3f829f46a89bb622b83d6d60f8ac55a5b68"}
Sep 16 14:29:24.429635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.429580    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c" event={"ID":"fe8fe988-1786-4c9f-833e-a0960ebd7849","Type":"ContainerStarted","Data":"abc23e2b861dab063c90893c59bca8c8e425668a685cfefc33e54eeac67c4204"}
Sep 16 14:29:24.452190 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.452137    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c" podStartSLOduration=1.4521204509999999 podStartE2EDuration="1.452120451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:29:24.450233807 +0000 UTC m=+914.996975730" watchObservedRunningTime="2026-09-16 14:29:24.452120451 +0000 UTC m=+914.998862381"
Sep 16 14:29:24.533823 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.533796    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"]
Sep 16 14:29:24.536627 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.536610    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.539752 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.539731    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-dfce3e96-openshift-default-dockercfg-dc5g8\""
Sep 16 14:29:24.585264 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.585240    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"]
Sep 16 14:29:24.671985 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.671939    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-credential-socket\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.672165 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.672007    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-token\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.672165 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.672042    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-podinfo\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.672285 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.672224    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-data\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.672285 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.672265    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-workload-certs\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.672385 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.672289    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-envoy\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.672385 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.672355    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cn55\" (UniqueName: \"kubernetes.io/projected/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-kube-api-access-2cn55\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.672481 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.672420    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-workload-socket\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.672481 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.672441    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istiod-ca-cert\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.773506 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.773477    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-data\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.773681 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.773514    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-workload-certs\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.773681 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.773531    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-envoy\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.773681 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.773550    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cn55\" (UniqueName: \"kubernetes.io/projected/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-kube-api-access-2cn55\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.773681 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.773576    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-workload-socket\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.773681 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.773601    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istiod-ca-cert\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.773681 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.773644    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-credential-socket\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.773681 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.773674    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-token\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.774090 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.773762    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-podinfo\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.774090 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.773937    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-workload-certs\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.774090 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.774010    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-data\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.774207 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.774108    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-workload-socket\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.774207 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.774159    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-credential-socket\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.774425 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.774402    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istiod-ca-cert\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.775941 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.775923    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-envoy\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.776494 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.776472    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-podinfo\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.781869 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.781792    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cn55\" (UniqueName: \"kubernetes.io/projected/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-kube-api-access-2cn55\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.782124 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.782103    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-token\") pod \"e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") " pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:24.846738 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:24.846713    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:25.134544 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:25.134510    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"]
Sep 16 14:29:25.136072 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:29:25.136043    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfb17f8_f4b7_479a_92a7_2d5adef6516c.slice/crio-fd7e5b9d9292b9c7a2c3b903f7c11e8244dfc66517add7355162cae847b6de9a WatchSource:0}: Error finding container fd7e5b9d9292b9c7a2c3b903f7c11e8244dfc66517add7355162cae847b6de9a: Status 404 returned error can't find the container with id fd7e5b9d9292b9c7a2c3b903f7c11e8244dfc66517add7355162cae847b6de9a
Sep 16 14:29:25.138632 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:25.138601    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:29:25.138740 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:25.138670    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:29:25.138740 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:25.138711    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:29:25.438127 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:25.438045    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k" event={"ID":"2dfb17f8-f4b7-479a-92a7-2d5adef6516c","Type":"ContainerStarted","Data":"894dee15ba14591ff65acf03cad019c028268288100b720985f6b24222a6d962"}
Sep 16 14:29:25.438127 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:25.438096    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k" event={"ID":"2dfb17f8-f4b7-479a-92a7-2d5adef6516c","Type":"ContainerStarted","Data":"fd7e5b9d9292b9c7a2c3b903f7c11e8244dfc66517add7355162cae847b6de9a"}
Sep 16 14:29:25.466119 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:25.466072    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k" podStartSLOduration=1.466053575 podStartE2EDuration="1.466053575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:29:25.463409448 +0000 UTC m=+916.010151379" watchObservedRunningTime="2026-09-16 14:29:25.466053575 +0000 UTC m=+916.012795506"
Sep 16 14:29:25.847582 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:25.847542    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:26.852208 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:26.852176    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:27.448592 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:27.448564    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:27.449528 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:27.449510    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:28.781197 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:28.781165    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"]
Sep 16 14:29:28.784736 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:28.784712    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:28.788322 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:28.788295    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qjgdq\""
Sep 16 14:29:28.788480 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:28.788318    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 14:29:28.788480 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:28.788317    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-dfce3e96-serving-cert\""
Sep 16 14:29:28.798518 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:28.798490    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"]
Sep 16 14:29:28.916273 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:28.916240    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2be57674-2741-4bd0-b02a-4f464a8e325a-maas-api-tls\") pod \"maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc\" (UID: \"2be57674-2741-4bd0-b02a-4f464a8e325a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:28.916435 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:28.916281    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be57674-2741-4bd0-b02a-4f464a8e325a-metrics-certs\") pod \"maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc\" (UID: \"2be57674-2741-4bd0-b02a-4f464a8e325a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:28.916435 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:28.916410    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2be57674-2741-4bd0-b02a-4f464a8e325a-openshift-service-ca\") pod \"maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc\" (UID: \"2be57674-2741-4bd0-b02a-4f464a8e325a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:28.916516 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:28.916496    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkl69\" (UniqueName: \"kubernetes.io/projected/2be57674-2741-4bd0-b02a-4f464a8e325a-kube-api-access-xkl69\") pod \"maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc\" (UID: \"2be57674-2741-4bd0-b02a-4f464a8e325a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:29.017390 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:29.017362    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkl69\" (UniqueName: \"kubernetes.io/projected/2be57674-2741-4bd0-b02a-4f464a8e325a-kube-api-access-xkl69\") pod \"maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc\" (UID: \"2be57674-2741-4bd0-b02a-4f464a8e325a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:29.017556 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:29.017410    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2be57674-2741-4bd0-b02a-4f464a8e325a-maas-api-tls\") pod \"maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc\" (UID: \"2be57674-2741-4bd0-b02a-4f464a8e325a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:29.017556 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:29.017439    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be57674-2741-4bd0-b02a-4f464a8e325a-metrics-certs\") pod \"maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc\" (UID: \"2be57674-2741-4bd0-b02a-4f464a8e325a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:29.017556 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:29.017534    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2be57674-2741-4bd0-b02a-4f464a8e325a-openshift-service-ca\") pod \"maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc\" (UID: \"2be57674-2741-4bd0-b02a-4f464a8e325a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:29.018278 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:29.018255    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2be57674-2741-4bd0-b02a-4f464a8e325a-openshift-service-ca\") pod \"maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc\" (UID: \"2be57674-2741-4bd0-b02a-4f464a8e325a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:29.019935 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:29.019909    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2be57674-2741-4bd0-b02a-4f464a8e325a-maas-api-tls\") pod \"maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc\" (UID: \"2be57674-2741-4bd0-b02a-4f464a8e325a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:29.020025 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:29.019953    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be57674-2741-4bd0-b02a-4f464a8e325a-metrics-certs\") pod \"maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc\" (UID: \"2be57674-2741-4bd0-b02a-4f464a8e325a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:29.030445 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:29.030395    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkl69\" (UniqueName: \"kubernetes.io/projected/2be57674-2741-4bd0-b02a-4f464a8e325a-kube-api-access-xkl69\") pod \"maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc\" (UID: \"2be57674-2741-4bd0-b02a-4f464a8e325a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:29.099669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:29.099598    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:29.392152 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:29.392125    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"]
Sep 16 14:29:29.394110 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:29:29.394085    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be57674_2741_4bd0_b02a_4f464a8e325a.slice/crio-d423ca7e94c6954d005d8868720a299471f77ab480bb091ff89bf0e9ec1e00cd WatchSource:0}: Error finding container d423ca7e94c6954d005d8868720a299471f77ab480bb091ff89bf0e9ec1e00cd: Status 404 returned error can't find the container with id d423ca7e94c6954d005d8868720a299471f77ab480bb091ff89bf0e9ec1e00cd
Sep 16 14:29:29.462248 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:29.462213    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc" event={"ID":"2be57674-2741-4bd0-b02a-4f464a8e325a","Type":"ContainerStarted","Data":"d423ca7e94c6954d005d8868720a299471f77ab480bb091ff89bf0e9ec1e00cd"}
Sep 16 14:29:30.469616 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:30.469581    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc" event={"ID":"2be57674-2741-4bd0-b02a-4f464a8e325a","Type":"ContainerStarted","Data":"0f86331e117968a97acb96e1b07e1cec1eaeb08776054213eeff53701059d3c9"}
Sep 16 14:29:30.470065 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:30.469687    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:30.497869 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:30.497824    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc" podStartSLOduration=2.000585051 podStartE2EDuration="2.497810708s" totalImagesPullingTime="497.225657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:29:30.496856606 +0000 UTC m=+921.043598542" watchObservedRunningTime="2026-09-16 14:29:30.497810708 +0000 UTC m=+921.044552652"
Sep 16 14:29:33.756845 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:33.756753    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:33.757334 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:33.756835    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:33.769673 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:33.769651    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:34.504649 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:34.504616    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:29:36.482141 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:36.482114    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:54.621824 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:54.621788    2553 generic.go:356] "Generic (PLEG): container finished" podID="2dfb17f8-f4b7-479a-92a7-2d5adef6516c" containerID="894dee15ba14591ff65acf03cad019c028268288100b720985f6b24222a6d962" exitCode=0
Sep 16 14:29:54.622273 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:54.621861    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k" event={"ID":"2dfb17f8-f4b7-479a-92a7-2d5adef6516c","Type":"ContainerDied","Data":"894dee15ba14591ff65acf03cad019c028268288100b720985f6b24222a6d962"}
Sep 16 14:29:54.622318 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:54.622302    2553 scope.go:117] "RemoveContainer" containerID="894dee15ba14591ff65acf03cad019c028268288100b720985f6b24222a6d962"
Sep 16 14:29:54.623528 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:54.623500    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:29:54.623582 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:54.623571    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:29:54.630395 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:54.624106    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:29:55.629731 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:55.629699    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k" event={"ID":"2dfb17f8-f4b7-479a-92a7-2d5adef6516c","Type":"ContainerStarted","Data":"aa19944ed638f29e266c2fcd193f1a910f03b8e628ce982c379052d9fb091d5d"}
Sep 16 14:29:55.847277 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:55.847241    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:55.848398 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:55.848373    2553 patch_prober.go:28] interesting pod/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 16 14:29:55.848507 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:55.848424    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k" podUID="2dfb17f8-f4b7-479a-92a7-2d5adef6516c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 16 14:29:56.847426 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:56.847395    2553 patch_prober.go:28] interesting pod/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 16 14:29:56.847808 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:56.847452    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k" podUID="2dfb17f8-f4b7-479a-92a7-2d5adef6516c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 16 14:29:57.449088 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:57.449050    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:29:57.848192 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:57.848156    2553 patch_prober.go:28] interesting pod/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 16 14:29:57.848553 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:57.848217    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k" podUID="2dfb17f8-f4b7-479a-92a7-2d5adef6516c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 16 14:29:58.784936 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:58.784702    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"]
Sep 16 14:29:58.785140 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:58.785107    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc" podUID="2be57674-2741-4bd0-b02a-4f464a8e325a" containerName="maas-api" containerID="cri-o://0f86331e117968a97acb96e1b07e1cec1eaeb08776054213eeff53701059d3c9" gracePeriod=30
Sep 16 14:29:58.849298 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:58.848885    2553 patch_prober.go:28] interesting pod/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 16 14:29:58.849298 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:58.848957    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k" podUID="2dfb17f8-f4b7-479a-92a7-2d5adef6516c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 16 14:29:59.097941 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.097857    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:59.180911 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.178054    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"]
Sep 16 14:29:59.180911 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.178578    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k" podUID="2dfb17f8-f4b7-479a-92a7-2d5adef6516c" containerName="istio-proxy" containerID="cri-o://aa19944ed638f29e266c2fcd193f1a910f03b8e628ce982c379052d9fb091d5d" gracePeriod=30
Sep 16 14:29:59.243326 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.243295    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2be57674-2741-4bd0-b02a-4f464a8e325a-openshift-service-ca\") pod \"2be57674-2741-4bd0-b02a-4f464a8e325a\" (UID: \"2be57674-2741-4bd0-b02a-4f464a8e325a\") "
Sep 16 14:29:59.243665 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.243647    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2be57674-2741-4bd0-b02a-4f464a8e325a-maas-api-tls\") pod \"2be57674-2741-4bd0-b02a-4f464a8e325a\" (UID: \"2be57674-2741-4bd0-b02a-4f464a8e325a\") "
Sep 16 14:29:59.244239 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.243842    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be57674-2741-4bd0-b02a-4f464a8e325a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2be57674-2741-4bd0-b02a-4f464a8e325a" (UID: "2be57674-2741-4bd0-b02a-4f464a8e325a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:29:59.244454 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.244434    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkl69\" (UniqueName: \"kubernetes.io/projected/2be57674-2741-4bd0-b02a-4f464a8e325a-kube-api-access-xkl69\") pod \"2be57674-2741-4bd0-b02a-4f464a8e325a\" (UID: \"2be57674-2741-4bd0-b02a-4f464a8e325a\") "
Sep 16 14:29:59.244636 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.244622    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be57674-2741-4bd0-b02a-4f464a8e325a-metrics-certs\") pod \"2be57674-2741-4bd0-b02a-4f464a8e325a\" (UID: \"2be57674-2741-4bd0-b02a-4f464a8e325a\") "
Sep 16 14:29:59.245488 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.245469    2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2be57674-2741-4bd0-b02a-4f464a8e325a-openshift-service-ca\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:29:59.246123 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.246101    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2be57674-2741-4bd0-b02a-4f464a8e325a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2be57674-2741-4bd0-b02a-4f464a8e325a" (UID: "2be57674-2741-4bd0-b02a-4f464a8e325a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:29:59.247379 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.247353    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be57674-2741-4bd0-b02a-4f464a8e325a-kube-api-access-xkl69" (OuterVolumeSpecName: "kube-api-access-xkl69") pod "2be57674-2741-4bd0-b02a-4f464a8e325a" (UID: "2be57674-2741-4bd0-b02a-4f464a8e325a"). InnerVolumeSpecName "kube-api-access-xkl69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:29:59.247632 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.247601    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2be57674-2741-4bd0-b02a-4f464a8e325a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2be57674-2741-4bd0-b02a-4f464a8e325a" (UID: "2be57674-2741-4bd0-b02a-4f464a8e325a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:29:59.346632 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.346532    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkl69\" (UniqueName: \"kubernetes.io/projected/2be57674-2741-4bd0-b02a-4f464a8e325a-kube-api-access-xkl69\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:29:59.346632 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.346558    2553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2be57674-2741-4bd0-b02a-4f464a8e325a-metrics-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:29:59.346632 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.346568    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2be57674-2741-4bd0-b02a-4f464a8e325a-maas-api-tls\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:29:59.658093 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.657993    2553 generic.go:356] "Generic (PLEG): container finished" podID="2be57674-2741-4bd0-b02a-4f464a8e325a" containerID="0f86331e117968a97acb96e1b07e1cec1eaeb08776054213eeff53701059d3c9" exitCode=0
Sep 16 14:29:59.658288 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.658171    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc" event={"ID":"2be57674-2741-4bd0-b02a-4f464a8e325a","Type":"ContainerDied","Data":"0f86331e117968a97acb96e1b07e1cec1eaeb08776054213eeff53701059d3c9"}
Sep 16 14:29:59.658288 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.658224    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc" event={"ID":"2be57674-2741-4bd0-b02a-4f464a8e325a","Type":"ContainerDied","Data":"d423ca7e94c6954d005d8868720a299471f77ab480bb091ff89bf0e9ec1e00cd"}
Sep 16 14:29:59.658288 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.658255    2553 scope.go:117] "RemoveContainer" containerID="0f86331e117968a97acb96e1b07e1cec1eaeb08776054213eeff53701059d3c9"
Sep 16 14:29:59.658464 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.658417    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"
Sep 16 14:29:59.679276 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.678416    2553 scope.go:117] "RemoveContainer" containerID="0f86331e117968a97acb96e1b07e1cec1eaeb08776054213eeff53701059d3c9"
Sep 16 14:29:59.679276 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:29:59.678820    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f86331e117968a97acb96e1b07e1cec1eaeb08776054213eeff53701059d3c9\": container with ID starting with 0f86331e117968a97acb96e1b07e1cec1eaeb08776054213eeff53701059d3c9 not found: ID does not exist" containerID="0f86331e117968a97acb96e1b07e1cec1eaeb08776054213eeff53701059d3c9"
Sep 16 14:29:59.679276 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.678859    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f86331e117968a97acb96e1b07e1cec1eaeb08776054213eeff53701059d3c9"} err="failed to get container status \"0f86331e117968a97acb96e1b07e1cec1eaeb08776054213eeff53701059d3c9\": rpc error: code = NotFound desc = could not find container \"0f86331e117968a97acb96e1b07e1cec1eaeb08776054213eeff53701059d3c9\": container with ID starting with 0f86331e117968a97acb96e1b07e1cec1eaeb08776054213eeff53701059d3c9 not found: ID does not exist"
Sep 16 14:29:59.704019 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.703993    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"]
Sep 16 14:29:59.709259 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.709228    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfce3e96-8cf76df66-jbcmc"]
Sep 16 14:29:59.919701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:29:59.919618    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2be57674-2741-4bd0-b02a-4f464a8e325a" path="/var/lib/kubelet/pods/2be57674-2741-4bd0-b02a-4f464a8e325a/volumes"
Sep 16 14:30:00.178661 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:00.178563    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826150-z4pdg"]
Sep 16 14:30:00.179272 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:00.179248    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2be57674-2741-4bd0-b02a-4f464a8e325a" containerName="maas-api"
Sep 16 14:30:00.179272 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:00.179273    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2be57674-2741-4bd0-b02a-4f464a8e325a" containerName="maas-api"
Sep 16 14:30:00.179458 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:00.179415    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2be57674-2741-4bd0-b02a-4f464a8e325a" containerName="maas-api"
Sep 16 14:30:00.182696 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:00.182669    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826150-z4pdg"
Sep 16 14:30:00.188443 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:00.188422    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:30:00.196742 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:00.196724    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826150-z4pdg"
Sep 16 14:30:00.204132 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:00.204112    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-094dab-29826150-krtdr"]
Sep 16 14:30:00.206533 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:00.206513    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-094dab-29826150-krtdr"
Sep 16 14:30:00.222677 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:00.222540    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826150-z4pdg"]
Sep 16 14:30:00.238714 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:00.238690    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-094dab-29826150-krtdr"
Sep 16 14:30:00.246540 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:00.246500    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-094dab-29826150-krtdr"]
Sep 16 14:30:00.500709 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:00.500681    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826150-z4pdg"]
Sep 16 14:30:00.502210 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:30:00.502186    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd519550_0e8b_4fe0_8010_777d0b5859aa.slice/crio-66de70b361480368e92fd02a47366567c1f0d74284073b85f028337ea7301300 WatchSource:0}: Error finding container 66de70b361480368e92fd02a47366567c1f0d74284073b85f028337ea7301300: Status 404 returned error can't find the container with id 66de70b361480368e92fd02a47366567c1f0d74284073b85f028337ea7301300
Sep 16 14:30:00.581730 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:00.581006    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-094dab-29826150-krtdr"]
Sep 16 14:30:00.586137 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:30:00.586110    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667b4ffc_3ea6_48b9_b92a_a46441e004fb.slice/crio-3edbeacff43373f141e0d5ced7691816b40a0eb71e0a8cb9179b48737184b602 WatchSource:0}: Error finding container 3edbeacff43373f141e0d5ced7691816b40a0eb71e0a8cb9179b48737184b602: Status 404 returned error can't find the container with id 3edbeacff43373f141e0d5ced7691816b40a0eb71e0a8cb9179b48737184b602
Sep 16 14:30:00.672457 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:00.672415    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826150-z4pdg" event={"ID":"fd519550-0e8b-4fe0-8010-777d0b5859aa","Type":"ContainerStarted","Data":"66de70b361480368e92fd02a47366567c1f0d74284073b85f028337ea7301300"}
Sep 16 14:30:00.678058 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:00.678019    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-094dab-29826150-krtdr" event={"ID":"667b4ffc-3ea6-48b9-b92a-a46441e004fb","Type":"ContainerStarted","Data":"3edbeacff43373f141e0d5ced7691816b40a0eb71e0a8cb9179b48737184b602"}
Sep 16 14:30:01.686512 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:01.686486    2553 generic.go:356] "Generic (PLEG): container finished" podID="fd519550-0e8b-4fe0-8010-777d0b5859aa" containerID="c028972106c0fc420f5872614c49fb7abca1648b355bd2664d065c91b9ab7f7c" exitCode=0
Sep 16 14:30:01.686862 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:01.686572    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826150-z4pdg" event={"ID":"fd519550-0e8b-4fe0-8010-777d0b5859aa","Type":"ContainerDied","Data":"c028972106c0fc420f5872614c49fb7abca1648b355bd2664d065c91b9ab7f7c"}
Sep 16 14:30:01.688046 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:01.688025    2553 generic.go:356] "Generic (PLEG): container finished" podID="667b4ffc-3ea6-48b9-b92a-a46441e004fb" containerID="94118db0eefa4f94f151c90e53a57c71da023c37e60ee6fe420e29b5e98bacc3" exitCode=0
Sep 16 14:30:01.688146 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:01.688082    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-094dab-29826150-krtdr" event={"ID":"667b4ffc-3ea6-48b9-b92a-a46441e004fb","Type":"ContainerDied","Data":"94118db0eefa4f94f151c90e53a57c71da023c37e60ee6fe420e29b5e98bacc3"}
Sep 16 14:30:02.851320 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:02.851289    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826150-z4pdg"
Sep 16 14:30:02.900824 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:02.900802    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-094dab-29826150-krtdr"
Sep 16 14:30:03.704238 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:03.704207    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826150-z4pdg"
Sep 16 14:30:03.704238 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:03.704225    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826150-z4pdg" event={"ID":"fd519550-0e8b-4fe0-8010-777d0b5859aa","Type":"ContainerDied","Data":"66de70b361480368e92fd02a47366567c1f0d74284073b85f028337ea7301300"}
Sep 16 14:30:03.704440 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:03.704273    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66de70b361480368e92fd02a47366567c1f0d74284073b85f028337ea7301300"
Sep 16 14:30:03.705543 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:03.705524    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-094dab-29826150-krtdr" event={"ID":"667b4ffc-3ea6-48b9-b92a-a46441e004fb","Type":"ContainerDied","Data":"3edbeacff43373f141e0d5ced7691816b40a0eb71e0a8cb9179b48737184b602"}
Sep 16 14:30:03.705650 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:03.705550    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-094dab-29826150-krtdr"
Sep 16 14:30:03.705650 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:03.705557    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3edbeacff43373f141e0d5ced7691816b40a0eb71e0a8cb9179b48737184b602"
Sep 16 14:30:06.337606 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.337583    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:30:06.431011 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.430928    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cn55\" (UniqueName: \"kubernetes.io/projected/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-kube-api-access-2cn55\") pod \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") "
Sep 16 14:30:06.431011 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.430963    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-podinfo\") pod \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") "
Sep 16 14:30:06.431011 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.430999    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-workload-socket\") pod \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") "
Sep 16 14:30:06.431266 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.431022    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-workload-certs\") pod \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") "
Sep 16 14:30:06.431266 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.431105    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-token\") pod \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") "
Sep 16 14:30:06.431266 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.431127    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-envoy\") pod \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") "
Sep 16 14:30:06.431266 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.431147    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istiod-ca-cert\") pod \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") "
Sep 16 14:30:06.431266 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.431180    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-credential-socket\") pod \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") "
Sep 16 14:30:06.431266 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.431214    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-data\") pod \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\" (UID: \"2dfb17f8-f4b7-479a-92a7-2d5adef6516c\") "
Sep 16 14:30:06.431540 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.431393    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2dfb17f8-f4b7-479a-92a7-2d5adef6516c" (UID: "2dfb17f8-f4b7-479a-92a7-2d5adef6516c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:30:06.431658 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.431626    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2dfb17f8-f4b7-479a-92a7-2d5adef6516c" (UID: "2dfb17f8-f4b7-479a-92a7-2d5adef6516c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:30:06.431823 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.431672    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2dfb17f8-f4b7-479a-92a7-2d5adef6516c" (UID: "2dfb17f8-f4b7-479a-92a7-2d5adef6516c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:30:06.431823 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.431691    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-workload-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:06.431823 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.431731    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2dfb17f8-f4b7-479a-92a7-2d5adef6516c" (UID: "2dfb17f8-f4b7-479a-92a7-2d5adef6516c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:30:06.432038 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.431828    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-data" (OuterVolumeSpecName: "istio-data") pod "2dfb17f8-f4b7-479a-92a7-2d5adef6516c" (UID: "2dfb17f8-f4b7-479a-92a7-2d5adef6516c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:30:06.433326 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.433281    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2dfb17f8-f4b7-479a-92a7-2d5adef6516c" (UID: "2dfb17f8-f4b7-479a-92a7-2d5adef6516c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:30:06.433326 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.433318    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-token" (OuterVolumeSpecName: "istio-token") pod "2dfb17f8-f4b7-479a-92a7-2d5adef6516c" (UID: "2dfb17f8-f4b7-479a-92a7-2d5adef6516c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:30:06.433502 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.433329    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-kube-api-access-2cn55" (OuterVolumeSpecName: "kube-api-access-2cn55") pod "2dfb17f8-f4b7-479a-92a7-2d5adef6516c" (UID: "2dfb17f8-f4b7-479a-92a7-2d5adef6516c"). InnerVolumeSpecName "kube-api-access-2cn55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:30:06.433502 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.433390    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2dfb17f8-f4b7-479a-92a7-2d5adef6516c" (UID: "2dfb17f8-f4b7-479a-92a7-2d5adef6516c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:30:06.533295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.533268    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-token\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:06.533295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.533290    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-envoy\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:06.533295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.533298    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istiod-ca-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:06.533486 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.533307    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-credential-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:06.533486 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.533315    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-data\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:06.533486 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.533323    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cn55\" (UniqueName: \"kubernetes.io/projected/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-kube-api-access-2cn55\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:06.533486 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.533332    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-istio-podinfo\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:06.533486 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.533342    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dfb17f8-f4b7-479a-92a7-2d5adef6516c-workload-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:06.724508 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.724473    2553 generic.go:356] "Generic (PLEG): container finished" podID="2dfb17f8-f4b7-479a-92a7-2d5adef6516c" containerID="aa19944ed638f29e266c2fcd193f1a910f03b8e628ce982c379052d9fb091d5d" exitCode=0
Sep 16 14:30:06.724671 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.724525    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"
Sep 16 14:30:06.724671 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.724567    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k" event={"ID":"2dfb17f8-f4b7-479a-92a7-2d5adef6516c","Type":"ContainerDied","Data":"aa19944ed638f29e266c2fcd193f1a910f03b8e628ce982c379052d9fb091d5d"}
Sep 16 14:30:06.724671 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.724633    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k" event={"ID":"2dfb17f8-f4b7-479a-92a7-2d5adef6516c","Type":"ContainerDied","Data":"fd7e5b9d9292b9c7a2c3b903f7c11e8244dfc66517add7355162cae847b6de9a"}
Sep 16 14:30:06.724671 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.724661    2553 scope.go:117] "RemoveContainer" containerID="aa19944ed638f29e266c2fcd193f1a910f03b8e628ce982c379052d9fb091d5d"
Sep 16 14:30:06.735138 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.735123    2553 scope.go:117] "RemoveContainer" containerID="894dee15ba14591ff65acf03cad019c028268288100b720985f6b24222a6d962"
Sep 16 14:30:06.743248 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.743232    2553 scope.go:117] "RemoveContainer" containerID="aa19944ed638f29e266c2fcd193f1a910f03b8e628ce982c379052d9fb091d5d"
Sep 16 14:30:06.743531 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:30:06.743512    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa19944ed638f29e266c2fcd193f1a910f03b8e628ce982c379052d9fb091d5d\": container with ID starting with aa19944ed638f29e266c2fcd193f1a910f03b8e628ce982c379052d9fb091d5d not found: ID does not exist" containerID="aa19944ed638f29e266c2fcd193f1a910f03b8e628ce982c379052d9fb091d5d"
Sep 16 14:30:06.743641 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.743538    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa19944ed638f29e266c2fcd193f1a910f03b8e628ce982c379052d9fb091d5d"} err="failed to get container status \"aa19944ed638f29e266c2fcd193f1a910f03b8e628ce982c379052d9fb091d5d\": rpc error: code = NotFound desc = could not find container \"aa19944ed638f29e266c2fcd193f1a910f03b8e628ce982c379052d9fb091d5d\": container with ID starting with aa19944ed638f29e266c2fcd193f1a910f03b8e628ce982c379052d9fb091d5d not found: ID does not exist"
Sep 16 14:30:06.743641 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.743553    2553 scope.go:117] "RemoveContainer" containerID="894dee15ba14591ff65acf03cad019c028268288100b720985f6b24222a6d962"
Sep 16 14:30:06.743807 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:30:06.743791    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"894dee15ba14591ff65acf03cad019c028268288100b720985f6b24222a6d962\": container with ID starting with 894dee15ba14591ff65acf03cad019c028268288100b720985f6b24222a6d962 not found: ID does not exist" containerID="894dee15ba14591ff65acf03cad019c028268288100b720985f6b24222a6d962"
Sep 16 14:30:06.743852 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.743810    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894dee15ba14591ff65acf03cad019c028268288100b720985f6b24222a6d962"} err="failed to get container status \"894dee15ba14591ff65acf03cad019c028268288100b720985f6b24222a6d962\": rpc error: code = NotFound desc = could not find container \"894dee15ba14591ff65acf03cad019c028268288100b720985f6b24222a6d962\": container with ID starting with 894dee15ba14591ff65acf03cad019c028268288100b720985f6b24222a6d962 not found: ID does not exist"
Sep 16 14:30:06.748610 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.748589    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"]
Sep 16 14:30:06.752592 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:06.752565    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-dfce3e96-openshift-default-5f447f79c6-65w6k"]
Sep 16 14:30:07.918211 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:07.918182    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dfb17f8-f4b7-479a-92a7-2d5adef6516c" path="/var/lib/kubelet/pods/2dfb17f8-f4b7-479a-92a7-2d5adef6516c/volumes"
Sep 16 14:30:12.559131 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:12.559094    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"]
Sep 16 14:30:12.559577 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:12.559351    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq" podUID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" containerName="istio-proxy" containerID="cri-o://2a2b60885e6fa389555cdb05466d283d2ff607cb29cfb2fc0cd81a937f8c4d0a" gracePeriod=30
Sep 16 14:30:17.788503 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.788471    2553 generic.go:356] "Generic (PLEG): container finished" podID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" containerID="2a2b60885e6fa389555cdb05466d283d2ff607cb29cfb2fc0cd81a937f8c4d0a" exitCode=0
Sep 16 14:30:17.788952 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.788504    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq" event={"ID":"9d64442e-52ea-46fe-a65e-6cbfeda11c7f","Type":"ContainerDied","Data":"2a2b60885e6fa389555cdb05466d283d2ff607cb29cfb2fc0cd81a937f8c4d0a"}
Sep 16 14:30:17.788952 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.788579    2553 scope.go:117] "RemoveContainer" containerID="a1f6397dafe5d11e42e899ae9c5a95d7ee60f882184f77f35ac9c76af0f65c27"
Sep 16 14:30:17.812443 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.812425    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:30:17.857097 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.857013    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-podinfo\") pod \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") "
Sep 16 14:30:17.857097 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.857079    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-data\") pod \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") "
Sep 16 14:30:17.857318 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.857127    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-workload-socket\") pod \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") "
Sep 16 14:30:17.857318 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.857155    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-workload-certs\") pod \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") "
Sep 16 14:30:17.857318 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.857179    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-credential-socket\") pod \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") "
Sep 16 14:30:17.857318 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.857220    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-token\") pod \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") "
Sep 16 14:30:17.857318 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.857259    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8vsm\" (UniqueName: \"kubernetes.io/projected/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-kube-api-access-c8vsm\") pod \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") "
Sep 16 14:30:17.857579 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.857324    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istiod-ca-cert\") pod \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") "
Sep 16 14:30:17.857579 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.857384    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-envoy\") pod \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\" (UID: \"9d64442e-52ea-46fe-a65e-6cbfeda11c7f\") "
Sep 16 14:30:17.857579 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.857426    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-data" (OuterVolumeSpecName: "istio-data") pod "9d64442e-52ea-46fe-a65e-6cbfeda11c7f" (UID: "9d64442e-52ea-46fe-a65e-6cbfeda11c7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:30:17.857737 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.857681    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9d64442e-52ea-46fe-a65e-6cbfeda11c7f" (UID: "9d64442e-52ea-46fe-a65e-6cbfeda11c7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:30:17.857821 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.857738    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-data\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:17.857877 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.857741    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9d64442e-52ea-46fe-a65e-6cbfeda11c7f" (UID: "9d64442e-52ea-46fe-a65e-6cbfeda11c7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:30:17.857877 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.857825    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9d64442e-52ea-46fe-a65e-6cbfeda11c7f" (UID: "9d64442e-52ea-46fe-a65e-6cbfeda11c7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:30:17.858042 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.858008    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9d64442e-52ea-46fe-a65e-6cbfeda11c7f" (UID: "9d64442e-52ea-46fe-a65e-6cbfeda11c7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:30:17.859668 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.859640    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9d64442e-52ea-46fe-a65e-6cbfeda11c7f" (UID: "9d64442e-52ea-46fe-a65e-6cbfeda11c7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:30:17.860038 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.860011    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9d64442e-52ea-46fe-a65e-6cbfeda11c7f" (UID: "9d64442e-52ea-46fe-a65e-6cbfeda11c7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:30:17.860130 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.860064    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-token" (OuterVolumeSpecName: "istio-token") pod "9d64442e-52ea-46fe-a65e-6cbfeda11c7f" (UID: "9d64442e-52ea-46fe-a65e-6cbfeda11c7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:30:17.860130 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.860078    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-kube-api-access-c8vsm" (OuterVolumeSpecName: "kube-api-access-c8vsm") pod "9d64442e-52ea-46fe-a65e-6cbfeda11c7f" (UID: "9d64442e-52ea-46fe-a65e-6cbfeda11c7f"). InnerVolumeSpecName "kube-api-access-c8vsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:30:17.958832 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.958806    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-podinfo\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:17.958832 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.958831    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-workload-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:17.959012 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.958841    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-workload-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:17.959012 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.958849    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-credential-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:17.959012 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.958858    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-token\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:17.959012 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.958867    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8vsm\" (UniqueName: \"kubernetes.io/projected/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-kube-api-access-c8vsm\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:17.959012 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.958874    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istiod-ca-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:17.959012 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:17.958882    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d64442e-52ea-46fe-a65e-6cbfeda11c7f-istio-envoy\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:30:18.795683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:18.795651    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"
Sep 16 14:30:18.795683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:18.795675    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq" event={"ID":"9d64442e-52ea-46fe-a65e-6cbfeda11c7f","Type":"ContainerDied","Data":"2882944ed518074278c183f076417cef258f0c3290759d41d95638e5b20203bd"}
Sep 16 14:30:18.796201 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:18.795734    2553 scope.go:117] "RemoveContainer" containerID="2a2b60885e6fa389555cdb05466d283d2ff607cb29cfb2fc0cd81a937f8c4d0a"
Sep 16 14:30:18.834903 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:18.834861    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"]
Sep 16 14:30:18.836944 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:18.836919    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-84fdff-openshift-default-6db8f8c5d-xw7wq"]
Sep 16 14:30:19.917579 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:19.917537    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" path="/var/lib/kubelet/pods/9d64442e-52ea-46fe-a65e-6cbfeda11c7f/volumes"
Sep 16 14:30:20.318830 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.318798    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"]
Sep 16 14:30:20.319262 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319249    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" containerName="istio-proxy"
Sep 16 14:30:20.319314 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319263    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" containerName="istio-proxy"
Sep 16 14:30:20.319314 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319280    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" containerName="istio-proxy"
Sep 16 14:30:20.319314 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319286    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" containerName="istio-proxy"
Sep 16 14:30:20.319314 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319301    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dfb17f8-f4b7-479a-92a7-2d5adef6516c" containerName="istio-proxy"
Sep 16 14:30:20.319314 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319307    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dfb17f8-f4b7-479a-92a7-2d5adef6516c" containerName="istio-proxy"
Sep 16 14:30:20.319314 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319315    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667b4ffc-3ea6-48b9-b92a-a46441e004fb" containerName="cleanup"
Sep 16 14:30:20.319514 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319321    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="667b4ffc-3ea6-48b9-b92a-a46441e004fb" containerName="cleanup"
Sep 16 14:30:20.319514 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319330    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd519550-0e8b-4fe0-8010-777d0b5859aa" containerName="cleanup"
Sep 16 14:30:20.319514 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319336    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd519550-0e8b-4fe0-8010-777d0b5859aa" containerName="cleanup"
Sep 16 14:30:20.319514 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319342    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dfb17f8-f4b7-479a-92a7-2d5adef6516c" containerName="istio-proxy"
Sep 16 14:30:20.319514 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319347    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dfb17f8-f4b7-479a-92a7-2d5adef6516c" containerName="istio-proxy"
Sep 16 14:30:20.319514 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319406    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd519550-0e8b-4fe0-8010-777d0b5859aa" containerName="cleanup"
Sep 16 14:30:20.319514 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319412    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dfb17f8-f4b7-479a-92a7-2d5adef6516c" containerName="istio-proxy"
Sep 16 14:30:20.319514 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319418    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="667b4ffc-3ea6-48b9-b92a-a46441e004fb" containerName="cleanup"
Sep 16 14:30:20.319514 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319427    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" containerName="istio-proxy"
Sep 16 14:30:20.319514 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319433    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dfb17f8-f4b7-479a-92a7-2d5adef6516c" containerName="istio-proxy"
Sep 16 14:30:20.319514 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.319440    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d64442e-52ea-46fe-a65e-6cbfeda11c7f" containerName="istio-proxy"
Sep 16 14:30:20.322501 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.322480    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:30:20.325873 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.325852    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-7ce904-serving-cert\""
Sep 16 14:30:20.326128 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.326112    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qjgdq\""
Sep 16 14:30:20.327135 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.327117    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 14:30:20.336700 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.336680    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"]
Sep 16 14:30:20.381200 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.381162    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3977480-40ec-424e-a321-87a26bdb841b-metrics-certs\") pod \"maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:30:20.381386 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.381234    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3977480-40ec-424e-a321-87a26bdb841b-maas-api-tls\") pod \"maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:30:20.381386 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.381307    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnd48\" (UniqueName: \"kubernetes.io/projected/c3977480-40ec-424e-a321-87a26bdb841b-kube-api-access-mnd48\") pod \"maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:30:20.381386 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.381375    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3977480-40ec-424e-a321-87a26bdb841b-openshift-service-ca\") pod \"maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:30:20.482403 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.482365    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3977480-40ec-424e-a321-87a26bdb841b-metrics-certs\") pod \"maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:30:20.482592 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.482423    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3977480-40ec-424e-a321-87a26bdb841b-maas-api-tls\") pod \"maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:30:20.482592 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.482457    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnd48\" (UniqueName: \"kubernetes.io/projected/c3977480-40ec-424e-a321-87a26bdb841b-kube-api-access-mnd48\") pod \"maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:30:20.482718 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.482597    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3977480-40ec-424e-a321-87a26bdb841b-openshift-service-ca\") pod \"maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:30:20.482869 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:30:20.482847    2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-serving-cert: secret "maas-api-e2e-explicit-7ce904-serving-cert" not found
Sep 16 14:30:20.482969 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:30:20.482956    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3977480-40ec-424e-a321-87a26bdb841b-maas-api-tls podName:c3977480-40ec-424e-a321-87a26bdb841b nodeName:}" failed. No retries permitted until 2026-09-16 14:30:20.982933178 +0000 UTC m=+971.529675106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3977480-40ec-424e-a321-87a26bdb841b-maas-api-tls") pod "maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57" (UID: "c3977480-40ec-424e-a321-87a26bdb841b") : secret "maas-api-e2e-explicit-7ce904-serving-cert" not found
Sep 16 14:30:20.483359 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.483342    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3977480-40ec-424e-a321-87a26bdb841b-openshift-service-ca\") pod \"maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:30:20.484848 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.484825    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3977480-40ec-424e-a321-87a26bdb841b-metrics-certs\") pod \"maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:30:20.493237 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.493213    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnd48\" (UniqueName: \"kubernetes.io/projected/c3977480-40ec-424e-a321-87a26bdb841b-kube-api-access-mnd48\") pod \"maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:30:20.988225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.988186    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3977480-40ec-424e-a321-87a26bdb841b-maas-api-tls\") pod \"maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:30:20.990601 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:20.990574    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3977480-40ec-424e-a321-87a26bdb841b-maas-api-tls\") pod \"maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:30:21.232635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:21.232599    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:30:21.513431 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:21.513406    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"]
Sep 16 14:30:21.514976 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:30:21.514947    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3977480_40ec_424e_a321_87a26bdb841b.slice/crio-2b1395a57d0b836f0fa8091eb5b8253da771ebcc4f2272a5506cad7bce8fbb04 WatchSource:0}: Error finding container 2b1395a57d0b836f0fa8091eb5b8253da771ebcc4f2272a5506cad7bce8fbb04: Status 404 returned error can't find the container with id 2b1395a57d0b836f0fa8091eb5b8253da771ebcc4f2272a5506cad7bce8fbb04
Sep 16 14:30:21.516306 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:21.516288    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:30:21.813950 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:21.813834    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57" event={"ID":"c3977480-40ec-424e-a321-87a26bdb841b","Type":"ContainerStarted","Data":"2b1395a57d0b836f0fa8091eb5b8253da771ebcc4f2272a5506cad7bce8fbb04"}
Sep 16 14:30:22.821462 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:22.821424    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57" event={"ID":"c3977480-40ec-424e-a321-87a26bdb841b","Type":"ContainerStarted","Data":"73d36a98bd22808093b1b8bd3acc65a45c10674a7a1561b8d2619d63a0d1775a"}
Sep 16 14:30:22.821965 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:22.821480    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:30:22.840996 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:22.840944    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57" podStartSLOduration=2.141598787 podStartE2EDuration="2.840925141s" totalImagesPullingTime="699.326354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:30:22.839275288 +0000 UTC m=+973.386017220" watchObservedRunningTime="2026-09-16 14:30:22.840925141 +0000 UTC m=+973.387667071"
Sep 16 14:30:28.832421 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:30:28.832390    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:31:50.009753 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.009717    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"]
Sep 16 14:31:50.013625 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.013607    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.016447 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.016422    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d0b89a\"/\"kube-root-ca.crt\""
Sep 16 14:31:50.017583 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.017556    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d0b89a\"/\"test-model-d0b89a-kserve-self-signed-certs\""
Sep 16 14:31:50.017583 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.017572    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d0b89a\"/\"default-dockercfg-vdqk9\""
Sep 16 14:31:50.017802 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.017584    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d0b89a\"/\"openshift-service-ca.crt\""
Sep 16 14:31:50.025038 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.025014    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"]
Sep 16 14:31:50.170899 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.170869    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-model-cache\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.171064 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.170934    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-home\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.171064 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.171016    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-dshm\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.171154 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.171062    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e076d7-0621-4671-a734-984379f27066-tls-certs\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.171154 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.171087    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z5rc\" (UniqueName: \"kubernetes.io/projected/76e076d7-0621-4671-a734-984379f27066-kube-api-access-8z5rc\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.171154 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.171116    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-tmp-dir\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.272712 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.272629    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-model-cache\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.272712 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.272678    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-home\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.272712 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.272709    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-dshm\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.273012 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.272739    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e076d7-0621-4671-a734-984379f27066-tls-certs\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.273012 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.272759    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z5rc\" (UniqueName: \"kubernetes.io/projected/76e076d7-0621-4671-a734-984379f27066-kube-api-access-8z5rc\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.273012 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.272780    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-tmp-dir\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.273175 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.273058    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-model-cache\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.273238 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.273195    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-home\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.273238 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.273222    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-tmp-dir\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.275045 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.275021    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-dshm\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.275294 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.275278    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e076d7-0621-4671-a734-984379f27066-tls-certs\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.283165 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.283145    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z5rc\" (UniqueName: \"kubernetes.io/projected/76e076d7-0621-4671-a734-984379f27066-kube-api-access-8z5rc\") pod \"test-model-d0b89a-kserve-7789cfc549-xtdgd\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") " pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.324003 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.323982    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:31:50.599471 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:50.599446    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"]
Sep 16 14:31:50.601428 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:31:50.601397    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76e076d7_0621_4671_a734_984379f27066.slice/crio-ad1544f537b5a112f632504d6b2c860d272ca06bb269b6cabca3e053e8f6717b WatchSource:0}: Error finding container ad1544f537b5a112f632504d6b2c860d272ca06bb269b6cabca3e053e8f6717b: Status 404 returned error can't find the container with id ad1544f537b5a112f632504d6b2c860d272ca06bb269b6cabca3e053e8f6717b
Sep 16 14:31:51.311058 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:51.311022    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd" event={"ID":"76e076d7-0621-4671-a734-984379f27066","Type":"ContainerStarted","Data":"618df7ea47fe3a36142e6d6e17296102bd05f65caabcb33f46a766b7cb18e7d1"}
Sep 16 14:31:51.311476 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:51.311071    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd" event={"ID":"76e076d7-0621-4671-a734-984379f27066","Type":"ContainerStarted","Data":"ad1544f537b5a112f632504d6b2c860d272ca06bb269b6cabca3e053e8f6717b"}
Sep 16 14:31:51.329510 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:51.329441    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd" podStartSLOduration=2.329426181 podStartE2EDuration="2.329426181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:31:51.328630922 +0000 UTC m=+1061.875372857" watchObservedRunningTime="2026-09-16 14:31:51.329426181 +0000 UTC m=+1061.876168109"
Sep 16 14:31:59.587774 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.587721    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"]
Sep 16 14:31:59.592559 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.592532    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.595276 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.595250    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5e366590-openshift-default-dockercfg-4m6d4\""
Sep 16 14:31:59.605671 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.605640    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"]
Sep 16 14:31:59.769569 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.769533    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w92gd\" (UniqueName: \"kubernetes.io/projected/39b8d440-1120-45f5-995d-68eda0eab96e-kube-api-access-w92gd\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.769741 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.769585    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b8d440-1120-45f5-995d-68eda0eab96e-istio-podinfo\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.769741 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.769646    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b8d440-1120-45f5-995d-68eda0eab96e-istio-token\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.769833 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.769745    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-workload-certs\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.769833 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.769766    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-istio-data\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.769833 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.769795    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b8d440-1120-45f5-995d-68eda0eab96e-istiod-ca-cert\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.770001 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.769928    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-workload-socket\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.770001 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.769966    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-istio-envoy\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.770001 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.769995    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-credential-socket\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.870978 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.870860    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b8d440-1120-45f5-995d-68eda0eab96e-istiod-ca-cert\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.870978 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.870971    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-workload-socket\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.871156 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.871010    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-istio-envoy\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.871156 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.871050    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-credential-socket\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.871156 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.871098    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w92gd\" (UniqueName: \"kubernetes.io/projected/39b8d440-1120-45f5-995d-68eda0eab96e-kube-api-access-w92gd\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.871156 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.871139    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b8d440-1120-45f5-995d-68eda0eab96e-istio-podinfo\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.871345 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.871173    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b8d440-1120-45f5-995d-68eda0eab96e-istio-token\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.871345 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.871252    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-workload-certs\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.871345 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.871277    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-istio-data\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.871517 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.871406    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-workload-socket\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.871702 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.871672    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b8d440-1120-45f5-995d-68eda0eab96e-istiod-ca-cert\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.871702 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.871687    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-credential-socket\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.871867 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.871720    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-workload-certs\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.871867 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.871743    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-istio-data\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.873257 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.873235    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-istio-envoy\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.873721 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.873699    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b8d440-1120-45f5-995d-68eda0eab96e-istio-podinfo\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.879879 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.879858    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b8d440-1120-45f5-995d-68eda0eab96e-istio-token\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.880064 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.880046    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w92gd\" (UniqueName: \"kubernetes.io/projected/39b8d440-1120-45f5-995d-68eda0eab96e-kube-api-access-w92gd\") pod \"e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") " pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:31:59.908077 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:31:59.908055    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:32:00.217520 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:00.217486    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"]
Sep 16 14:32:00.218708 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:32:00.218676    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b8d440_1120_45f5_995d_68eda0eab96e.slice/crio-30b1ebd0f6ac6f0aa72b14ba736fd971ba6b39b35c8470a16b099ea57f4c3548 WatchSource:0}: Error finding container 30b1ebd0f6ac6f0aa72b14ba736fd971ba6b39b35c8470a16b099ea57f4c3548: Status 404 returned error can't find the container with id 30b1ebd0f6ac6f0aa72b14ba736fd971ba6b39b35c8470a16b099ea57f4c3548
Sep 16 14:32:00.221179 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:00.221138    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:32:00.221272 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:00.221223    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:32:00.221314 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:00.221268    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:32:00.324425 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:00.324392    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:32:00.324612 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:00.324445    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:32:00.342163 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:00.342133    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:32:00.364273 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:00.364161    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm" event={"ID":"39b8d440-1120-45f5-995d-68eda0eab96e","Type":"ContainerStarted","Data":"93373ad7a1f876ee129ba280c4a8436aa60ceca7f2aa0afdd18509f88fbe2459"}
Sep 16 14:32:00.364273 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:00.364229    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm" event={"ID":"39b8d440-1120-45f5-995d-68eda0eab96e","Type":"ContainerStarted","Data":"30b1ebd0f6ac6f0aa72b14ba736fd971ba6b39b35c8470a16b099ea57f4c3548"}
Sep 16 14:32:00.377261 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:00.377229    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:32:00.389050 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:00.389002    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm" podStartSLOduration=1.38898731 podStartE2EDuration="1.38898731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:32:00.387924589 +0000 UTC m=+1070.934666512" watchObservedRunningTime="2026-09-16 14:32:00.38898731 +0000 UTC m=+1070.935729241"
Sep 16 14:32:00.908163 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:00.908132    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:32:01.923595 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:01.923566    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:32:02.375588 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:02.375554    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:32:02.376388 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:02.376363    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:32:11.035775 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.035739    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"]
Sep 16 14:32:11.036252 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.035989    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57" podUID="c3977480-40ec-424e-a321-87a26bdb841b" containerName="maas-api" containerID="cri-o://73d36a98bd22808093b1b8bd3acc65a45c10674a7a1561b8d2619d63a0d1775a" gracePeriod=30
Sep 16 14:32:11.290605 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.290523    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:32:11.401990 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.401955    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3977480-40ec-424e-a321-87a26bdb841b-maas-api-tls\") pod \"c3977480-40ec-424e-a321-87a26bdb841b\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") "
Sep 16 14:32:11.402152 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.401999    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3977480-40ec-424e-a321-87a26bdb841b-openshift-service-ca\") pod \"c3977480-40ec-424e-a321-87a26bdb841b\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") "
Sep 16 14:32:11.402152 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.402053    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3977480-40ec-424e-a321-87a26bdb841b-metrics-certs\") pod \"c3977480-40ec-424e-a321-87a26bdb841b\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") "
Sep 16 14:32:11.402232 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.402164    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnd48\" (UniqueName: \"kubernetes.io/projected/c3977480-40ec-424e-a321-87a26bdb841b-kube-api-access-mnd48\") pod \"c3977480-40ec-424e-a321-87a26bdb841b\" (UID: \"c3977480-40ec-424e-a321-87a26bdb841b\") "
Sep 16 14:32:11.402374 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.402347    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3977480-40ec-424e-a321-87a26bdb841b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c3977480-40ec-424e-a321-87a26bdb841b" (UID: "c3977480-40ec-424e-a321-87a26bdb841b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:32:11.402471 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.402457    2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3977480-40ec-424e-a321-87a26bdb841b-openshift-service-ca\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:32:11.404194 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.404167    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3977480-40ec-424e-a321-87a26bdb841b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c3977480-40ec-424e-a321-87a26bdb841b" (UID: "c3977480-40ec-424e-a321-87a26bdb841b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:32:11.404297 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.404191    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3977480-40ec-424e-a321-87a26bdb841b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c3977480-40ec-424e-a321-87a26bdb841b" (UID: "c3977480-40ec-424e-a321-87a26bdb841b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:32:11.404297 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.404260    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3977480-40ec-424e-a321-87a26bdb841b-kube-api-access-mnd48" (OuterVolumeSpecName: "kube-api-access-mnd48") pod "c3977480-40ec-424e-a321-87a26bdb841b" (UID: "c3977480-40ec-424e-a321-87a26bdb841b"). InnerVolumeSpecName "kube-api-access-mnd48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:32:11.430930 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.430885    2553 generic.go:356] "Generic (PLEG): container finished" podID="c3977480-40ec-424e-a321-87a26bdb841b" containerID="73d36a98bd22808093b1b8bd3acc65a45c10674a7a1561b8d2619d63a0d1775a" exitCode=0
Sep 16 14:32:11.431061 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.430972    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"
Sep 16 14:32:11.431140 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.430976    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57" event={"ID":"c3977480-40ec-424e-a321-87a26bdb841b","Type":"ContainerDied","Data":"73d36a98bd22808093b1b8bd3acc65a45c10674a7a1561b8d2619d63a0d1775a"}
Sep 16 14:32:11.431140 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.431104    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57" event={"ID":"c3977480-40ec-424e-a321-87a26bdb841b","Type":"ContainerDied","Data":"2b1395a57d0b836f0fa8091eb5b8253da771ebcc4f2272a5506cad7bce8fbb04"}
Sep 16 14:32:11.431140 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.431128    2553 scope.go:117] "RemoveContainer" containerID="73d36a98bd22808093b1b8bd3acc65a45c10674a7a1561b8d2619d63a0d1775a"
Sep 16 14:32:11.458027 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.456033    2553 scope.go:117] "RemoveContainer" containerID="73d36a98bd22808093b1b8bd3acc65a45c10674a7a1561b8d2619d63a0d1775a"
Sep 16 14:32:11.458027 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:32:11.456491    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d36a98bd22808093b1b8bd3acc65a45c10674a7a1561b8d2619d63a0d1775a\": container with ID starting with 73d36a98bd22808093b1b8bd3acc65a45c10674a7a1561b8d2619d63a0d1775a not found: ID does not exist" containerID="73d36a98bd22808093b1b8bd3acc65a45c10674a7a1561b8d2619d63a0d1775a"
Sep 16 14:32:11.458027 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.456543    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d36a98bd22808093b1b8bd3acc65a45c10674a7a1561b8d2619d63a0d1775a"} err="failed to get container status \"73d36a98bd22808093b1b8bd3acc65a45c10674a7a1561b8d2619d63a0d1775a\": rpc error: code = NotFound desc = could not find container \"73d36a98bd22808093b1b8bd3acc65a45c10674a7a1561b8d2619d63a0d1775a\": container with ID starting with 73d36a98bd22808093b1b8bd3acc65a45c10674a7a1561b8d2619d63a0d1775a not found: ID does not exist"
Sep 16 14:32:11.463983 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.462089    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"]
Sep 16 14:32:11.470146 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.469907    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7ce904-6c76ddb858-8kn57"]
Sep 16 14:32:11.503405 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.503372    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnd48\" (UniqueName: \"kubernetes.io/projected/c3977480-40ec-424e-a321-87a26bdb841b-kube-api-access-mnd48\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:32:11.503405 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.503403    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3977480-40ec-424e-a321-87a26bdb841b-maas-api-tls\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:32:11.503405 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.503412    2553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3977480-40ec-424e-a321-87a26bdb841b-metrics-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:32:11.918550 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:11.918520    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3977480-40ec-424e-a321-87a26bdb841b" path="/var/lib/kubelet/pods/c3977480-40ec-424e-a321-87a26bdb841b/volumes"
Sep 16 14:32:32.553210 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:32.553170    2553 patch_prober.go:28] interesting pod/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:45746->10.132.0.54:15021: read: connection reset by peer" start-of-body=
Sep 16 14:32:32.553640 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:32.553244    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm" podUID="39b8d440-1120-45f5-995d-68eda0eab96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:45746->10.132.0.54:15021: read: connection reset by peer"
Sep 16 14:32:33.567484 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:33.567448    2553 generic.go:356] "Generic (PLEG): container finished" podID="39b8d440-1120-45f5-995d-68eda0eab96e" containerID="93373ad7a1f876ee129ba280c4a8436aa60ceca7f2aa0afdd18509f88fbe2459" exitCode=0
Sep 16 14:32:33.567967 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:33.567523    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm" event={"ID":"39b8d440-1120-45f5-995d-68eda0eab96e","Type":"ContainerDied","Data":"93373ad7a1f876ee129ba280c4a8436aa60ceca7f2aa0afdd18509f88fbe2459"}
Sep 16 14:32:33.568110 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:33.568093    2553 scope.go:117] "RemoveContainer" containerID="93373ad7a1f876ee129ba280c4a8436aa60ceca7f2aa0afdd18509f88fbe2459"
Sep 16 14:32:33.569266 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:33.569236    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:32:33.569376 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:33.569321    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:32:33.569440 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:33.569393    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:32:34.580830 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:34.580795    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm" event={"ID":"39b8d440-1120-45f5-995d-68eda0eab96e","Type":"ContainerStarted","Data":"39d3955fa8633cd97f0ffe2bf9f8f8ffb1c3a9c3f2e57bbcb8f0cfda0eac1940"}
Sep 16 14:32:34.908566 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:34.908473    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:32:34.908753 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:34.908685    2553 patch_prober.go:28] interesting pod/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 14:32:34.908753 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:34.908738    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm" podUID="39b8d440-1120-45f5-995d-68eda0eab96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 14:32:35.909297 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:35.909264    2553 patch_prober.go:28] interesting pod/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 14:32:35.909671 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:35.909319    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm" podUID="39b8d440-1120-45f5-995d-68eda0eab96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 14:32:36.909263 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:36.909226    2553 patch_prober.go:28] interesting pod/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 14:32:36.909449 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:36.909284    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm" podUID="39b8d440-1120-45f5-995d-68eda0eab96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 14:32:37.909185 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:37.909148    2553 patch_prober.go:28] interesting pod/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 14:32:37.909382 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:37.909211    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm" podUID="39b8d440-1120-45f5-995d-68eda0eab96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 14:32:38.909303 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:38.909268    2553 patch_prober.go:28] interesting pod/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 14:32:38.909691 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:38.909329    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm" podUID="39b8d440-1120-45f5-995d-68eda0eab96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 14:32:39.919248 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:39.919221    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:32:39.919627 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:39.919575    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:32:39.920315 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:39.920293    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:32:45.194248 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:45.194213    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e366590-cpwjc"]
Sep 16 14:32:45.194661 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:45.194647    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3977480-40ec-424e-a321-87a26bdb841b" containerName="maas-api"
Sep 16 14:32:45.194661 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:45.194662    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3977480-40ec-424e-a321-87a26bdb841b" containerName="maas-api"
Sep 16 14:32:45.194743 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:45.194737    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3977480-40ec-424e-a321-87a26bdb841b" containerName="maas-api"
Sep 16 14:32:45.197584 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:45.197568    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e366590-cpwjc"
Sep 16 14:32:45.204632 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:45.204611    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:32:45.216419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:45.216398    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e366590-cpwjc"]
Sep 16 14:32:45.260240 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:45.260207    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b445a93-3e79-4f34-b351-2b82a801b369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5e366590-cpwjc\" (UID: \"1b445a93-3e79-4f34-b351-2b82a801b369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e366590-cpwjc"
Sep 16 14:32:45.361091 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:45.361055    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b445a93-3e79-4f34-b351-2b82a801b369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5e366590-cpwjc\" (UID: \"1b445a93-3e79-4f34-b351-2b82a801b369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e366590-cpwjc"
Sep 16 14:32:45.361728 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:45.361707    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b445a93-3e79-4f34-b351-2b82a801b369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5e366590-cpwjc\" (UID: \"1b445a93-3e79-4f34-b351-2b82a801b369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e366590-cpwjc"
Sep 16 14:32:45.508173 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:45.508148    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e366590-cpwjc"
Sep 16 14:32:45.798104 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:45.798080    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e366590-cpwjc"]
Sep 16 14:32:45.800583 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:32:45.800552    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b445a93_3e79_4f34_b351_2b82a801b369.slice/crio-a2e5465c304fcb032d47a920c1f0a2743a60c6b8185ef524476ceb2c7fd850fc WatchSource:0}: Error finding container a2e5465c304fcb032d47a920c1f0a2743a60c6b8185ef524476ceb2c7fd850fc: Status 404 returned error can't find the container with id a2e5465c304fcb032d47a920c1f0a2743a60c6b8185ef524476ceb2c7fd850fc
Sep 16 14:32:46.653199 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:46.653160    2553 generic.go:356] "Generic (PLEG): container finished" podID="1b445a93-3e79-4f34-b351-2b82a801b369" containerID="4c02d364ac0d10ed88ef1c81edc599b6b7454c58914c645c4384ed15f8a00b33" exitCode=0
Sep 16 14:32:46.653622 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:46.653245    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e366590-cpwjc" event={"ID":"1b445a93-3e79-4f34-b351-2b82a801b369","Type":"ContainerDied","Data":"4c02d364ac0d10ed88ef1c81edc599b6b7454c58914c645c4384ed15f8a00b33"}
Sep 16 14:32:46.653622 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:46.653293    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e366590-cpwjc" event={"ID":"1b445a93-3e79-4f34-b351-2b82a801b369","Type":"ContainerStarted","Data":"a2e5465c304fcb032d47a920c1f0a2743a60c6b8185ef524476ceb2c7fd850fc"}
Sep 16 14:32:47.813762 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:47.813739    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e366590-cpwjc"
Sep 16 14:32:47.886597 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:47.886564    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b445a93-3e79-4f34-b351-2b82a801b369-maas-api-service-ca\") pod \"1b445a93-3e79-4f34-b351-2b82a801b369\" (UID: \"1b445a93-3e79-4f34-b351-2b82a801b369\") "
Sep 16 14:32:47.886870 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:47.886847    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b445a93-3e79-4f34-b351-2b82a801b369-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b445a93-3e79-4f34-b351-2b82a801b369" (UID: "1b445a93-3e79-4f34-b351-2b82a801b369"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:32:47.988166 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:47.988134    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b445a93-3e79-4f34-b351-2b82a801b369-maas-api-service-ca\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:32:48.668255 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:48.668220    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e366590-cpwjc" event={"ID":"1b445a93-3e79-4f34-b351-2b82a801b369","Type":"ContainerDied","Data":"a2e5465c304fcb032d47a920c1f0a2743a60c6b8185ef524476ceb2c7fd850fc"}
Sep 16 14:32:48.668432 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:48.668275    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2e5465c304fcb032d47a920c1f0a2743a60c6b8185ef524476ceb2c7fd850fc"
Sep 16 14:32:48.668432 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:48.668237    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e366590-cpwjc"
Sep 16 14:32:56.294903 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:56.294831    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e366590-cpwjc"]
Sep 16 14:32:56.297852 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:56.297828    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5e366590-cpwjc"]
Sep 16 14:32:57.918061 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:32:57.918017    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b445a93-3e79-4f34-b351-2b82a801b369" path="/var/lib/kubelet/pods/1b445a93-3e79-4f34-b351-2b82a801b369/volumes"
Sep 16 14:33:04.507121 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:04.507089    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"]
Sep 16 14:33:04.507692 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:04.507300    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm" podUID="39b8d440-1120-45f5-995d-68eda0eab96e" containerName="istio-proxy" containerID="cri-o://39d3955fa8633cd97f0ffe2bf9f8f8ffb1c3a9c3f2e57bbcb8f0cfda0eac1940" gracePeriod=30
Sep 16 14:33:09.782389 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.782333    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:33:09.794171 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.794134    2553 generic.go:356] "Generic (PLEG): container finished" podID="39b8d440-1120-45f5-995d-68eda0eab96e" containerID="39d3955fa8633cd97f0ffe2bf9f8f8ffb1c3a9c3f2e57bbcb8f0cfda0eac1940" exitCode=0
Sep 16 14:33:09.794291 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.794179    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"
Sep 16 14:33:09.794291 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.794221    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm" event={"ID":"39b8d440-1120-45f5-995d-68eda0eab96e","Type":"ContainerDied","Data":"39d3955fa8633cd97f0ffe2bf9f8f8ffb1c3a9c3f2e57bbcb8f0cfda0eac1940"}
Sep 16 14:33:09.794372 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.794321    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm" event={"ID":"39b8d440-1120-45f5-995d-68eda0eab96e","Type":"ContainerDied","Data":"30b1ebd0f6ac6f0aa72b14ba736fd971ba6b39b35c8470a16b099ea57f4c3548"}
Sep 16 14:33:09.794372 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.794355    2553 scope.go:117] "RemoveContainer" containerID="39d3955fa8633cd97f0ffe2bf9f8f8ffb1c3a9c3f2e57bbcb8f0cfda0eac1940"
Sep 16 14:33:09.803687 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.803672    2553 scope.go:117] "RemoveContainer" containerID="93373ad7a1f876ee129ba280c4a8436aa60ceca7f2aa0afdd18509f88fbe2459"
Sep 16 14:33:09.805090 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.805068    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-credential-socket\") pod \"39b8d440-1120-45f5-995d-68eda0eab96e\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") "
Sep 16 14:33:09.805213 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.805138    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b8d440-1120-45f5-995d-68eda0eab96e-istio-podinfo\") pod \"39b8d440-1120-45f5-995d-68eda0eab96e\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") "
Sep 16 14:33:09.805213 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.805207    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-istio-data\") pod \"39b8d440-1120-45f5-995d-68eda0eab96e\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") "
Sep 16 14:33:09.805323 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.805265    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-workload-certs\") pod \"39b8d440-1120-45f5-995d-68eda0eab96e\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") "
Sep 16 14:33:09.805379 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.805335    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w92gd\" (UniqueName: \"kubernetes.io/projected/39b8d440-1120-45f5-995d-68eda0eab96e-kube-api-access-w92gd\") pod \"39b8d440-1120-45f5-995d-68eda0eab96e\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") "
Sep 16 14:33:09.805438 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.805405    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-workload-socket\") pod \"39b8d440-1120-45f5-995d-68eda0eab96e\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") "
Sep 16 14:33:09.805438 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.805431    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b8d440-1120-45f5-995d-68eda0eab96e-istio-token\") pod \"39b8d440-1120-45f5-995d-68eda0eab96e\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") "
Sep 16 14:33:09.805545 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.805477    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b8d440-1120-45f5-995d-68eda0eab96e-istiod-ca-cert\") pod \"39b8d440-1120-45f5-995d-68eda0eab96e\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") "
Sep 16 14:33:09.805545 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.805501    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-istio-envoy\") pod \"39b8d440-1120-45f5-995d-68eda0eab96e\" (UID: \"39b8d440-1120-45f5-995d-68eda0eab96e\") "
Sep 16 14:33:09.805886 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.805861    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "39b8d440-1120-45f5-995d-68eda0eab96e" (UID: "39b8d440-1120-45f5-995d-68eda0eab96e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:33:09.806950 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.806696    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b8d440-1120-45f5-995d-68eda0eab96e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "39b8d440-1120-45f5-995d-68eda0eab96e" (UID: "39b8d440-1120-45f5-995d-68eda0eab96e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:33:09.807153 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.807100    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "39b8d440-1120-45f5-995d-68eda0eab96e" (UID: "39b8d440-1120-45f5-995d-68eda0eab96e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:33:09.807277 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.807252    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "39b8d440-1120-45f5-995d-68eda0eab96e" (UID: "39b8d440-1120-45f5-995d-68eda0eab96e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:33:09.807795 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.807776    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-istio-data" (OuterVolumeSpecName: "istio-data") pod "39b8d440-1120-45f5-995d-68eda0eab96e" (UID: "39b8d440-1120-45f5-995d-68eda0eab96e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:33:09.808997 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.808740    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b8d440-1120-45f5-995d-68eda0eab96e-kube-api-access-w92gd" (OuterVolumeSpecName: "kube-api-access-w92gd") pod "39b8d440-1120-45f5-995d-68eda0eab96e" (UID: "39b8d440-1120-45f5-995d-68eda0eab96e"). InnerVolumeSpecName "kube-api-access-w92gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:33:09.808997 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.808938    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39b8d440-1120-45f5-995d-68eda0eab96e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "39b8d440-1120-45f5-995d-68eda0eab96e" (UID: "39b8d440-1120-45f5-995d-68eda0eab96e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:33:09.809213 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.809185    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b8d440-1120-45f5-995d-68eda0eab96e-istio-token" (OuterVolumeSpecName: "istio-token") pod "39b8d440-1120-45f5-995d-68eda0eab96e" (UID: "39b8d440-1120-45f5-995d-68eda0eab96e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:33:09.809499 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.809480    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "39b8d440-1120-45f5-995d-68eda0eab96e" (UID: "39b8d440-1120-45f5-995d-68eda0eab96e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:33:09.829774 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.829755    2553 scope.go:117] "RemoveContainer" containerID="39d3955fa8633cd97f0ffe2bf9f8f8ffb1c3a9c3f2e57bbcb8f0cfda0eac1940"
Sep 16 14:33:09.830097 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:33:09.830076    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d3955fa8633cd97f0ffe2bf9f8f8ffb1c3a9c3f2e57bbcb8f0cfda0eac1940\": container with ID starting with 39d3955fa8633cd97f0ffe2bf9f8f8ffb1c3a9c3f2e57bbcb8f0cfda0eac1940 not found: ID does not exist" containerID="39d3955fa8633cd97f0ffe2bf9f8f8ffb1c3a9c3f2e57bbcb8f0cfda0eac1940"
Sep 16 14:33:09.830173 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.830104    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d3955fa8633cd97f0ffe2bf9f8f8ffb1c3a9c3f2e57bbcb8f0cfda0eac1940"} err="failed to get container status \"39d3955fa8633cd97f0ffe2bf9f8f8ffb1c3a9c3f2e57bbcb8f0cfda0eac1940\": rpc error: code = NotFound desc = could not find container \"39d3955fa8633cd97f0ffe2bf9f8f8ffb1c3a9c3f2e57bbcb8f0cfda0eac1940\": container with ID starting with 39d3955fa8633cd97f0ffe2bf9f8f8ffb1c3a9c3f2e57bbcb8f0cfda0eac1940 not found: ID does not exist"
Sep 16 14:33:09.830173 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.830124    2553 scope.go:117] "RemoveContainer" containerID="93373ad7a1f876ee129ba280c4a8436aa60ceca7f2aa0afdd18509f88fbe2459"
Sep 16 14:33:09.830436 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:33:09.830421    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93373ad7a1f876ee129ba280c4a8436aa60ceca7f2aa0afdd18509f88fbe2459\": container with ID starting with 93373ad7a1f876ee129ba280c4a8436aa60ceca7f2aa0afdd18509f88fbe2459 not found: ID does not exist" containerID="93373ad7a1f876ee129ba280c4a8436aa60ceca7f2aa0afdd18509f88fbe2459"
Sep 16 14:33:09.830487 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.830438    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93373ad7a1f876ee129ba280c4a8436aa60ceca7f2aa0afdd18509f88fbe2459"} err="failed to get container status \"93373ad7a1f876ee129ba280c4a8436aa60ceca7f2aa0afdd18509f88fbe2459\": rpc error: code = NotFound desc = could not find container \"93373ad7a1f876ee129ba280c4a8436aa60ceca7f2aa0afdd18509f88fbe2459\": container with ID starting with 93373ad7a1f876ee129ba280c4a8436aa60ceca7f2aa0afdd18509f88fbe2459 not found: ID does not exist"
Sep 16 14:33:09.907358 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.907329    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-credential-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:33:09.907358 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.907354    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b8d440-1120-45f5-995d-68eda0eab96e-istio-podinfo\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:33:09.907523 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.907368    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-istio-data\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:33:09.907523 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.907380    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-workload-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:33:09.907523 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.907392    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w92gd\" (UniqueName: \"kubernetes.io/projected/39b8d440-1120-45f5-995d-68eda0eab96e-kube-api-access-w92gd\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:33:09.907523 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.907405    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-workload-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:33:09.907523 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.907417    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b8d440-1120-45f5-995d-68eda0eab96e-istio-token\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:33:09.907523 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.907428    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b8d440-1120-45f5-995d-68eda0eab96e-istiod-ca-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:33:09.907523 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:09.907439    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b8d440-1120-45f5-995d-68eda0eab96e-istio-envoy\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:33:10.113237 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:10.113167    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"]
Sep 16 14:33:10.120904 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:10.120864    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5e366590-openshift-default-77fb9896d-w7dhm"]
Sep 16 14:33:11.918055 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:11.918014    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39b8d440-1120-45f5-995d-68eda0eab96e" path="/var/lib/kubelet/pods/39b8d440-1120-45f5-995d-68eda0eab96e/volumes"
Sep 16 14:33:19.208880 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.208851    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"]
Sep 16 14:33:19.209324 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.209311    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b445a93-3e79-4f34-b351-2b82a801b369" containerName="revoke-keys"
Sep 16 14:33:19.209324 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.209325    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b445a93-3e79-4f34-b351-2b82a801b369" containerName="revoke-keys"
Sep 16 14:33:19.209401 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.209338    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39b8d440-1120-45f5-995d-68eda0eab96e" containerName="istio-proxy"
Sep 16 14:33:19.209401 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.209343    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="39b8d440-1120-45f5-995d-68eda0eab96e" containerName="istio-proxy"
Sep 16 14:33:19.209401 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.209350    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39b8d440-1120-45f5-995d-68eda0eab96e" containerName="istio-proxy"
Sep 16 14:33:19.209401 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.209356    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="39b8d440-1120-45f5-995d-68eda0eab96e" containerName="istio-proxy"
Sep 16 14:33:19.209526 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.209421    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="39b8d440-1120-45f5-995d-68eda0eab96e" containerName="istio-proxy"
Sep 16 14:33:19.209526 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.209432    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="39b8d440-1120-45f5-995d-68eda0eab96e" containerName="istio-proxy"
Sep 16 14:33:19.209526 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.209440    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b445a93-3e79-4f34-b351-2b82a801b369" containerName="revoke-keys"
Sep 16 14:33:19.214605 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.214579    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.218479 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.218460    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-4c2787-openshift-default-dockercfg-znzqn\""
Sep 16 14:33:19.229521 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.229495    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"]
Sep 16 14:33:19.304372 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.304327    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-workload-socket\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.304719 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.304698    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-token\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.304999 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.304980    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4lh6\" (UniqueName: \"kubernetes.io/projected/e0d5f096-6536-44e4-b6f5-13719958d76e-kube-api-access-t4lh6\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.305176 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.305160    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-data\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.305339 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.305323    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-podinfo\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.305496 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.305481    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d5f096-6536-44e4-b6f5-13719958d76e-istiod-ca-cert\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.305681 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.305666    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-workload-certs\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.305855 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.305840    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-credential-socket\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.306032 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.306016    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-envoy\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.406770 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.406734    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-workload-certs\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.407004 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.406783    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-credential-socket\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.407004 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.406816    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-envoy\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.407004 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.406866    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-workload-socket\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.407004 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.406958    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-token\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.407234 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.407020    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4lh6\" (UniqueName: \"kubernetes.io/projected/e0d5f096-6536-44e4-b6f5-13719958d76e-kube-api-access-t4lh6\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.407234 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.407058    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-data\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.407234 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.407087    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-podinfo\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.407234 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.407123    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d5f096-6536-44e4-b6f5-13719958d76e-istiod-ca-cert\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.407234 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.407167    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-workload-certs\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.407502 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.407236    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-credential-socket\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.407502 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.407310    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-workload-socket\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.407502 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.407497    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-data\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.407946 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.407926    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d5f096-6536-44e4-b6f5-13719958d76e-istiod-ca-cert\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.409247 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.409220    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-envoy\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.409568 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.409549    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-podinfo\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.416970 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.416950    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-token\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.417165 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.417148    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4lh6\" (UniqueName: \"kubernetes.io/projected/e0d5f096-6536-44e4-b6f5-13719958d76e-kube-api-access-t4lh6\") pod \"e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") " pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.524537 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.524507    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:19.813127 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.813095    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"]
Sep 16 14:33:19.814399 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:33:19.814372    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d5f096_6536_44e4_b6f5_13719958d76e.slice/crio-2a625970925c332c7e5ae27a299c6e222f357c3c1f101ba5d4632e6a3ba98971 WatchSource:0}: Error finding container 2a625970925c332c7e5ae27a299c6e222f357c3c1f101ba5d4632e6a3ba98971: Status 404 returned error can't find the container with id 2a625970925c332c7e5ae27a299c6e222f357c3c1f101ba5d4632e6a3ba98971
Sep 16 14:33:19.816624 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.816591    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:33:19.816685 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.816657    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:33:19.816722 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.816683    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:33:19.849343 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:19.849317    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm" event={"ID":"e0d5f096-6536-44e4-b6f5-13719958d76e","Type":"ContainerStarted","Data":"2a625970925c332c7e5ae27a299c6e222f357c3c1f101ba5d4632e6a3ba98971"}
Sep 16 14:33:20.857285 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:20.857247    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm" event={"ID":"e0d5f096-6536-44e4-b6f5-13719958d76e","Type":"ContainerStarted","Data":"1eaf5df6946d8a8a4af61c815cce50665f4f89f14e6497cd1d6849785efd50d3"}
Sep 16 14:33:20.883834 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:20.883778    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm" podStartSLOduration=1.883762833 podStartE2EDuration="1.883762833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:33:20.880961667 +0000 UTC m=+1151.427703599" watchObservedRunningTime="2026-09-16 14:33:20.883762833 +0000 UTC m=+1151.430504833"
Sep 16 14:33:21.525632 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:21.525593    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:21.530781 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:21.530756    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:21.863712 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:21.863634    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:21.864647 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:21.864628    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:51.030090 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:51.030053    2553 generic.go:356] "Generic (PLEG): container finished" podID="e0d5f096-6536-44e4-b6f5-13719958d76e" containerID="1eaf5df6946d8a8a4af61c815cce50665f4f89f14e6497cd1d6849785efd50d3" exitCode=0
Sep 16 14:33:51.030557 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:51.030130    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm" event={"ID":"e0d5f096-6536-44e4-b6f5-13719958d76e","Type":"ContainerDied","Data":"1eaf5df6946d8a8a4af61c815cce50665f4f89f14e6497cd1d6849785efd50d3"}
Sep 16 14:33:51.030680 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:51.030598    2553 scope.go:117] "RemoveContainer" containerID="1eaf5df6946d8a8a4af61c815cce50665f4f89f14e6497cd1d6849785efd50d3"
Sep 16 14:33:51.031781 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:51.031755    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:33:51.031849 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:51.031819    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:33:51.031886 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:51.031851    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:33:51.863862 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:51.863828    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:52.037394 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:52.037359    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm" event={"ID":"e0d5f096-6536-44e4-b6f5-13719958d76e","Type":"ContainerStarted","Data":"a416539105a62c013c4f17cd158e07e9ed457413f0d63733719a9773055b445c"}
Sep 16 14:33:52.525403 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:52.525369    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:52.526459 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:52.526431    2553 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 14:33:52.526579 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:52.526483    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm" podUID="e0d5f096-6536-44e4-b6f5-13719958d76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 14:33:53.525009 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:53.524973    2553 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 14:33:53.525380 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:53.525030    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm" podUID="e0d5f096-6536-44e4-b6f5-13719958d76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 14:33:54.524977 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:54.524939    2553 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 14:33:54.525153 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:54.525012    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm" podUID="e0d5f096-6536-44e4-b6f5-13719958d76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 14:33:54.863244 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:54.859192    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"]
Sep 16 14:33:55.525245 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:55.525208    2553 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 14:33:55.525620 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:55.525265    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm" podUID="e0d5f096-6536-44e4-b6f5-13719958d76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 14:33:56.526132 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:56.526100    2553 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 14:33:56.526880 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:56.526286    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm" podUID="e0d5f096-6536-44e4-b6f5-13719958d76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 14:33:57.525302 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:57.525263    2553 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 14:33:57.525466 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:57.525329    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm" podUID="e0d5f096-6536-44e4-b6f5-13719958d76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 14:33:58.528633 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:58.528602    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:58.529105 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:58.529010    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:33:58.529590 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:33:58.529573    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:34:01.371864 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.371828    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"]
Sep 16 14:34:01.374671 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.374650    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:01.377178 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.377152    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qjgdq\""
Sep 16 14:34:01.377294 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.377198    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-2539c0-serving-cert\""
Sep 16 14:34:01.377294 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.377219    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 14:34:01.388114 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.388093    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"]
Sep 16 14:34:01.466502 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.466464    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:01.466661 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.466508    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:01.466708 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.466665    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brm48\" (UniqueName: \"kubernetes.io/projected/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-kube-api-access-brm48\") pod \"maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:01.466755 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.466707    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:01.567861 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.567821    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:01.568085 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.567962    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brm48\" (UniqueName: \"kubernetes.io/projected/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-kube-api-access-brm48\") pod \"maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:01.568085 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.567994    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:01.568085 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.568054    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:01.568257 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:34:01.568161    2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-serving-cert: secret "maas-api-e2e-shared-b-w2-2539c0-serving-cert" not found
Sep 16 14:34:01.568257 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:34:01.568239    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-maas-api-tls podName:087f1e0c-e646-4ec5-a5fb-2d6df8b56572 nodeName:}" failed. No retries permitted until 2026-09-16 14:34:02.068219243 +0000 UTC m=+1192.614961169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-maas-api-tls") pod "maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg" (UID: "087f1e0c-e646-4ec5-a5fb-2d6df8b56572") : secret "maas-api-e2e-shared-b-w2-2539c0-serving-cert" not found
Sep 16 14:34:01.568495 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.568475    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:01.570626 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.570602    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:01.578600 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:01.578572    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brm48\" (UniqueName: \"kubernetes.io/projected/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-kube-api-access-brm48\") pod \"maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:02.076034 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:02.075995    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:02.080903 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:02.080864    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:02.285928 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:02.285881    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:02.568266 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:02.568237    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"]
Sep 16 14:34:02.570440 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:34:02.570415    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087f1e0c_e646_4ec5_a5fb_2d6df8b56572.slice/crio-94e5a4153cdc19c3f58377c6fa72fc5123d1cd83a45af0aea6a316b543363dbe WatchSource:0}: Error finding container 94e5a4153cdc19c3f58377c6fa72fc5123d1cd83a45af0aea6a316b543363dbe: Status 404 returned error can't find the container with id 94e5a4153cdc19c3f58377c6fa72fc5123d1cd83a45af0aea6a316b543363dbe
Sep 16 14:34:03.113257 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:03.113223    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg" event={"ID":"087f1e0c-e646-4ec5-a5fb-2d6df8b56572","Type":"ContainerStarted","Data":"850b36b338fe405379e3adaa203e72cfb17de105b6bc8dd9b60ec95daa0377b3"}
Sep 16 14:34:03.113429 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:03.113280    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:03.113429 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:03.113301    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg" event={"ID":"087f1e0c-e646-4ec5-a5fb-2d6df8b56572","Type":"ContainerStarted","Data":"94e5a4153cdc19c3f58377c6fa72fc5123d1cd83a45af0aea6a316b543363dbe"}
Sep 16 14:34:03.130664 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:03.130605    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg" podStartSLOduration=1.758428723 podStartE2EDuration="2.130587519s" totalImagesPullingTime="372.158796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:34:03.13013985 +0000 UTC m=+1193.676881781" watchObservedRunningTime="2026-09-16 14:34:03.130587519 +0000 UTC m=+1193.677329449"
Sep 16 14:34:09.125470 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:09.125436    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:34:09.896399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:09.896303    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c" podUID="fe8fe988-1786-4c9f-833e-a0960ebd7849" containerName="main" containerID="cri-o://df037354ec85934b70685c33d7eaf3f829f46a89bb622b83d6d60f8ac55a5b68" gracePeriod=45
Sep 16 14:34:09.901865 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:09.901839    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:34:09.904289 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:09.904270    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:34:09.954080 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:09.954038    2553 scope.go:117] "RemoveContainer" containerID="20b9583d0cd96b18586ea3dc3f252701868eca38c48d4d47ff3c838157044256"
Sep 16 14:34:10.155559 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.155530    2553 generic.go:356] "Generic (PLEG): container finished" podID="fe8fe988-1786-4c9f-833e-a0960ebd7849" containerID="df037354ec85934b70685c33d7eaf3f829f46a89bb622b83d6d60f8ac55a5b68" exitCode=0
Sep 16 14:34:10.155969 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.155642    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c" event={"ID":"fe8fe988-1786-4c9f-833e-a0960ebd7849","Type":"ContainerDied","Data":"df037354ec85934b70685c33d7eaf3f829f46a89bb622b83d6d60f8ac55a5b68"}
Sep 16 14:34:10.177701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.177681    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:34:10.261590 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.261554    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-model-cache\") pod \"fe8fe988-1786-4c9f-833e-a0960ebd7849\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") "
Sep 16 14:34:10.261785 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.261635    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8fe988-1786-4c9f-833e-a0960ebd7849-tls-certs\") pod \"fe8fe988-1786-4c9f-833e-a0960ebd7849\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") "
Sep 16 14:34:10.261785 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.261744    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cvbf\" (UniqueName: \"kubernetes.io/projected/fe8fe988-1786-4c9f-833e-a0960ebd7849-kube-api-access-5cvbf\") pod \"fe8fe988-1786-4c9f-833e-a0960ebd7849\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") "
Sep 16 14:34:10.261942 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.261796    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-dshm\") pod \"fe8fe988-1786-4c9f-833e-a0960ebd7849\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") "
Sep 16 14:34:10.261942 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.261837    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-tmp-dir\") pod \"fe8fe988-1786-4c9f-833e-a0960ebd7849\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") "
Sep 16 14:34:10.261942 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.261835    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe8fe988-1786-4c9f-833e-a0960ebd7849" (UID: "fe8fe988-1786-4c9f-833e-a0960ebd7849"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:10.261942 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.261870    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-home\") pod \"fe8fe988-1786-4c9f-833e-a0960ebd7849\" (UID: \"fe8fe988-1786-4c9f-833e-a0960ebd7849\") "
Sep 16 14:34:10.262299 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.262187    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-model-cache\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:10.262984 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.262959    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe8fe988-1786-4c9f-833e-a0960ebd7849" (UID: "fe8fe988-1786-4c9f-833e-a0960ebd7849"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:10.263478 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.263459    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-home" (OuterVolumeSpecName: "home") pod "fe8fe988-1786-4c9f-833e-a0960ebd7849" (UID: "fe8fe988-1786-4c9f-833e-a0960ebd7849"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:10.264609 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.264581    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8fe988-1786-4c9f-833e-a0960ebd7849-kube-api-access-5cvbf" (OuterVolumeSpecName: "kube-api-access-5cvbf") pod "fe8fe988-1786-4c9f-833e-a0960ebd7849" (UID: "fe8fe988-1786-4c9f-833e-a0960ebd7849"). InnerVolumeSpecName "kube-api-access-5cvbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:34:10.264847 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.264815    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-dshm" (OuterVolumeSpecName: "dshm") pod "fe8fe988-1786-4c9f-833e-a0960ebd7849" (UID: "fe8fe988-1786-4c9f-833e-a0960ebd7849"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:10.265077 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.265055    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe8fe988-1786-4c9f-833e-a0960ebd7849-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe8fe988-1786-4c9f-833e-a0960ebd7849" (UID: "fe8fe988-1786-4c9f-833e-a0960ebd7849"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:34:10.363420 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.363389    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cvbf\" (UniqueName: \"kubernetes.io/projected/fe8fe988-1786-4c9f-833e-a0960ebd7849-kube-api-access-5cvbf\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:10.363420 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.363416    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-dshm\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:10.363420 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.363425    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-tmp-dir\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:10.363627 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.363433    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8fe988-1786-4c9f-833e-a0960ebd7849-home\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:10.363627 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:10.363442    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8fe988-1786-4c9f-833e-a0960ebd7849-tls-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:11.163972 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:11.163932    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c" event={"ID":"fe8fe988-1786-4c9f-833e-a0960ebd7849","Type":"ContainerDied","Data":"abc23e2b861dab063c90893c59bca8c8e425668a685cfefc33e54eeac67c4204"}
Sep 16 14:34:11.164418 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:11.163994    2553 scope.go:117] "RemoveContainer" containerID="df037354ec85934b70685c33d7eaf3f829f46a89bb622b83d6d60f8ac55a5b68"
Sep 16 14:34:11.164418 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:11.163953    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"
Sep 16 14:34:11.195800 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:11.195763    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"]
Sep 16 14:34:11.196640 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:11.196617    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-094dab/test-model-094dab-kserve-7cf6c47449-lkh7c"]
Sep 16 14:34:11.918251 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:11.918208    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe8fe988-1786-4c9f-833e-a0960ebd7849" path="/var/lib/kubelet/pods/fe8fe988-1786-4c9f-833e-a0960ebd7849/volumes"
Sep 16 14:34:32.964014 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:32.963981    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-094dab-29826150-krtdr"]
Sep 16 14:34:32.973467 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:32.973436    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-094dab-29826150-krtdr"]
Sep 16 14:34:33.925195 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:33.925159    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="667b4ffc-3ea6-48b9-b92a-a46441e004fb" path="/var/lib/kubelet/pods/667b4ffc-3ea6-48b9-b92a-a46441e004fb/volumes"
Sep 16 14:34:36.300855 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:36.300820    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"]
Sep 16 14:34:41.358538 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:41.358505    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"]
Sep 16 14:34:41.358980 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:41.358728    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4" podUID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerName="istio-proxy" containerID="cri-o://8619d11cfcc69701967d9edec253c07fae7819c5f4e72b17214ec9c49d3c6ee0" gracePeriod=30
Sep 16 14:34:42.087600 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.087560    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"]
Sep 16 14:34:42.088122 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.088101    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe8fe988-1786-4c9f-833e-a0960ebd7849" containerName="main"
Sep 16 14:34:42.088122 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.088123    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe8fe988-1786-4c9f-833e-a0960ebd7849" containerName="main"
Sep 16 14:34:42.088288 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.088253    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe8fe988-1786-4c9f-833e-a0960ebd7849" containerName="main"
Sep 16 14:34:42.091222 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.091202    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.094139 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.094122    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7b12b9e3-openshift-default-dockercfg-dpqh8\""
Sep 16 14:34:42.105296 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.105275    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"]
Sep 16 14:34:42.198085 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.198054    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-envoy\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.198225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.198124    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-workload-socket\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.198225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.198146    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-credential-socket\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.198225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.198170    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-data\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.198358 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.198230    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istiod-ca-cert\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.198358 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.198265    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-token\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.198358 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.198295    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-workload-certs\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.198358 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.198314    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-podinfo\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.198358 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.198350    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbbhv\" (UniqueName: \"kubernetes.io/projected/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-kube-api-access-qbbhv\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.298883 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.298853    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-workload-socket\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.298883 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.298884    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-credential-socket\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.299131 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.298922    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-data\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.299131 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.298960    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istiod-ca-cert\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.299131 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.298981    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-token\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.299131 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.299002    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-workload-certs\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.299131 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.299030    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-podinfo\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.299131 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.299094    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbbhv\" (UniqueName: \"kubernetes.io/projected/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-kube-api-access-qbbhv\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.299465 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.299148    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-envoy\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.299465 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.299323    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-credential-socket\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.299465 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.299356    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-workload-certs\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.299465 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.299435    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-data\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.299662 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.299604    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-workload-socket\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.299716 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.299686    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istiod-ca-cert\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.301548 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.301528    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-envoy\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.301761 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.301743    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-podinfo\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.316262 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.316240    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-token\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.318988 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.318969    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbbhv\" (UniqueName: \"kubernetes.io/projected/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-kube-api-access-qbbhv\") pod \"e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") " pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.401589 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.401525    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:42.683751 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.683723    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"]
Sep 16 14:34:42.686522 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:34:42.686498    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3860f76_5f0b_4468_bf9b_a8d6cd1a5c0f.slice/crio-4cbd4288757d94ae8f4c4853042585aa412cc87e184eaf87ac145d0644e3bb84 WatchSource:0}: Error finding container 4cbd4288757d94ae8f4c4853042585aa412cc87e184eaf87ac145d0644e3bb84: Status 404 returned error can't find the container with id 4cbd4288757d94ae8f4c4853042585aa412cc87e184eaf87ac145d0644e3bb84
Sep 16 14:34:42.688719 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.688687    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:34:42.688821 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.688788    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:34:42.688904 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:42.688833    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:34:43.353169 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:43.353131    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6" event={"ID":"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f","Type":"ContainerStarted","Data":"3e977250f209ca917a916cf65488eafde3baea544acde0f8f42a7255677d0be2"}
Sep 16 14:34:43.353359 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:43.353197    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6" event={"ID":"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f","Type":"ContainerStarted","Data":"4cbd4288757d94ae8f4c4853042585aa412cc87e184eaf87ac145d0644e3bb84"}
Sep 16 14:34:43.401806 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:43.401770    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:43.406667 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:43.406644    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:43.415270 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:43.415231    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6" podStartSLOduration=1.415217487 podStartE2EDuration="1.415217487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:34:43.412027887 +0000 UTC m=+1233.958769819" watchObservedRunningTime="2026-09-16 14:34:43.415217487 +0000 UTC m=+1233.961959418"
Sep 16 14:34:44.359908 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:44.359869    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:44.360880 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:44.360856    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:34:46.238263 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.238230    2553 patch_prober.go:28] interesting pod/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 14:34:46.238643 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.238288    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4" podUID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 14:34:46.618387 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.618362    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:34:46.746527 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.746498    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15beb83-9f62-4765-9efe-74a0be02cbfe-istiod-ca-cert\") pod \"e15beb83-9f62-4765-9efe-74a0be02cbfe\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") "
Sep 16 14:34:46.746664 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.746583    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-data\") pod \"e15beb83-9f62-4765-9efe-74a0be02cbfe\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") "
Sep 16 14:34:46.746664 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.746609    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-credential-socket\") pod \"e15beb83-9f62-4765-9efe-74a0be02cbfe\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") "
Sep 16 14:34:46.746664 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.746633    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-envoy\") pod \"e15beb83-9f62-4765-9efe-74a0be02cbfe\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") "
Sep 16 14:34:46.746808 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.746664    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-podinfo\") pod \"e15beb83-9f62-4765-9efe-74a0be02cbfe\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") "
Sep 16 14:34:46.746808 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.746694    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf55h\" (UniqueName: \"kubernetes.io/projected/e15beb83-9f62-4765-9efe-74a0be02cbfe-kube-api-access-cf55h\") pod \"e15beb83-9f62-4765-9efe-74a0be02cbfe\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") "
Sep 16 14:34:46.746808 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.746752    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-workload-certs\") pod \"e15beb83-9f62-4765-9efe-74a0be02cbfe\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") "
Sep 16 14:34:46.746983 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.746963    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-token\") pod \"e15beb83-9f62-4765-9efe-74a0be02cbfe\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") "
Sep 16 14:34:46.747048 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.746964    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-data" (OuterVolumeSpecName: "istio-data") pod "e15beb83-9f62-4765-9efe-74a0be02cbfe" (UID: "e15beb83-9f62-4765-9efe-74a0be02cbfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:46.747048 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.746983    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e15beb83-9f62-4765-9efe-74a0be02cbfe" (UID: "e15beb83-9f62-4765-9efe-74a0be02cbfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:46.747048 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.746997    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15beb83-9f62-4765-9efe-74a0be02cbfe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e15beb83-9f62-4765-9efe-74a0be02cbfe" (UID: "e15beb83-9f62-4765-9efe-74a0be02cbfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:34:46.747048 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.747010    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-workload-socket\") pod \"e15beb83-9f62-4765-9efe-74a0be02cbfe\" (UID: \"e15beb83-9f62-4765-9efe-74a0be02cbfe\") "
Sep 16 14:34:46.747270 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.747223    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e15beb83-9f62-4765-9efe-74a0be02cbfe" (UID: "e15beb83-9f62-4765-9efe-74a0be02cbfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:46.747484 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.747404    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e15beb83-9f62-4765-9efe-74a0be02cbfe" (UID: "e15beb83-9f62-4765-9efe-74a0be02cbfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:46.747595 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.747569    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-workload-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:46.747595 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.747588    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15beb83-9f62-4765-9efe-74a0be02cbfe-istiod-ca-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:46.747686 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.747603    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-data\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:46.747686 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.747615    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-credential-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:46.747686 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.747629    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-workload-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:46.748841 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.748817    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15beb83-9f62-4765-9efe-74a0be02cbfe-kube-api-access-cf55h" (OuterVolumeSpecName: "kube-api-access-cf55h") pod "e15beb83-9f62-4765-9efe-74a0be02cbfe" (UID: "e15beb83-9f62-4765-9efe-74a0be02cbfe"). InnerVolumeSpecName "kube-api-access-cf55h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:34:46.748957 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.748931    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e15beb83-9f62-4765-9efe-74a0be02cbfe" (UID: "e15beb83-9f62-4765-9efe-74a0be02cbfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:34:46.749207 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.749182    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e15beb83-9f62-4765-9efe-74a0be02cbfe" (UID: "e15beb83-9f62-4765-9efe-74a0be02cbfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:46.749315 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.749229    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-token" (OuterVolumeSpecName: "istio-token") pod "e15beb83-9f62-4765-9efe-74a0be02cbfe" (UID: "e15beb83-9f62-4765-9efe-74a0be02cbfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:34:46.848852 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.848774    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-token\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:46.848852 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.848801    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-envoy\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:46.848852 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.848809    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15beb83-9f62-4765-9efe-74a0be02cbfe-istio-podinfo\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:46.848852 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:46.848817    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf55h\" (UniqueName: \"kubernetes.io/projected/e15beb83-9f62-4765-9efe-74a0be02cbfe-kube-api-access-cf55h\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:47.382826 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:47.382789    2553 generic.go:356] "Generic (PLEG): container finished" podID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerID="8619d11cfcc69701967d9edec253c07fae7819c5f4e72b17214ec9c49d3c6ee0" exitCode=0
Sep 16 14:34:47.383225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:47.382840    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"
Sep 16 14:34:47.383225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:47.382882    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4" event={"ID":"e15beb83-9f62-4765-9efe-74a0be02cbfe","Type":"ContainerDied","Data":"8619d11cfcc69701967d9edec253c07fae7819c5f4e72b17214ec9c49d3c6ee0"}
Sep 16 14:34:47.383225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:47.382961    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4" event={"ID":"e15beb83-9f62-4765-9efe-74a0be02cbfe","Type":"ContainerDied","Data":"31317e1486cbf8f6e5842f6bc269ab8144bcecff2ff41af9f09e65fafc61b8e3"}
Sep 16 14:34:47.383225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:47.382990    2553 scope.go:117] "RemoveContainer" containerID="8619d11cfcc69701967d9edec253c07fae7819c5f4e72b17214ec9c49d3c6ee0"
Sep 16 14:34:47.392470 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:47.392450    2553 scope.go:117] "RemoveContainer" containerID="44ed5742430b80f8a168537b51fd601584769d8ed7bf271313de6a0a9fc20a23"
Sep 16 14:34:47.400721 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:47.400704    2553 scope.go:117] "RemoveContainer" containerID="8619d11cfcc69701967d9edec253c07fae7819c5f4e72b17214ec9c49d3c6ee0"
Sep 16 14:34:47.400997 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:34:47.400977    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8619d11cfcc69701967d9edec253c07fae7819c5f4e72b17214ec9c49d3c6ee0\": container with ID starting with 8619d11cfcc69701967d9edec253c07fae7819c5f4e72b17214ec9c49d3c6ee0 not found: ID does not exist" containerID="8619d11cfcc69701967d9edec253c07fae7819c5f4e72b17214ec9c49d3c6ee0"
Sep 16 14:34:47.401049 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:47.401006    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8619d11cfcc69701967d9edec253c07fae7819c5f4e72b17214ec9c49d3c6ee0"} err="failed to get container status \"8619d11cfcc69701967d9edec253c07fae7819c5f4e72b17214ec9c49d3c6ee0\": rpc error: code = NotFound desc = could not find container \"8619d11cfcc69701967d9edec253c07fae7819c5f4e72b17214ec9c49d3c6ee0\": container with ID starting with 8619d11cfcc69701967d9edec253c07fae7819c5f4e72b17214ec9c49d3c6ee0 not found: ID does not exist"
Sep 16 14:34:47.401049 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:47.401026    2553 scope.go:117] "RemoveContainer" containerID="44ed5742430b80f8a168537b51fd601584769d8ed7bf271313de6a0a9fc20a23"
Sep 16 14:34:47.401258 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:34:47.401241    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ed5742430b80f8a168537b51fd601584769d8ed7bf271313de6a0a9fc20a23\": container with ID starting with 44ed5742430b80f8a168537b51fd601584769d8ed7bf271313de6a0a9fc20a23 not found: ID does not exist" containerID="44ed5742430b80f8a168537b51fd601584769d8ed7bf271313de6a0a9fc20a23"
Sep 16 14:34:47.401310 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:47.401263    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ed5742430b80f8a168537b51fd601584769d8ed7bf271313de6a0a9fc20a23"} err="failed to get container status \"44ed5742430b80f8a168537b51fd601584769d8ed7bf271313de6a0a9fc20a23\": rpc error: code = NotFound desc = could not find container \"44ed5742430b80f8a168537b51fd601584769d8ed7bf271313de6a0a9fc20a23\": container with ID starting with 44ed5742430b80f8a168537b51fd601584769d8ed7bf271313de6a0a9fc20a23 not found: ID does not exist"
Sep 16 14:34:47.407432 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:47.407410    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"]
Sep 16 14:34:47.412645 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:47.412624    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-094dab-openshift-default-7d86774f74-gw2j4"]
Sep 16 14:34:47.918206 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:47.918174    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e15beb83-9f62-4765-9efe-74a0be02cbfe" path="/var/lib/kubelet/pods/e15beb83-9f62-4765-9efe-74a0be02cbfe/volumes"
Sep 16 14:34:48.700437 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.700405    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"]
Sep 16 14:34:48.700873 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.700832    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerName="istio-proxy"
Sep 16 14:34:48.700873 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.700843    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerName="istio-proxy"
Sep 16 14:34:48.700873 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.700855    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerName="istio-proxy"
Sep 16 14:34:48.700873 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.700862    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerName="istio-proxy"
Sep 16 14:34:48.701124 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.700958    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerName="istio-proxy"
Sep 16 14:34:48.705729 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.705711    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:34:48.709139 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.709116    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7b12b9e3-serving-cert\""
Sep 16 14:34:48.719120 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.719098    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"]
Sep 16 14:34:48.771452 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.771427    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b04d15-3fe7-4565-8f5a-27394035c01b-maas-api-tls\") pod \"maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:34:48.771576 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.771456    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcrnp\" (UniqueName: \"kubernetes.io/projected/76b04d15-3fe7-4565-8f5a-27394035c01b-kube-api-access-bcrnp\") pod \"maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:34:48.771576 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.771486    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b04d15-3fe7-4565-8f5a-27394035c01b-openshift-service-ca\") pod \"maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:34:48.771576 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.771560    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b04d15-3fe7-4565-8f5a-27394035c01b-metrics-certs\") pod \"maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:34:48.872516 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.872476    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b04d15-3fe7-4565-8f5a-27394035c01b-metrics-certs\") pod \"maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:34:48.872678 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.872593    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b04d15-3fe7-4565-8f5a-27394035c01b-maas-api-tls\") pod \"maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:34:48.872678 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.872613    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcrnp\" (UniqueName: \"kubernetes.io/projected/76b04d15-3fe7-4565-8f5a-27394035c01b-kube-api-access-bcrnp\") pod \"maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:34:48.872678 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.872644    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b04d15-3fe7-4565-8f5a-27394035c01b-openshift-service-ca\") pod \"maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:34:48.872791 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:34:48.872740    2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-serving-cert: secret "maas-api-e2e-mt-7b12b9e3-serving-cert" not found
Sep 16 14:34:48.872826 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:34:48.872810    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b04d15-3fe7-4565-8f5a-27394035c01b-maas-api-tls podName:76b04d15-3fe7-4565-8f5a-27394035c01b nodeName:}" failed. No retries permitted until 2026-09-16 14:34:49.372789856 +0000 UTC m=+1239.919531785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/76b04d15-3fe7-4565-8f5a-27394035c01b-maas-api-tls") pod "maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr" (UID: "76b04d15-3fe7-4565-8f5a-27394035c01b") : secret "maas-api-e2e-mt-7b12b9e3-serving-cert" not found
Sep 16 14:34:48.873360 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.873334    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b04d15-3fe7-4565-8f5a-27394035c01b-openshift-service-ca\") pod \"maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:34:48.874865 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.874842    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b04d15-3fe7-4565-8f5a-27394035c01b-metrics-certs\") pod \"maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:34:48.883249 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:48.883221    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcrnp\" (UniqueName: \"kubernetes.io/projected/76b04d15-3fe7-4565-8f5a-27394035c01b-kube-api-access-bcrnp\") pod \"maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:34:49.377997 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:49.377965    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b04d15-3fe7-4565-8f5a-27394035c01b-maas-api-tls\") pod \"maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:34:49.380281 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:49.380253    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b04d15-3fe7-4565-8f5a-27394035c01b-maas-api-tls\") pod \"maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:34:49.618411 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:49.618373    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:34:49.904569 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:49.904544    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"]
Sep 16 14:34:49.907227 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:34:49.907202    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b04d15_3fe7_4565_8f5a_27394035c01b.slice/crio-563a9a847e29b9de4b082e4b76abec1caf33da2b11caae7e44e81a036203cfc0 WatchSource:0}: Error finding container 563a9a847e29b9de4b082e4b76abec1caf33da2b11caae7e44e81a036203cfc0: Status 404 returned error can't find the container with id 563a9a847e29b9de4b082e4b76abec1caf33da2b11caae7e44e81a036203cfc0
Sep 16 14:34:50.404463 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:50.404436    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr" event={"ID":"76b04d15-3fe7-4565-8f5a-27394035c01b","Type":"ContainerStarted","Data":"563a9a847e29b9de4b082e4b76abec1caf33da2b11caae7e44e81a036203cfc0"}
Sep 16 14:34:51.320445 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.320402    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd" podUID="76e076d7-0621-4671-a734-984379f27066" containerName="main" containerID="cri-o://618df7ea47fe3a36142e6d6e17296102bd05f65caabcb33f46a766b7cb18e7d1" gracePeriod=45
Sep 16 14:34:51.412472 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.412444    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr" event={"ID":"76b04d15-3fe7-4565-8f5a-27394035c01b","Type":"ContainerStarted","Data":"3bf4d3a68c6627a9c6170f0932ce107da83906173a823a7454084a45bd7d061a"}
Sep 16 14:34:51.412605 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.412527    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:34:51.442584 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.442543    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr" podStartSLOduration=2.975847636 podStartE2EDuration="3.442526006s" totalImagesPullingTime="466.67837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:34:51.442117611 +0000 UTC m=+1241.988859543" watchObservedRunningTime="2026-09-16 14:34:51.442526006 +0000 UTC m=+1241.989267987"
Sep 16 14:34:51.586430 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.586369    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:34:51.705091 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.705059    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-tmp-dir\") pod \"76e076d7-0621-4671-a734-984379f27066\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") "
Sep 16 14:34:51.705265 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.705126    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z5rc\" (UniqueName: \"kubernetes.io/projected/76e076d7-0621-4671-a734-984379f27066-kube-api-access-8z5rc\") pod \"76e076d7-0621-4671-a734-984379f27066\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") "
Sep 16 14:34:51.705265 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.705232    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-dshm\") pod \"76e076d7-0621-4671-a734-984379f27066\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") "
Sep 16 14:34:51.705265 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.705230    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76e076d7-0621-4671-a734-984379f27066" (UID: "76e076d7-0621-4671-a734-984379f27066"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:51.705452 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.705299    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-home\") pod \"76e076d7-0621-4671-a734-984379f27066\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") "
Sep 16 14:34:51.705452 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.705365    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e076d7-0621-4671-a734-984379f27066-tls-certs\") pod \"76e076d7-0621-4671-a734-984379f27066\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") "
Sep 16 14:34:51.705452 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.705392    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-model-cache\") pod \"76e076d7-0621-4671-a734-984379f27066\" (UID: \"76e076d7-0621-4671-a734-984379f27066\") "
Sep 16 14:34:51.705594 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.705568    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-home" (OuterVolumeSpecName: "home") pod "76e076d7-0621-4671-a734-984379f27066" (UID: "76e076d7-0621-4671-a734-984379f27066"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:51.705668 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.705643    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-model-cache" (OuterVolumeSpecName: "model-cache") pod "76e076d7-0621-4671-a734-984379f27066" (UID: "76e076d7-0621-4671-a734-984379f27066"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:51.705827 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.705807    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-home\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:51.705827 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.705824    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-model-cache\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:51.706001 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.705836    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-tmp-dir\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:51.707486 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.707468    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76e076d7-0621-4671-a734-984379f27066-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76e076d7-0621-4671-a734-984379f27066" (UID: "76e076d7-0621-4671-a734-984379f27066"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:34:51.707723 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.707696    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-dshm" (OuterVolumeSpecName: "dshm") pod "76e076d7-0621-4671-a734-984379f27066" (UID: "76e076d7-0621-4671-a734-984379f27066"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:51.707805 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.707725    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e076d7-0621-4671-a734-984379f27066-kube-api-access-8z5rc" (OuterVolumeSpecName: "kube-api-access-8z5rc") pod "76e076d7-0621-4671-a734-984379f27066" (UID: "76e076d7-0621-4671-a734-984379f27066"). InnerVolumeSpecName "kube-api-access-8z5rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:34:51.806747 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.806715    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e076d7-0621-4671-a734-984379f27066-dshm\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:51.806747 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.806743    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e076d7-0621-4671-a734-984379f27066-tls-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:51.806958 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:51.806757    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z5rc\" (UniqueName: \"kubernetes.io/projected/76e076d7-0621-4671-a734-984379f27066-kube-api-access-8z5rc\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:34:52.419172 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:52.419139    2553 generic.go:356] "Generic (PLEG): container finished" podID="76e076d7-0621-4671-a734-984379f27066" containerID="618df7ea47fe3a36142e6d6e17296102bd05f65caabcb33f46a766b7cb18e7d1" exitCode=0
Sep 16 14:34:52.419642 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:52.419181    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"
Sep 16 14:34:52.419642 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:52.419222    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd" event={"ID":"76e076d7-0621-4671-a734-984379f27066","Type":"ContainerDied","Data":"618df7ea47fe3a36142e6d6e17296102bd05f65caabcb33f46a766b7cb18e7d1"}
Sep 16 14:34:52.419642 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:52.419264    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd" event={"ID":"76e076d7-0621-4671-a734-984379f27066","Type":"ContainerDied","Data":"ad1544f537b5a112f632504d6b2c860d272ca06bb269b6cabca3e053e8f6717b"}
Sep 16 14:34:52.419642 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:52.419284    2553 scope.go:117] "RemoveContainer" containerID="618df7ea47fe3a36142e6d6e17296102bd05f65caabcb33f46a766b7cb18e7d1"
Sep 16 14:34:52.428111 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:52.428083    2553 scope.go:117] "RemoveContainer" containerID="618df7ea47fe3a36142e6d6e17296102bd05f65caabcb33f46a766b7cb18e7d1"
Sep 16 14:34:52.428377 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:34:52.428358    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"618df7ea47fe3a36142e6d6e17296102bd05f65caabcb33f46a766b7cb18e7d1\": container with ID starting with 618df7ea47fe3a36142e6d6e17296102bd05f65caabcb33f46a766b7cb18e7d1 not found: ID does not exist" containerID="618df7ea47fe3a36142e6d6e17296102bd05f65caabcb33f46a766b7cb18e7d1"
Sep 16 14:34:52.428446 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:52.428384    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618df7ea47fe3a36142e6d6e17296102bd05f65caabcb33f46a766b7cb18e7d1"} err="failed to get container status \"618df7ea47fe3a36142e6d6e17296102bd05f65caabcb33f46a766b7cb18e7d1\": rpc error: code = NotFound desc = could not find container \"618df7ea47fe3a36142e6d6e17296102bd05f65caabcb33f46a766b7cb18e7d1\": container with ID starting with 618df7ea47fe3a36142e6d6e17296102bd05f65caabcb33f46a766b7cb18e7d1 not found: ID does not exist"
Sep 16 14:34:52.448767 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:52.448742    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"]
Sep 16 14:34:52.453712 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:52.453693    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d0b89a/test-model-d0b89a-kserve-7789cfc549-xtdgd"]
Sep 16 14:34:53.920667 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:53.920632    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76e076d7-0621-4671-a734-984379f27066" path="/var/lib/kubelet/pods/76e076d7-0621-4671-a734-984379f27066/volumes"
Sep 16 14:34:57.425017 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:34:57.424990    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:35:04.200425 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:04.200348    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"]
Sep 16 14:35:04.200804 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:04.200648    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r" podUID="37282c70-7f7e-43f4-9bbd-448859955468" containerName="istio-proxy" containerID="cri-o://0181beca7ed6ab3a49c807fee282b3075bd75df6bddeaa5ca472ba4041770b1d" gracePeriod=30
Sep 16 14:35:06.341566 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:06.341523    2553 patch_prober.go:28] interesting pod/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 14:35:06.342086 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:06.341594    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r" podUID="37282c70-7f7e-43f4-9bbd-448859955468" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 14:35:07.524108 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:07.524076    2553 generic.go:356] "Generic (PLEG): container finished" podID="b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" containerID="3e977250f209ca917a916cf65488eafde3baea544acde0f8f42a7255677d0be2" exitCode=0
Sep 16 14:35:07.524514 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:07.524149    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6" event={"ID":"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f","Type":"ContainerDied","Data":"3e977250f209ca917a916cf65488eafde3baea544acde0f8f42a7255677d0be2"}
Sep 16 14:35:07.524634 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:07.524620    2553 scope.go:117] "RemoveContainer" containerID="3e977250f209ca917a916cf65488eafde3baea544acde0f8f42a7255677d0be2"
Sep 16 14:35:07.525747 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:07.525715    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:35:07.525856 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:07.525783    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:35:07.525856 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:07.525822    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:35:08.531797 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:08.531763    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6" event={"ID":"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f","Type":"ContainerStarted","Data":"31f3c5a8af5575e6d8ac29e2a9bd74cd6c2ebd444e7a8b85f52e0511b0b16e10"}
Sep 16 14:35:09.402440 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.402358    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:35:09.402623 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.402597    2553 patch_prober.go:28] interesting pod/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 16 14:35:09.402703 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.402652    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6" podUID="b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 16 14:35:09.512689 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.512667    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:35:09.539576 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.539547    2553 generic.go:356] "Generic (PLEG): container finished" podID="37282c70-7f7e-43f4-9bbd-448859955468" containerID="0181beca7ed6ab3a49c807fee282b3075bd75df6bddeaa5ca472ba4041770b1d" exitCode=0
Sep 16 14:35:09.539928 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.539590    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"
Sep 16 14:35:09.539928 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.539626    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r" event={"ID":"37282c70-7f7e-43f4-9bbd-448859955468","Type":"ContainerDied","Data":"0181beca7ed6ab3a49c807fee282b3075bd75df6bddeaa5ca472ba4041770b1d"}
Sep 16 14:35:09.539928 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.539680    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r" event={"ID":"37282c70-7f7e-43f4-9bbd-448859955468","Type":"ContainerDied","Data":"55068d86656a3bc30a223e47b422382de508999ef5919c3257ab0e00cba0dc8c"}
Sep 16 14:35:09.539928 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.539706    2553 scope.go:117] "RemoveContainer" containerID="0181beca7ed6ab3a49c807fee282b3075bd75df6bddeaa5ca472ba4041770b1d"
Sep 16 14:35:09.548980 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.548964    2553 scope.go:117] "RemoveContainer" containerID="a5011758fb67b36fb471f59fca185d2c77d0b4c235f689a0c7729f8cc9dd8fd5"
Sep 16 14:35:09.574770 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.574639    2553 scope.go:117] "RemoveContainer" containerID="0181beca7ed6ab3a49c807fee282b3075bd75df6bddeaa5ca472ba4041770b1d"
Sep 16 14:35:09.579063 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:35:09.579018    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0181beca7ed6ab3a49c807fee282b3075bd75df6bddeaa5ca472ba4041770b1d\": container with ID starting with 0181beca7ed6ab3a49c807fee282b3075bd75df6bddeaa5ca472ba4041770b1d not found: ID does not exist" containerID="0181beca7ed6ab3a49c807fee282b3075bd75df6bddeaa5ca472ba4041770b1d"
Sep 16 14:35:09.579211 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.579070    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0181beca7ed6ab3a49c807fee282b3075bd75df6bddeaa5ca472ba4041770b1d"} err="failed to get container status \"0181beca7ed6ab3a49c807fee282b3075bd75df6bddeaa5ca472ba4041770b1d\": rpc error: code = NotFound desc = could not find container \"0181beca7ed6ab3a49c807fee282b3075bd75df6bddeaa5ca472ba4041770b1d\": container with ID starting with 0181beca7ed6ab3a49c807fee282b3075bd75df6bddeaa5ca472ba4041770b1d not found: ID does not exist"
Sep 16 14:35:09.579211 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.579097    2553 scope.go:117] "RemoveContainer" containerID="a5011758fb67b36fb471f59fca185d2c77d0b4c235f689a0c7729f8cc9dd8fd5"
Sep 16 14:35:09.579579 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:35:09.579498    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5011758fb67b36fb471f59fca185d2c77d0b4c235f689a0c7729f8cc9dd8fd5\": container with ID starting with a5011758fb67b36fb471f59fca185d2c77d0b4c235f689a0c7729f8cc9dd8fd5 not found: ID does not exist" containerID="a5011758fb67b36fb471f59fca185d2c77d0b4c235f689a0c7729f8cc9dd8fd5"
Sep 16 14:35:09.579579 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.579535    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5011758fb67b36fb471f59fca185d2c77d0b4c235f689a0c7729f8cc9dd8fd5"} err="failed to get container status \"a5011758fb67b36fb471f59fca185d2c77d0b4c235f689a0c7729f8cc9dd8fd5\": rpc error: code = NotFound desc = could not find container \"a5011758fb67b36fb471f59fca185d2c77d0b4c235f689a0c7729f8cc9dd8fd5\": container with ID starting with a5011758fb67b36fb471f59fca185d2c77d0b4c235f689a0c7729f8cc9dd8fd5 not found: ID does not exist"
Sep 16 14:35:09.593173 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.593067    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-istio-envoy\") pod \"37282c70-7f7e-43f4-9bbd-448859955468\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") "
Sep 16 14:35:09.593173 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.593125    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37282c70-7f7e-43f4-9bbd-448859955468-istio-podinfo\") pod \"37282c70-7f7e-43f4-9bbd-448859955468\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") "
Sep 16 14:35:09.593173 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.593162    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-credential-socket\") pod \"37282c70-7f7e-43f4-9bbd-448859955468\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") "
Sep 16 14:35:09.593460 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.593220    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37282c70-7f7e-43f4-9bbd-448859955468-istiod-ca-cert\") pod \"37282c70-7f7e-43f4-9bbd-448859955468\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") "
Sep 16 14:35:09.593460 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.593257    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-istio-data\") pod \"37282c70-7f7e-43f4-9bbd-448859955468\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") "
Sep 16 14:35:09.593460 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.593285    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37282c70-7f7e-43f4-9bbd-448859955468-istio-token\") pod \"37282c70-7f7e-43f4-9bbd-448859955468\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") "
Sep 16 14:35:09.593460 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.593337    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-workload-certs\") pod \"37282c70-7f7e-43f4-9bbd-448859955468\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") "
Sep 16 14:35:09.593460 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.593396    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr9fq\" (UniqueName: \"kubernetes.io/projected/37282c70-7f7e-43f4-9bbd-448859955468-kube-api-access-vr9fq\") pod \"37282c70-7f7e-43f4-9bbd-448859955468\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") "
Sep 16 14:35:09.593460 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.593447    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-workload-socket\") pod \"37282c70-7f7e-43f4-9bbd-448859955468\" (UID: \"37282c70-7f7e-43f4-9bbd-448859955468\") "
Sep 16 14:35:09.593755 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.593694    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-istio-data" (OuterVolumeSpecName: "istio-data") pod "37282c70-7f7e-43f4-9bbd-448859955468" (UID: "37282c70-7f7e-43f4-9bbd-448859955468"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:35:09.593755 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.593704    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "37282c70-7f7e-43f4-9bbd-448859955468" (UID: "37282c70-7f7e-43f4-9bbd-448859955468"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:35:09.593876 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.593862    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "37282c70-7f7e-43f4-9bbd-448859955468" (UID: "37282c70-7f7e-43f4-9bbd-448859955468"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:35:09.593985 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.593972    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-credential-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:35:09.594046 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.593990    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-istio-data\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:35:09.594046 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.594006    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-workload-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:35:09.594146 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.594123    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37282c70-7f7e-43f4-9bbd-448859955468-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "37282c70-7f7e-43f4-9bbd-448859955468" (UID: "37282c70-7f7e-43f4-9bbd-448859955468"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:35:09.594517 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.594490    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "37282c70-7f7e-43f4-9bbd-448859955468" (UID: "37282c70-7f7e-43f4-9bbd-448859955468"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:35:09.596307 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.596276    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37282c70-7f7e-43f4-9bbd-448859955468-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "37282c70-7f7e-43f4-9bbd-448859955468" (UID: "37282c70-7f7e-43f4-9bbd-448859955468"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:35:09.596537 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.596500    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37282c70-7f7e-43f4-9bbd-448859955468-istio-token" (OuterVolumeSpecName: "istio-token") pod "37282c70-7f7e-43f4-9bbd-448859955468" (UID: "37282c70-7f7e-43f4-9bbd-448859955468"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:35:09.596946 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.596924    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37282c70-7f7e-43f4-9bbd-448859955468-kube-api-access-vr9fq" (OuterVolumeSpecName: "kube-api-access-vr9fq") pod "37282c70-7f7e-43f4-9bbd-448859955468" (UID: "37282c70-7f7e-43f4-9bbd-448859955468"). InnerVolumeSpecName "kube-api-access-vr9fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:35:09.597633 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.597608    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "37282c70-7f7e-43f4-9bbd-448859955468" (UID: "37282c70-7f7e-43f4-9bbd-448859955468"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:35:09.694734 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.694696    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr9fq\" (UniqueName: \"kubernetes.io/projected/37282c70-7f7e-43f4-9bbd-448859955468-kube-api-access-vr9fq\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:35:09.694734 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.694730    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-workload-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:35:09.694734 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.694739    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37282c70-7f7e-43f4-9bbd-448859955468-istio-envoy\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:35:09.695020 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.694748    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37282c70-7f7e-43f4-9bbd-448859955468-istio-podinfo\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:35:09.695020 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.694756    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37282c70-7f7e-43f4-9bbd-448859955468-istiod-ca-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:35:09.695020 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.694763    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37282c70-7f7e-43f4-9bbd-448859955468-istio-token\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:35:09.885438 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.885404    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"]
Sep 16 14:35:09.893245 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.893211    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d0b89a-openshift-default-97dff7896-ddz6r"]
Sep 16 14:35:09.918279 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:09.918251    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37282c70-7f7e-43f4-9bbd-448859955468" path="/var/lib/kubelet/pods/37282c70-7f7e-43f4-9bbd-448859955468/volumes"
Sep 16 14:35:10.024976 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:10.024947    2553 scope.go:117] "RemoveContainer" containerID="e178f90e60707bce24099f115c035c5f0eeb922d4cc7c3aefd89f2556925b374"
Sep 16 14:35:10.403289 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:10.403251    2553 patch_prober.go:28] interesting pod/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 16 14:35:10.403467 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:10.403320    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6" podUID="b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 16 14:35:11.402145 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:11.402111    2553 patch_prober.go:28] interesting pod/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 16 14:35:11.402526 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:11.402197    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6" podUID="b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 16 14:35:12.402678 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:12.402632    2553 patch_prober.go:28] interesting pod/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 16 14:35:12.403196 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:12.402717    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6" podUID="b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 16 14:35:13.401964 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:13.401923    2553 patch_prober.go:28] interesting pod/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 16 14:35:13.402139 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:13.402005    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6" podUID="b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 16 14:35:14.360249 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:14.360214    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:35:14.406059 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:14.406032    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:35:14.575185 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:14.575152    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:35:37.706772 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.706742    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"]
Sep 16 14:35:37.707501 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.707480    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37282c70-7f7e-43f4-9bbd-448859955468" containerName="istio-proxy"
Sep 16 14:35:37.707501 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.707502    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="37282c70-7f7e-43f4-9bbd-448859955468" containerName="istio-proxy"
Sep 16 14:35:37.707689 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.707519    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37282c70-7f7e-43f4-9bbd-448859955468" containerName="istio-proxy"
Sep 16 14:35:37.707689 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.707526    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="37282c70-7f7e-43f4-9bbd-448859955468" containerName="istio-proxy"
Sep 16 14:35:37.707689 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.707550    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e076d7-0621-4671-a734-984379f27066" containerName="main"
Sep 16 14:35:37.707689 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.707558    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e076d7-0621-4671-a734-984379f27066" containerName="main"
Sep 16 14:35:37.707689 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.707686    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="76e076d7-0621-4671-a734-984379f27066" containerName="main"
Sep 16 14:35:37.707973 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.707702    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="37282c70-7f7e-43f4-9bbd-448859955468" containerName="istio-proxy"
Sep 16 14:35:37.707973 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.707715    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="37282c70-7f7e-43f4-9bbd-448859955468" containerName="istio-proxy"
Sep 16 14:35:37.707973 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.707725    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e15beb83-9f62-4765-9efe-74a0be02cbfe" containerName="istio-proxy"
Sep 16 14:35:37.718256 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.718232    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.726153 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.726131    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-924916\"/\"distinct-2-w0-924916-kserve-self-signed-certs\""
Sep 16 14:35:37.749981 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.749949    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"]
Sep 16 14:35:37.880958 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.880927    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-home\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.880958 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.880956    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-dshm\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.881173 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.880977    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-model-cache\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.881173 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.881157    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-tmp-dir\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.881252 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.881206    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-tls-certs\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.881333 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.881262    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b5f6\" (UniqueName: \"kubernetes.io/projected/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-kube-api-access-2b5f6\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.981800 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.981758    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-tls-certs\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.982022 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.981826    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b5f6\" (UniqueName: \"kubernetes.io/projected/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-kube-api-access-2b5f6\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.982022 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.981853    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-home\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.982022 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.981872    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-dshm\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.982022 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.981925    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-model-cache\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.982250 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.982075    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-tmp-dir\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.982323 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.982305    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-home\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.982416 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.982395    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-model-cache\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.982488 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.982471    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-tmp-dir\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.984296 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.984277    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-dshm\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:37.984488 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:37.984472    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-tls-certs\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:38.005643 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.005620    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b5f6\" (UniqueName: \"kubernetes.io/projected/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-kube-api-access-2b5f6\") pod \"distinct-2-w0-924916-kserve-7f9bcd755-zc98q\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") " pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:38.027550 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.027524    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:38.317806 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.317711    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"]
Sep 16 14:35:38.323946 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:35:38.323914    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod696ae56f_a5a2_48a4_8ba7_f26332ca7c09.slice/crio-07e6c7382082d1239d1ce63fa7b2e6a9aac34c9657f9ab13de18db26e0024147 WatchSource:0}: Error finding container 07e6c7382082d1239d1ce63fa7b2e6a9aac34c9657f9ab13de18db26e0024147: Status 404 returned error can't find the container with id 07e6c7382082d1239d1ce63fa7b2e6a9aac34c9657f9ab13de18db26e0024147
Sep 16 14:35:38.486512 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.486476    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"]
Sep 16 14:35:38.489050 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.489031    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:35:38.494667 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.494648    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-9f88e1-serving-cert\""
Sep 16 14:35:38.515977 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.515950    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"]
Sep 16 14:35:38.588242 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.588163    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95f00493-e2a6-44af-987f-6d07769594dc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv\" (UID: \"95f00493-e2a6-44af-987f-6d07769594dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:35:38.588242 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.588210    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f00493-e2a6-44af-987f-6d07769594dc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv\" (UID: \"95f00493-e2a6-44af-987f-6d07769594dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:35:38.588423 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.588328    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxkfc\" (UniqueName: \"kubernetes.io/projected/95f00493-e2a6-44af-987f-6d07769594dc-kube-api-access-pxkfc\") pod \"maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv\" (UID: \"95f00493-e2a6-44af-987f-6d07769594dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:35:38.588460 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.588428    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95f00493-e2a6-44af-987f-6d07769594dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv\" (UID: \"95f00493-e2a6-44af-987f-6d07769594dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:35:38.689651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.689603    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxkfc\" (UniqueName: \"kubernetes.io/projected/95f00493-e2a6-44af-987f-6d07769594dc-kube-api-access-pxkfc\") pod \"maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv\" (UID: \"95f00493-e2a6-44af-987f-6d07769594dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:35:38.689827 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.689739    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95f00493-e2a6-44af-987f-6d07769594dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv\" (UID: \"95f00493-e2a6-44af-987f-6d07769594dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:35:38.689827 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.689779    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95f00493-e2a6-44af-987f-6d07769594dc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv\" (UID: \"95f00493-e2a6-44af-987f-6d07769594dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:35:38.689827 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.689814    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f00493-e2a6-44af-987f-6d07769594dc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv\" (UID: \"95f00493-e2a6-44af-987f-6d07769594dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:35:38.690479 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.690458    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f00493-e2a6-44af-987f-6d07769594dc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv\" (UID: \"95f00493-e2a6-44af-987f-6d07769594dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:35:38.692182 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.692163    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95f00493-e2a6-44af-987f-6d07769594dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv\" (UID: \"95f00493-e2a6-44af-987f-6d07769594dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:35:38.692269 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.692218    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95f00493-e2a6-44af-987f-6d07769594dc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv\" (UID: \"95f00493-e2a6-44af-987f-6d07769594dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:35:38.704509 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.704485    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxkfc\" (UniqueName: \"kubernetes.io/projected/95f00493-e2a6-44af-987f-6d07769594dc-kube-api-access-pxkfc\") pod \"maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv\" (UID: \"95f00493-e2a6-44af-987f-6d07769594dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:35:38.714955 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.714930    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q" event={"ID":"696ae56f-a5a2-48a4-8ba7-f26332ca7c09","Type":"ContainerStarted","Data":"18c399cf2f22aaddd498b6ae24b34ff12db32069f88b439cf67d90b0fb3acd08"}
Sep 16 14:35:38.715245 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.714967    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q" event={"ID":"696ae56f-a5a2-48a4-8ba7-f26332ca7c09","Type":"ContainerStarted","Data":"07e6c7382082d1239d1ce63fa7b2e6a9aac34c9657f9ab13de18db26e0024147"}
Sep 16 14:35:38.757301 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.757253    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q" podStartSLOduration=1.757238689 podStartE2EDuration="1.757238689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:35:38.755312167 +0000 UTC m=+1289.302054098" watchObservedRunningTime="2026-09-16 14:35:38.757238689 +0000 UTC m=+1289.303980619"
Sep 16 14:35:38.799315 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:38.799292    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:35:39.108346 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:39.108315    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"]
Sep 16 14:35:39.109924 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:35:39.109874    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f00493_e2a6_44af_987f_6d07769594dc.slice/crio-606276da53c4ffd98fc1ce3a5a706ca1bd065c3bb2ae41a0af6d0923e08a780a WatchSource:0}: Error finding container 606276da53c4ffd98fc1ce3a5a706ca1bd065c3bb2ae41a0af6d0923e08a780a: Status 404 returned error can't find the container with id 606276da53c4ffd98fc1ce3a5a706ca1bd065c3bb2ae41a0af6d0923e08a780a
Sep 16 14:35:39.111343 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:39.111322    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:35:39.724314 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:39.724260    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv" event={"ID":"95f00493-e2a6-44af-987f-6d07769594dc","Type":"ContainerStarted","Data":"f9c43237a06600dbe11cc00399358112ac3d0974b1441a41aae5afaf75adb091"}
Sep 16 14:35:39.724783 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:39.724348    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv" event={"ID":"95f00493-e2a6-44af-987f-6d07769594dc","Type":"ContainerStarted","Data":"606276da53c4ffd98fc1ce3a5a706ca1bd065c3bb2ae41a0af6d0923e08a780a"}
Sep 16 14:35:39.724783 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:39.724724    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:35:39.783356 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:39.783300    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv" podStartSLOduration=1.293785294 podStartE2EDuration="1.783281898s" totalImagesPullingTime="489.496604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:35:39.781315371 +0000 UTC m=+1290.328057301" watchObservedRunningTime="2026-09-16 14:35:39.783281898 +0000 UTC m=+1290.330023828"
Sep 16 14:35:45.734931 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:45.734885    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:35:48.028091 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:48.028061    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:48.028492 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:48.028129    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:48.041035 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:48.041011    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:35:48.790424 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:35:48.790397    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:36:03.146589 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.146559    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"]
Sep 16 14:36:03.150216 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.150192    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.153504 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.153473    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-055ce3-openshift-default-dockercfg-trgtn\""
Sep 16 14:36:03.188568 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.188542    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"]
Sep 16 14:36:03.228368 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.228328    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242319a3-bd8b-4739-afc9-d885f8d7d65d-istiod-ca-cert\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.228543 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.228375    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-workload-socket\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.228543 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.228407    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-data\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.228543 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.228467    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62b6h\" (UniqueName: \"kubernetes.io/projected/242319a3-bd8b-4739-afc9-d885f8d7d65d-kube-api-access-62b6h\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.228701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.228554    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-token\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.228701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.228606    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-workload-certs\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.228701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.228655    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-credential-socket\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.228701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.228684    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-podinfo\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.228866 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.228720    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-envoy\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.330124 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.330089    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62b6h\" (UniqueName: \"kubernetes.io/projected/242319a3-bd8b-4739-afc9-d885f8d7d65d-kube-api-access-62b6h\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.330271 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.330159    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-token\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.330271 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.330183    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-workload-certs\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.330271 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.330210    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-credential-socket\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.330271 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.330247    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-podinfo\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.330476 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.330286    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-envoy\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.330476 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.330343    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242319a3-bd8b-4739-afc9-d885f8d7d65d-istiod-ca-cert\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.330476 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.330459    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-workload-socket\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.330623 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.330508    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-data\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.330683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.330646    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-credential-socket\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.330683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.330667    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-workload-certs\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.330918 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.330871    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-data\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.331031 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.330930    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-workload-socket\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.331080 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.331065    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242319a3-bd8b-4739-afc9-d885f8d7d65d-istiod-ca-cert\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.332658 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.332637    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-envoy\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.332812 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.332787    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-podinfo\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.349529 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.349502    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-token\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.353485 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.353456    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62b6h\" (UniqueName: \"kubernetes.io/projected/242319a3-bd8b-4739-afc9-d885f8d7d65d-kube-api-access-62b6h\") pod \"e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") " pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.462783 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.462693    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:03.784547 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.784516    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"]
Sep 16 14:36:03.785880 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:36:03.785854    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242319a3_bd8b_4739_afc9_d885f8d7d65d.slice/crio-bd8861283855dcf7624da99353f76cbead0235a46112ceda690e08e67d3d4842 WatchSource:0}: Error finding container bd8861283855dcf7624da99353f76cbead0235a46112ceda690e08e67d3d4842: Status 404 returned error can't find the container with id bd8861283855dcf7624da99353f76cbead0235a46112ceda690e08e67d3d4842
Sep 16 14:36:03.788747 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.788703    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:36:03.788848 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.788796    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:36:03.788848 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.788841    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:36:03.874179 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:03.874143    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz" event={"ID":"242319a3-bd8b-4739-afc9-d885f8d7d65d","Type":"ContainerStarted","Data":"bd8861283855dcf7624da99353f76cbead0235a46112ceda690e08e67d3d4842"}
Sep 16 14:36:04.881360 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:04.881330    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz" event={"ID":"242319a3-bd8b-4739-afc9-d885f8d7d65d","Type":"ContainerStarted","Data":"d35afeea70ee758e77fb98608684650650fb9cae841cd3831579e6a7a6a62da2"}
Sep 16 14:36:05.010578 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:05.010527    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz" podStartSLOduration=2.010511562 podStartE2EDuration="2.010511562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:04.98957833 +0000 UTC m=+1315.536320262" watchObservedRunningTime="2026-09-16 14:36:05.010511562 +0000 UTC m=+1315.557253560"
Sep 16 14:36:05.462864 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:05.462829    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:05.468445 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:05.468419    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:05.887347 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:05.887317    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:05.888414 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:05.888392    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:10.069749 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:10.069715    2553 scope.go:117] "RemoveContainer" containerID="94118db0eefa4f94f151c90e53a57c71da023c37e60ee6fe420e29b5e98bacc3"
Sep 16 14:36:26.964815 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:26.964772    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"]
Sep 16 14:36:26.971718 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:26.971686    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:26.976285 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:26.976258    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7b12b9e3\"/\"openshift-service-ca.crt\""
Sep 16 14:36:26.976391 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:26.976284    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7b12b9e3\"/\"kube-root-ca.crt\""
Sep 16 14:36:26.977477 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:26.977453    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7b12b9e3\"/\"e2e-lifecycle-model-7b12b9e3-kserve-self-signed-certs\""
Sep 16 14:36:26.977765 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:26.977746    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7b12b9e3\"/\"default-dockercfg-pmwbd\""
Sep 16 14:36:26.991926 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:26.991882    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"]
Sep 16 14:36:27.093263 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.093231    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-home\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.093451 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.093271    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhmz\" (UniqueName: \"kubernetes.io/projected/8617636a-09a0-4f5e-8763-006fed508bdf-kube-api-access-qjhmz\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.093451 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.093325    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8617636a-09a0-4f5e-8763-006fed508bdf-tls-certs\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.093451 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.093373    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-tmp-dir\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.093451 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.093433    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-dshm\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.093642 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.093580    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-model-cache\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.194940 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.194877    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-model-cache\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.195112 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.194973    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-home\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.195112 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.195014    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhmz\" (UniqueName: \"kubernetes.io/projected/8617636a-09a0-4f5e-8763-006fed508bdf-kube-api-access-qjhmz\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.195112 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.195059    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8617636a-09a0-4f5e-8763-006fed508bdf-tls-certs\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.195369 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.195125    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-tmp-dir\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.195451 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.195212    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-dshm\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.195512 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.195488    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-tmp-dir\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.195512 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.195413    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-model-cache\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.195624 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.195601    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-home\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.197555 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.197533    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-dshm\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.197741 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.197721    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8617636a-09a0-4f5e-8763-006fed508bdf-tls-certs\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.205480 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.205457    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhmz\" (UniqueName: \"kubernetes.io/projected/8617636a-09a0-4f5e-8763-006fed508bdf-kube-api-access-qjhmz\") pod \"e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") " pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.285039 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.284999    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:27.563528 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:27.563412    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"]
Sep 16 14:36:27.568189 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:36:27.568142    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8617636a_09a0_4f5e_8763_006fed508bdf.slice/crio-5669e40ff083339bb32be5670d4165424f54c065e6dbaffcd8f310d6215224c7 WatchSource:0}: Error finding container 5669e40ff083339bb32be5670d4165424f54c065e6dbaffcd8f310d6215224c7: Status 404 returned error can't find the container with id 5669e40ff083339bb32be5670d4165424f54c065e6dbaffcd8f310d6215224c7
Sep 16 14:36:28.014698 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:28.014654    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4" event={"ID":"8617636a-09a0-4f5e-8763-006fed508bdf","Type":"ContainerStarted","Data":"acb548b8415bed22cf075b878fd898df639366d3789b680cd744d50aab6abb0e"}
Sep 16 14:36:28.015092 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:28.014711    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4" event={"ID":"8617636a-09a0-4f5e-8763-006fed508bdf","Type":"ContainerStarted","Data":"5669e40ff083339bb32be5670d4165424f54c065e6dbaffcd8f310d6215224c7"}
Sep 16 14:36:28.046317 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:28.046223    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4" podStartSLOduration=2.046207731 podStartE2EDuration="2.046207731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:28.045866908 +0000 UTC m=+1338.592608841" watchObservedRunningTime="2026-09-16 14:36:28.046207731 +0000 UTC m=+1338.592949661"
Sep 16 14:36:32.037293 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:32.037261    2553 generic.go:356] "Generic (PLEG): container finished" podID="242319a3-bd8b-4739-afc9-d885f8d7d65d" containerID="d35afeea70ee758e77fb98608684650650fb9cae841cd3831579e6a7a6a62da2" exitCode=0
Sep 16 14:36:32.037682 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:32.037335    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz" event={"ID":"242319a3-bd8b-4739-afc9-d885f8d7d65d","Type":"ContainerDied","Data":"d35afeea70ee758e77fb98608684650650fb9cae841cd3831579e6a7a6a62da2"}
Sep 16 14:36:32.037724 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:32.037705    2553 scope.go:117] "RemoveContainer" containerID="d35afeea70ee758e77fb98608684650650fb9cae841cd3831579e6a7a6a62da2"
Sep 16 14:36:32.038798 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:32.038775    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:36:32.038856 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:32.038839    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:36:32.038907 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:32.038872    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:36:33.045136 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:33.045104    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz" event={"ID":"242319a3-bd8b-4739-afc9-d885f8d7d65d","Type":"ContainerStarted","Data":"bd08f2d16605e0aa13e0071bf9f5c73de63e92df1cf819150c6684acfeae6a0c"}
Sep 16 14:36:33.462924 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:33.462804    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:33.464817 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:33.464788    2553 patch_prober.go:28] interesting pod/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 14:36:33.465095 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:33.465057    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz" podUID="242319a3-bd8b-4739-afc9-d885f8d7d65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 14:36:34.464436 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:34.463992    2553 patch_prober.go:28] interesting pod/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 14:36:34.464436 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:34.464055    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz" podUID="242319a3-bd8b-4739-afc9-d885f8d7d65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 14:36:35.464161 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:35.464121    2553 patch_prober.go:28] interesting pod/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 14:36:35.464462 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:35.464432    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz" podUID="242319a3-bd8b-4739-afc9-d885f8d7d65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 14:36:35.888085 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:35.888042    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:36.463875 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:36.463827    2553 patch_prober.go:28] interesting pod/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 14:36:36.464074 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:36.463930    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz" podUID="242319a3-bd8b-4739-afc9-d885f8d7d65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 14:36:37.286443 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:37.286404    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:37.286824 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:37.286460    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:37.299313 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:37.299280    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:37.464202 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:37.464160    2553 patch_prober.go:28] interesting pod/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 14:36:37.464404 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:37.464240    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz" podUID="242319a3-bd8b-4739-afc9-d885f8d7d65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 14:36:38.093484 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:38.093454    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:36:38.467680 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:38.467600    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:36:38.468730 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:36:38.468709    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:37:38.131208 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.131172    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"]
Sep 16 14:37:38.131814 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.131462    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr" podUID="76b04d15-3fe7-4565-8f5a-27394035c01b" containerName="maas-api" containerID="cri-o://3bf4d3a68c6627a9c6170f0932ce107da83906173a823a7454084a45bd7d061a" gracePeriod=30
Sep 16 14:37:38.386024 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.385962    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:37:38.420776 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.420743    2553 generic.go:356] "Generic (PLEG): container finished" podID="76b04d15-3fe7-4565-8f5a-27394035c01b" containerID="3bf4d3a68c6627a9c6170f0932ce107da83906173a823a7454084a45bd7d061a" exitCode=0
Sep 16 14:37:38.421042 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.420787    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr" event={"ID":"76b04d15-3fe7-4565-8f5a-27394035c01b","Type":"ContainerDied","Data":"3bf4d3a68c6627a9c6170f0932ce107da83906173a823a7454084a45bd7d061a"}
Sep 16 14:37:38.421042 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.420809    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"
Sep 16 14:37:38.421042 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.420842    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr" event={"ID":"76b04d15-3fe7-4565-8f5a-27394035c01b","Type":"ContainerDied","Data":"563a9a847e29b9de4b082e4b76abec1caf33da2b11caae7e44e81a036203cfc0"}
Sep 16 14:37:38.421042 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.420869    2553 scope.go:117] "RemoveContainer" containerID="3bf4d3a68c6627a9c6170f0932ce107da83906173a823a7454084a45bd7d061a"
Sep 16 14:37:38.435191 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.435171    2553 scope.go:117] "RemoveContainer" containerID="3bf4d3a68c6627a9c6170f0932ce107da83906173a823a7454084a45bd7d061a"
Sep 16 14:37:38.435521 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:37:38.435489    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf4d3a68c6627a9c6170f0932ce107da83906173a823a7454084a45bd7d061a\": container with ID starting with 3bf4d3a68c6627a9c6170f0932ce107da83906173a823a7454084a45bd7d061a not found: ID does not exist" containerID="3bf4d3a68c6627a9c6170f0932ce107da83906173a823a7454084a45bd7d061a"
Sep 16 14:37:38.435596 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.435525    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf4d3a68c6627a9c6170f0932ce107da83906173a823a7454084a45bd7d061a"} err="failed to get container status \"3bf4d3a68c6627a9c6170f0932ce107da83906173a823a7454084a45bd7d061a\": rpc error: code = NotFound desc = could not find container \"3bf4d3a68c6627a9c6170f0932ce107da83906173a823a7454084a45bd7d061a\": container with ID starting with 3bf4d3a68c6627a9c6170f0932ce107da83906173a823a7454084a45bd7d061a not found: ID does not exist"
Sep 16 14:37:38.467508 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.467487    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b04d15-3fe7-4565-8f5a-27394035c01b-metrics-certs\") pod \"76b04d15-3fe7-4565-8f5a-27394035c01b\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") "
Sep 16 14:37:38.467598 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.467575    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b04d15-3fe7-4565-8f5a-27394035c01b-openshift-service-ca\") pod \"76b04d15-3fe7-4565-8f5a-27394035c01b\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") "
Sep 16 14:37:38.467675 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.467660    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b04d15-3fe7-4565-8f5a-27394035c01b-maas-api-tls\") pod \"76b04d15-3fe7-4565-8f5a-27394035c01b\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") "
Sep 16 14:37:38.467758 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.467735    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcrnp\" (UniqueName: \"kubernetes.io/projected/76b04d15-3fe7-4565-8f5a-27394035c01b-kube-api-access-bcrnp\") pod \"76b04d15-3fe7-4565-8f5a-27394035c01b\" (UID: \"76b04d15-3fe7-4565-8f5a-27394035c01b\") "
Sep 16 14:37:38.467962 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.467933    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b04d15-3fe7-4565-8f5a-27394035c01b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "76b04d15-3fe7-4565-8f5a-27394035c01b" (UID: "76b04d15-3fe7-4565-8f5a-27394035c01b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:38.468133 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.468116    2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b04d15-3fe7-4565-8f5a-27394035c01b-openshift-service-ca\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:37:38.469700 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.469676    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b04d15-3fe7-4565-8f5a-27394035c01b-kube-api-access-bcrnp" (OuterVolumeSpecName: "kube-api-access-bcrnp") pod "76b04d15-3fe7-4565-8f5a-27394035c01b" (UID: "76b04d15-3fe7-4565-8f5a-27394035c01b"). InnerVolumeSpecName "kube-api-access-bcrnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:37:38.470107 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.470089    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b04d15-3fe7-4565-8f5a-27394035c01b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "76b04d15-3fe7-4565-8f5a-27394035c01b" (UID: "76b04d15-3fe7-4565-8f5a-27394035c01b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:38.470185 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.470106    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b04d15-3fe7-4565-8f5a-27394035c01b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "76b04d15-3fe7-4565-8f5a-27394035c01b" (UID: "76b04d15-3fe7-4565-8f5a-27394035c01b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:38.569180 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.569150    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76b04d15-3fe7-4565-8f5a-27394035c01b-maas-api-tls\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:37:38.569180 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.569180    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcrnp\" (UniqueName: \"kubernetes.io/projected/76b04d15-3fe7-4565-8f5a-27394035c01b-kube-api-access-bcrnp\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:37:38.569391 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.569193    2553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b04d15-3fe7-4565-8f5a-27394035c01b-metrics-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:37:38.751651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.751626    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"]
Sep 16 14:37:38.756300 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:38.756275    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b12b9e3-868946c86-hk8hr"]
Sep 16 14:37:39.921968 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:39.921927    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76b04d15-3fe7-4565-8f5a-27394035c01b" path="/var/lib/kubelet/pods/76b04d15-3fe7-4565-8f5a-27394035c01b/volumes"
Sep 16 14:37:46.580463 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:37:46.580422    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"]
Sep 16 14:38:01.606119 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:01.606070    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4" podUID="8617636a-09a0-4f5e-8763-006fed508bdf" containerName="main" containerID="cri-o://acb548b8415bed22cf075b878fd898df639366d3789b680cd744d50aab6abb0e" gracePeriod=45
Sep 16 14:38:01.864661 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:01.864597    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:38:01.916075 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:01.916044    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-tmp-dir\") pod \"8617636a-09a0-4f5e-8763-006fed508bdf\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") "
Sep 16 14:38:01.916235 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:01.916124    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjhmz\" (UniqueName: \"kubernetes.io/projected/8617636a-09a0-4f5e-8763-006fed508bdf-kube-api-access-qjhmz\") pod \"8617636a-09a0-4f5e-8763-006fed508bdf\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") "
Sep 16 14:38:01.916235 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:01.916177    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-home\") pod \"8617636a-09a0-4f5e-8763-006fed508bdf\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") "
Sep 16 14:38:01.916235 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:01.916215    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-dshm\") pod \"8617636a-09a0-4f5e-8763-006fed508bdf\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") "
Sep 16 14:38:01.916469 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:01.916262    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-model-cache\") pod \"8617636a-09a0-4f5e-8763-006fed508bdf\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") "
Sep 16 14:38:01.916469 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:01.916309    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8617636a-09a0-4f5e-8763-006fed508bdf-tls-certs\") pod \"8617636a-09a0-4f5e-8763-006fed508bdf\" (UID: \"8617636a-09a0-4f5e-8763-006fed508bdf\") "
Sep 16 14:38:01.916469 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:01.916354    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8617636a-09a0-4f5e-8763-006fed508bdf" (UID: "8617636a-09a0-4f5e-8763-006fed508bdf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:38:01.916758 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:01.916727    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-model-cache" (OuterVolumeSpecName: "model-cache") pod "8617636a-09a0-4f5e-8763-006fed508bdf" (UID: "8617636a-09a0-4f5e-8763-006fed508bdf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:38:01.916829 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:01.916739    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-home" (OuterVolumeSpecName: "home") pod "8617636a-09a0-4f5e-8763-006fed508bdf" (UID: "8617636a-09a0-4f5e-8763-006fed508bdf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:38:01.916883 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:01.916861    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-model-cache\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:38:01.916883 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:01.916879    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-tmp-dir\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:38:01.919168 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:01.919129    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-dshm" (OuterVolumeSpecName: "dshm") pod "8617636a-09a0-4f5e-8763-006fed508bdf" (UID: "8617636a-09a0-4f5e-8763-006fed508bdf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:38:01.919168 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:01.919133    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8617636a-09a0-4f5e-8763-006fed508bdf-kube-api-access-qjhmz" (OuterVolumeSpecName: "kube-api-access-qjhmz") pod "8617636a-09a0-4f5e-8763-006fed508bdf" (UID: "8617636a-09a0-4f5e-8763-006fed508bdf"). InnerVolumeSpecName "kube-api-access-qjhmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:38:01.919370 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:01.919193    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8617636a-09a0-4f5e-8763-006fed508bdf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8617636a-09a0-4f5e-8763-006fed508bdf" (UID: "8617636a-09a0-4f5e-8763-006fed508bdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:38:02.017980 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:02.017947    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8617636a-09a0-4f5e-8763-006fed508bdf-tls-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:38:02.017980 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:02.017975    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjhmz\" (UniqueName: \"kubernetes.io/projected/8617636a-09a0-4f5e-8763-006fed508bdf-kube-api-access-qjhmz\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:38:02.017980 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:02.017984    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-home\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:38:02.018180 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:02.017992    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8617636a-09a0-4f5e-8763-006fed508bdf-dshm\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:38:02.567395 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:02.567352    2553 generic.go:356] "Generic (PLEG): container finished" podID="8617636a-09a0-4f5e-8763-006fed508bdf" containerID="acb548b8415bed22cf075b878fd898df639366d3789b680cd744d50aab6abb0e" exitCode=0
Sep 16 14:38:02.567583 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:02.567424    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"
Sep 16 14:38:02.567583 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:02.567433    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4" event={"ID":"8617636a-09a0-4f5e-8763-006fed508bdf","Type":"ContainerDied","Data":"acb548b8415bed22cf075b878fd898df639366d3789b680cd744d50aab6abb0e"}
Sep 16 14:38:02.567583 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:02.567485    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4" event={"ID":"8617636a-09a0-4f5e-8763-006fed508bdf","Type":"ContainerDied","Data":"5669e40ff083339bb32be5670d4165424f54c065e6dbaffcd8f310d6215224c7"}
Sep 16 14:38:02.567583 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:02.567508    2553 scope.go:117] "RemoveContainer" containerID="acb548b8415bed22cf075b878fd898df639366d3789b680cd744d50aab6abb0e"
Sep 16 14:38:02.577067 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:02.577052    2553 scope.go:117] "RemoveContainer" containerID="acb548b8415bed22cf075b878fd898df639366d3789b680cd744d50aab6abb0e"
Sep 16 14:38:02.577342 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:38:02.577321    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb548b8415bed22cf075b878fd898df639366d3789b680cd744d50aab6abb0e\": container with ID starting with acb548b8415bed22cf075b878fd898df639366d3789b680cd744d50aab6abb0e not found: ID does not exist" containerID="acb548b8415bed22cf075b878fd898df639366d3789b680cd744d50aab6abb0e"
Sep 16 14:38:02.577405 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:02.577349    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb548b8415bed22cf075b878fd898df639366d3789b680cd744d50aab6abb0e"} err="failed to get container status \"acb548b8415bed22cf075b878fd898df639366d3789b680cd744d50aab6abb0e\": rpc error: code = NotFound desc = could not find container \"acb548b8415bed22cf075b878fd898df639366d3789b680cd744d50aab6abb0e\": container with ID starting with acb548b8415bed22cf075b878fd898df639366d3789b680cd744d50aab6abb0e not found: ID does not exist"
Sep 16 14:38:02.585554 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:02.585529    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"]
Sep 16 14:38:02.589212 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:02.589187    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7b12b9e3/e2e-lifecycle-model-7b12b9e3-kserve-5f4669c979-czzd4"]
Sep 16 14:38:03.918002 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:03.917967    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8617636a-09a0-4f5e-8763-006fed508bdf" path="/var/lib/kubelet/pods/8617636a-09a0-4f5e-8763-006fed508bdf/volumes"
Sep 16 14:38:04.415427 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.412004    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"]
Sep 16 14:38:04.415427 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.413081    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b04d15-3fe7-4565-8f5a-27394035c01b" containerName="maas-api"
Sep 16 14:38:04.415427 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.413102    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b04d15-3fe7-4565-8f5a-27394035c01b" containerName="maas-api"
Sep 16 14:38:04.415427 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.413127    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8617636a-09a0-4f5e-8763-006fed508bdf" containerName="main"
Sep 16 14:38:04.415427 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.413136    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8617636a-09a0-4f5e-8763-006fed508bdf" containerName="main"
Sep 16 14:38:04.415427 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.413382    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b04d15-3fe7-4565-8f5a-27394035c01b" containerName="maas-api"
Sep 16 14:38:04.415427 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.413400    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8617636a-09a0-4f5e-8763-006fed508bdf" containerName="main"
Sep 16 14:38:04.419296 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.419261    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.420634 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.420610    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"]
Sep 16 14:38:04.423629 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.423599    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-055ce3\"/\"kube-root-ca.crt\""
Sep 16 14:38:04.423762 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.423643    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-055ce3\"/\"openshift-service-ca.crt\""
Sep 16 14:38:04.423762 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.423653    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-055ce3\"/\"auth-test-model-055ce3-kserve-self-signed-certs\""
Sep 16 14:38:04.423762 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.423599    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-055ce3\"/\"default-dockercfg-v45lm\""
Sep 16 14:38:04.540106 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.540073    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-dshm\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.540106 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.540110    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-home\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.540313 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.540144    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvxzc\" (UniqueName: \"kubernetes.io/projected/cc8cd338-9b14-42ac-ac98-c315286ec8bf-kube-api-access-zvxzc\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.540313 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.540209    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc8cd338-9b14-42ac-ac98-c315286ec8bf-tls-certs\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.540313 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.540286    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-model-cache\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.540434 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.540316    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-tmp-dir\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.641945 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.641913    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-dshm\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.642120 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.641956    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-home\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.642120 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.642001    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvxzc\" (UniqueName: \"kubernetes.io/projected/cc8cd338-9b14-42ac-ac98-c315286ec8bf-kube-api-access-zvxzc\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.642120 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.642024    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc8cd338-9b14-42ac-ac98-c315286ec8bf-tls-certs\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.642234 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.642128    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-model-cache\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.642234 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.642168    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-tmp-dir\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.642443 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.642416    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-home\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.642585 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.642468    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-model-cache\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.642585 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.642492    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-tmp-dir\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.644320 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.644293    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-dshm\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.644570 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.644551    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc8cd338-9b14-42ac-ac98-c315286ec8bf-tls-certs\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.650451 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.650427    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvxzc\" (UniqueName: \"kubernetes.io/projected/cc8cd338-9b14-42ac-ac98-c315286ec8bf-kube-api-access-zvxzc\") pod \"auth-test-model-055ce3-kserve-76fc74d67-g2fkg\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:04.731935 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:04.731870    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:05.006672 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:05.006648    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"]
Sep 16 14:38:05.008253 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:38:05.008222    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc8cd338_9b14_42ac_ac98_c315286ec8bf.slice/crio-f601b94b41a560b18c40b831f19f541d3b831a61e67c882a74e9cd95dc188fc5 WatchSource:0}: Error finding container f601b94b41a560b18c40b831f19f541d3b831a61e67c882a74e9cd95dc188fc5: Status 404 returned error can't find the container with id f601b94b41a560b18c40b831f19f541d3b831a61e67c882a74e9cd95dc188fc5
Sep 16 14:38:05.586233 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:05.586198    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg" event={"ID":"cc8cd338-9b14-42ac-ac98-c315286ec8bf","Type":"ContainerStarted","Data":"694f2300a9563085fe7e643d8885a2233f65d36838507a21eca349c06d4ccc9a"}
Sep 16 14:38:05.586233 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:05.586240    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg" event={"ID":"cc8cd338-9b14-42ac-ac98-c315286ec8bf","Type":"ContainerStarted","Data":"f601b94b41a560b18c40b831f19f541d3b831a61e67c882a74e9cd95dc188fc5"}
Sep 16 14:38:05.606305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:05.606260    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg" podStartSLOduration=1.606245195 podStartE2EDuration="1.606245195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:38:05.604850664 +0000 UTC m=+1436.151592589" watchObservedRunningTime="2026-09-16 14:38:05.606245195 +0000 UTC m=+1436.152987129"
Sep 16 14:38:14.732730 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:14.732692    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:14.733139 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:14.732772    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:14.744906 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:14.744874    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:15.653480 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:15.653445    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:38:46.800476 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:46.800444    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"]
Sep 16 14:38:46.801012 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:46.800726    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6" podUID="b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" containerName="istio-proxy" containerID="cri-o://31f3c5a8af5575e6d8ac29e2a9bd74cd6c2ebd444e7a8b85f52e0511b0b16e10" gracePeriod=30
Sep 16 14:38:52.064051 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.063993    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:38:52.208824 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.208797    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-workload-certs\") pod \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") "
Sep 16 14:38:52.208992 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.208832    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istiod-ca-cert\") pod \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") "
Sep 16 14:38:52.208992 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.208854    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-envoy\") pod \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") "
Sep 16 14:38:52.208992 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.208874    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-podinfo\") pod \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") "
Sep 16 14:38:52.208992 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.208921    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-data\") pod \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") "
Sep 16 14:38:52.208992 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.208947    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-credential-socket\") pod \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") "
Sep 16 14:38:52.209266 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.209048    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-workload-socket\") pod \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") "
Sep 16 14:38:52.209266 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.209075    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbbhv\" (UniqueName: \"kubernetes.io/projected/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-kube-api-access-qbbhv\") pod \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") "
Sep 16 14:38:52.209266 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.209103    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-token\") pod \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\" (UID: \"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f\") "
Sep 16 14:38:52.209266 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.209100    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" (UID: "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:38:52.209472 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.209270    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" (UID: "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:38:52.209472 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.209309    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-data" (OuterVolumeSpecName: "istio-data") pod "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" (UID: "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:38:52.209472 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.209329    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" (UID: "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:38:52.209592 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.209505    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" (UID: "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:38:52.209649 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.209632    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-workload-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:38:52.209704 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.209656    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istiod-ca-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:38:52.209704 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.209668    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-data\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:38:52.209704 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.209680    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-credential-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:38:52.209704 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.209693    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-workload-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:38:52.211258 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.211234    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-token" (OuterVolumeSpecName: "istio-token") pod "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" (UID: "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:38:52.211366 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.211230    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" (UID: "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:38:52.211509 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.211482    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" (UID: "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:38:52.211509 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.211498    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-kube-api-access-qbbhv" (OuterVolumeSpecName: "kube-api-access-qbbhv") pod "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" (UID: "b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f"). InnerVolumeSpecName "kube-api-access-qbbhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:38:52.310772 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.310743    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbbhv\" (UniqueName: \"kubernetes.io/projected/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-kube-api-access-qbbhv\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:38:52.310772 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.310769    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-token\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:38:52.310772 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.310778    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-envoy\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:38:52.310986 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.310785    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f-istio-podinfo\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:38:52.859964 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.859930    2553 generic.go:356] "Generic (PLEG): container finished" podID="b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" containerID="31f3c5a8af5575e6d8ac29e2a9bd74cd6c2ebd444e7a8b85f52e0511b0b16e10" exitCode=0
Sep 16 14:38:52.860139 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.859985    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"
Sep 16 14:38:52.860139 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.859988    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6" event={"ID":"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f","Type":"ContainerDied","Data":"31f3c5a8af5575e6d8ac29e2a9bd74cd6c2ebd444e7a8b85f52e0511b0b16e10"}
Sep 16 14:38:52.860139 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.860111    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6" event={"ID":"b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f","Type":"ContainerDied","Data":"4cbd4288757d94ae8f4c4853042585aa412cc87e184eaf87ac145d0644e3bb84"}
Sep 16 14:38:52.860139 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.860135    2553 scope.go:117] "RemoveContainer" containerID="31f3c5a8af5575e6d8ac29e2a9bd74cd6c2ebd444e7a8b85f52e0511b0b16e10"
Sep 16 14:38:52.871281 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.871262    2553 scope.go:117] "RemoveContainer" containerID="3e977250f209ca917a916cf65488eafde3baea544acde0f8f42a7255677d0be2"
Sep 16 14:38:52.879306 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.879287    2553 scope.go:117] "RemoveContainer" containerID="31f3c5a8af5575e6d8ac29e2a9bd74cd6c2ebd444e7a8b85f52e0511b0b16e10"
Sep 16 14:38:52.879566 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:38:52.879547    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f3c5a8af5575e6d8ac29e2a9bd74cd6c2ebd444e7a8b85f52e0511b0b16e10\": container with ID starting with 31f3c5a8af5575e6d8ac29e2a9bd74cd6c2ebd444e7a8b85f52e0511b0b16e10 not found: ID does not exist" containerID="31f3c5a8af5575e6d8ac29e2a9bd74cd6c2ebd444e7a8b85f52e0511b0b16e10"
Sep 16 14:38:52.879620 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.879572    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f3c5a8af5575e6d8ac29e2a9bd74cd6c2ebd444e7a8b85f52e0511b0b16e10"} err="failed to get container status \"31f3c5a8af5575e6d8ac29e2a9bd74cd6c2ebd444e7a8b85f52e0511b0b16e10\": rpc error: code = NotFound desc = could not find container \"31f3c5a8af5575e6d8ac29e2a9bd74cd6c2ebd444e7a8b85f52e0511b0b16e10\": container with ID starting with 31f3c5a8af5575e6d8ac29e2a9bd74cd6c2ebd444e7a8b85f52e0511b0b16e10 not found: ID does not exist"
Sep 16 14:38:52.879620 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.879590    2553 scope.go:117] "RemoveContainer" containerID="3e977250f209ca917a916cf65488eafde3baea544acde0f8f42a7255677d0be2"
Sep 16 14:38:52.879834 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:38:52.879816    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e977250f209ca917a916cf65488eafde3baea544acde0f8f42a7255677d0be2\": container with ID starting with 3e977250f209ca917a916cf65488eafde3baea544acde0f8f42a7255677d0be2 not found: ID does not exist" containerID="3e977250f209ca917a916cf65488eafde3baea544acde0f8f42a7255677d0be2"
Sep 16 14:38:52.879874 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.879840    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e977250f209ca917a916cf65488eafde3baea544acde0f8f42a7255677d0be2"} err="failed to get container status \"3e977250f209ca917a916cf65488eafde3baea544acde0f8f42a7255677d0be2\": rpc error: code = NotFound desc = could not find container \"3e977250f209ca917a916cf65488eafde3baea544acde0f8f42a7255677d0be2\": container with ID starting with 3e977250f209ca917a916cf65488eafde3baea544acde0f8f42a7255677d0be2 not found: ID does not exist"
Sep 16 14:38:52.884036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.884014    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"]
Sep 16 14:38:52.887839 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:52.887819    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7b12b9e3-openshift-default-554c95f56d-4fpp6"]
Sep 16 14:38:53.917702 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:38:53.917669    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" path="/var/lib/kubelet/pods/b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f/volumes"
Sep 16 14:39:09.953762 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:39:09.953728    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:39:09.958720 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:39:09.958692    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:39:10.120599 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:39:10.120560    2553 scope.go:117] "RemoveContainer" containerID="4c02d364ac0d10ed88ef1c81edc599b6b7454c58914c645c4384ed15f8a00b33"
Sep 16 14:40:05.341784 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:05.341745    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"]
Sep 16 14:40:05.377874 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:05.377842    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"]
Sep 16 14:40:08.734190 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.734157    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"]
Sep 16 14:40:08.734877 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.734858    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" containerName="istio-proxy"
Sep 16 14:40:08.734984 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.734880    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" containerName="istio-proxy"
Sep 16 14:40:08.735023 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.735012    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" containerName="istio-proxy"
Sep 16 14:40:08.735058 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.735025    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" containerName="istio-proxy"
Sep 16 14:40:08.738603 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.738585    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.741385 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.741365    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-71d83f-openshift-default-dockercfg-ns8h6\""
Sep 16 14:40:08.749639 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.749615    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"]
Sep 16 14:40:08.788451 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.788423    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922143eb-a7d5-4910-b16b-a490824fdfed-istiod-ca-cert\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.788604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.788486    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-credential-socket\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.788604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.788548    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922143eb-a7d5-4910-b16b-a490824fdfed-istio-podinfo\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.788604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.788576    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8xt9\" (UniqueName: \"kubernetes.io/projected/922143eb-a7d5-4910-b16b-a490824fdfed-kube-api-access-x8xt9\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.788716 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.788618    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-workload-certs\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.788716 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.788662    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922143eb-a7d5-4910-b16b-a490824fdfed-istio-token\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.788716 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.788691    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-workload-socket\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.788813 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.788718    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-istio-data\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.788813 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.788738    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-istio-envoy\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.889494 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.889459    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-istio-data\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.889494 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.889500    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-istio-envoy\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.889700 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.889543    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922143eb-a7d5-4910-b16b-a490824fdfed-istiod-ca-cert\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.889700 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.889619    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-credential-socket\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.889700 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.889665    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922143eb-a7d5-4910-b16b-a490824fdfed-istio-podinfo\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.889700 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.889696    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8xt9\" (UniqueName: \"kubernetes.io/projected/922143eb-a7d5-4910-b16b-a490824fdfed-kube-api-access-x8xt9\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.889944 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.889753    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-workload-certs\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.889944 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.889808    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922143eb-a7d5-4910-b16b-a490824fdfed-istio-token\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.889944 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.889847    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-workload-socket\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.890118 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.889963    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-istio-data\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.890242 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.890211    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-workload-certs\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.890301 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.890255    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-workload-socket\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.890419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.890399    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-credential-socket\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.890471 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.890421    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922143eb-a7d5-4910-b16b-a490824fdfed-istiod-ca-cert\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.891933 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.891877    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-istio-envoy\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.892169 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.892150    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922143eb-a7d5-4910-b16b-a490824fdfed-istio-podinfo\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.897736 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.897707    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922143eb-a7d5-4910-b16b-a490824fdfed-istio-token\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:08.897944 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:08.897929    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8xt9\" (UniqueName: \"kubernetes.io/projected/922143eb-a7d5-4910-b16b-a490824fdfed-kube-api-access-x8xt9\") pod \"e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") " pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:09.050091 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:09.050009    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:09.336067 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:09.336035    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"]
Sep 16 14:40:09.343099 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:09.343052    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:40:09.343234 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:09.343140    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:40:09.343234 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:09.343180    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:40:10.296986 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:10.296947    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh" event={"ID":"922143eb-a7d5-4910-b16b-a490824fdfed","Type":"ContainerStarted","Data":"9725ac2f26ffe82a4e1da1db8ef5cb6475211fb190ba6e344222d67c619e74d1"}
Sep 16 14:40:10.297430 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:10.297020    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh" event={"ID":"922143eb-a7d5-4910-b16b-a490824fdfed","Type":"ContainerStarted","Data":"a0f8819f3fcd6ac80cb4c8e81072ded72ad6a03bea82f1aa49c3ddc38c757015"}
Sep 16 14:40:10.321276 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:10.321231    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh" podStartSLOduration=2.321215535 podStartE2EDuration="2.321215535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:40:10.319880643 +0000 UTC m=+1560.866622575" watchObservedRunningTime="2026-09-16 14:40:10.321215535 +0000 UTC m=+1560.867957477"
Sep 16 14:40:11.050751 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:11.050721    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:11.057928 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:11.057882    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:11.303365 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:11.303283    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:11.304288 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:11.304270    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:16.406400 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.406363    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"]
Sep 16 14:40:16.407138 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.407115    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" containerName="istio-proxy"
Sep 16 14:40:16.407226 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.407142    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3860f76-5f0b-4468-bf9b-a8d6cd1a5c0f" containerName="istio-proxy"
Sep 16 14:40:16.411097 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.411076    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:40:16.413648 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.413625    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-71d83f-serving-cert\""
Sep 16 14:40:16.420042 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.420019    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"]
Sep 16 14:40:16.468751 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.468721    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e09621d0-91c1-44da-9f84-9a4761ebc9a6-openshift-service-ca\") pod \"maas-api-e2e-api-a-71d83f-677778d489-jmjmb\" (UID: \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:40:16.468924 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.468765    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e09621d0-91c1-44da-9f84-9a4761ebc9a6-maas-api-tls\") pod \"maas-api-e2e-api-a-71d83f-677778d489-jmjmb\" (UID: \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:40:16.468924 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.468851    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckds8\" (UniqueName: \"kubernetes.io/projected/e09621d0-91c1-44da-9f84-9a4761ebc9a6-kube-api-access-ckds8\") pod \"maas-api-e2e-api-a-71d83f-677778d489-jmjmb\" (UID: \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:40:16.469041 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.468944    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e09621d0-91c1-44da-9f84-9a4761ebc9a6-metrics-certs\") pod \"maas-api-e2e-api-a-71d83f-677778d489-jmjmb\" (UID: \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:40:16.569535 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.569502    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e09621d0-91c1-44da-9f84-9a4761ebc9a6-openshift-service-ca\") pod \"maas-api-e2e-api-a-71d83f-677778d489-jmjmb\" (UID: \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:40:16.569730 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.569561    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e09621d0-91c1-44da-9f84-9a4761ebc9a6-maas-api-tls\") pod \"maas-api-e2e-api-a-71d83f-677778d489-jmjmb\" (UID: \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:40:16.569730 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.569626    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckds8\" (UniqueName: \"kubernetes.io/projected/e09621d0-91c1-44da-9f84-9a4761ebc9a6-kube-api-access-ckds8\") pod \"maas-api-e2e-api-a-71d83f-677778d489-jmjmb\" (UID: \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:40:16.569730 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.569690    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e09621d0-91c1-44da-9f84-9a4761ebc9a6-metrics-certs\") pod \"maas-api-e2e-api-a-71d83f-677778d489-jmjmb\" (UID: \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:40:16.570227 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.570208    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e09621d0-91c1-44da-9f84-9a4761ebc9a6-openshift-service-ca\") pod \"maas-api-e2e-api-a-71d83f-677778d489-jmjmb\" (UID: \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:40:16.572044 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.572016    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e09621d0-91c1-44da-9f84-9a4761ebc9a6-maas-api-tls\") pod \"maas-api-e2e-api-a-71d83f-677778d489-jmjmb\" (UID: \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:40:16.572179 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.572075    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e09621d0-91c1-44da-9f84-9a4761ebc9a6-metrics-certs\") pod \"maas-api-e2e-api-a-71d83f-677778d489-jmjmb\" (UID: \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:40:16.595532 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.595496    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckds8\" (UniqueName: \"kubernetes.io/projected/e09621d0-91c1-44da-9f84-9a4761ebc9a6-kube-api-access-ckds8\") pod \"maas-api-e2e-api-a-71d83f-677778d489-jmjmb\" (UID: \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:40:16.724641 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:16.724604    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:40:17.005268 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:17.005243    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"]
Sep 16 14:40:17.007001 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:40:17.006970    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09621d0_91c1_44da_9f84_9a4761ebc9a6.slice/crio-99ee5e8b1a640c3e903210a76d0097aad2a0050f2eb6e13c2e7aee3d5fb0ec85 WatchSource:0}: Error finding container 99ee5e8b1a640c3e903210a76d0097aad2a0050f2eb6e13c2e7aee3d5fb0ec85: Status 404 returned error can't find the container with id 99ee5e8b1a640c3e903210a76d0097aad2a0050f2eb6e13c2e7aee3d5fb0ec85
Sep 16 14:40:17.337255 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:17.337173    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb" event={"ID":"e09621d0-91c1-44da-9f84-9a4761ebc9a6","Type":"ContainerStarted","Data":"99ee5e8b1a640c3e903210a76d0097aad2a0050f2eb6e13c2e7aee3d5fb0ec85"}
Sep 16 14:40:18.344213 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:18.344174    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb" event={"ID":"e09621d0-91c1-44da-9f84-9a4761ebc9a6","Type":"ContainerStarted","Data":"68d13025b5c110591558a358e632142b451b121cdca955ca3ef53272f4a1aa1f"}
Sep 16 14:40:18.344599 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:18.344406    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:40:18.371778 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:18.371727    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb" podStartSLOduration=1.831974353 podStartE2EDuration="2.371711665s" totalImagesPullingTime="539.737312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:40:18.370435085 +0000 UTC m=+1568.917177017" watchObservedRunningTime="2026-09-16 14:40:18.371711665 +0000 UTC m=+1568.918453783"
Sep 16 14:40:20.362220 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.362180    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q" podUID="696ae56f-a5a2-48a4-8ba7-f26332ca7c09" containerName="main" containerID="cri-o://18c399cf2f22aaddd498b6ae24b34ff12db32069f88b439cf67d90b0fb3acd08" gracePeriod=45
Sep 16 14:40:20.397648 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.397601    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8" podUID="67217cc6-2a5b-4e5a-941a-67d5d5c1fff0" containerName="main" containerID="cri-o://6f063678471b3d8958c482111c200f42af7efaa09054685d20551676f4f8b502" gracePeriod=45
Sep 16 14:40:20.669251 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.669224    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:40:20.682023 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.682001    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:40:20.715786 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.715752    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-tls-certs\") pod \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") "
Sep 16 14:40:20.715941 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.715813    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-tmp-dir\") pod \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") "
Sep 16 14:40:20.715941 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.715838    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-dshm\") pod \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") "
Sep 16 14:40:20.715941 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.715863    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-tmp-dir\") pod \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") "
Sep 16 14:40:20.715941 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.715937    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-model-cache\") pod \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") "
Sep 16 14:40:20.716180 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.715966    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-tls-certs\") pod \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") "
Sep 16 14:40:20.716180 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.716014    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-model-cache\") pod \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") "
Sep 16 14:40:20.716180 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.716099    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-home\") pod \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") "
Sep 16 14:40:20.716180 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.716128    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-dshm\") pod \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") "
Sep 16 14:40:20.716180 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.716159    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24zjv\" (UniqueName: \"kubernetes.io/projected/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-kube-api-access-24zjv\") pod \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\" (UID: \"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0\") "
Sep 16 14:40:20.716438 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.716193    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b5f6\" (UniqueName: \"kubernetes.io/projected/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-kube-api-access-2b5f6\") pod \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") "
Sep 16 14:40:20.716438 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.716226    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-home\") pod \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\" (UID: \"696ae56f-a5a2-48a4-8ba7-f26332ca7c09\") "
Sep 16 14:40:20.716438 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.716375    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-model-cache" (OuterVolumeSpecName: "model-cache") pod "696ae56f-a5a2-48a4-8ba7-f26332ca7c09" (UID: "696ae56f-a5a2-48a4-8ba7-f26332ca7c09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:40:20.716908 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.716607    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-home" (OuterVolumeSpecName: "home") pod "696ae56f-a5a2-48a4-8ba7-f26332ca7c09" (UID: "696ae56f-a5a2-48a4-8ba7-f26332ca7c09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:40:20.716908 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.716703    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-model-cache\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.716908 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.716720    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-home\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.717137 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.716920    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67217cc6-2a5b-4e5a-941a-67d5d5c1fff0" (UID: "67217cc6-2a5b-4e5a-941a-67d5d5c1fff0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:40:20.717662 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.717580    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-model-cache" (OuterVolumeSpecName: "model-cache") pod "67217cc6-2a5b-4e5a-941a-67d5d5c1fff0" (UID: "67217cc6-2a5b-4e5a-941a-67d5d5c1fff0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:40:20.717948 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.717906    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-home" (OuterVolumeSpecName: "home") pod "67217cc6-2a5b-4e5a-941a-67d5d5c1fff0" (UID: "67217cc6-2a5b-4e5a-941a-67d5d5c1fff0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:40:20.718116 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.718080    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "696ae56f-a5a2-48a4-8ba7-f26332ca7c09" (UID: "696ae56f-a5a2-48a4-8ba7-f26332ca7c09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:40:20.719996 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.719973    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-dshm" (OuterVolumeSpecName: "dshm") pod "696ae56f-a5a2-48a4-8ba7-f26332ca7c09" (UID: "696ae56f-a5a2-48a4-8ba7-f26332ca7c09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:40:20.720165 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.720142    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "696ae56f-a5a2-48a4-8ba7-f26332ca7c09" (UID: "696ae56f-a5a2-48a4-8ba7-f26332ca7c09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:40:20.720276 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.720175    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-dshm" (OuterVolumeSpecName: "dshm") pod "67217cc6-2a5b-4e5a-941a-67d5d5c1fff0" (UID: "67217cc6-2a5b-4e5a-941a-67d5d5c1fff0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:40:20.720276 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.720177    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-kube-api-access-24zjv" (OuterVolumeSpecName: "kube-api-access-24zjv") pod "67217cc6-2a5b-4e5a-941a-67d5d5c1fff0" (UID: "67217cc6-2a5b-4e5a-941a-67d5d5c1fff0"). InnerVolumeSpecName "kube-api-access-24zjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:40:20.721050 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.721032    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67217cc6-2a5b-4e5a-941a-67d5d5c1fff0" (UID: "67217cc6-2a5b-4e5a-941a-67d5d5c1fff0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:40:20.721179 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.721162    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-kube-api-access-2b5f6" (OuterVolumeSpecName: "kube-api-access-2b5f6") pod "696ae56f-a5a2-48a4-8ba7-f26332ca7c09" (UID: "696ae56f-a5a2-48a4-8ba7-f26332ca7c09"). InnerVolumeSpecName "kube-api-access-2b5f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:40:20.817983 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.817946    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-tls-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.817983 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.817975    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-model-cache\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.817983 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.817984    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-home\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.817983 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.817991    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-dshm\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.818240 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.818000    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24zjv\" (UniqueName: \"kubernetes.io/projected/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-kube-api-access-24zjv\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.818240 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.818008    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b5f6\" (UniqueName: \"kubernetes.io/projected/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-kube-api-access-2b5f6\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.818240 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.818016    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-tls-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.818240 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.818024    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-tmp-dir\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.818240 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.818031    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0-dshm\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.818240 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:20.818038    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696ae56f-a5a2-48a4-8ba7-f26332ca7c09-tmp-dir\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:40:21.283651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.283616    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"]
Sep 16 14:40:21.284719 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.283936    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6" podUID="fb3dc797-c169-49f0-a387-b6760e6a1f80" containerName="payload-processing" containerID="cri-o://c699ec3265b52db13c0125baff66ff92e196a95c2d1ce6628c2421e50c73d3d3" gracePeriod=30
Sep 16 14:40:21.362774 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.362743    2553 generic.go:356] "Generic (PLEG): container finished" podID="696ae56f-a5a2-48a4-8ba7-f26332ca7c09" containerID="18c399cf2f22aaddd498b6ae24b34ff12db32069f88b439cf67d90b0fb3acd08" exitCode=0
Sep 16 14:40:21.363259 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.362786    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"
Sep 16 14:40:21.363259 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.362825    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q" event={"ID":"696ae56f-a5a2-48a4-8ba7-f26332ca7c09","Type":"ContainerDied","Data":"18c399cf2f22aaddd498b6ae24b34ff12db32069f88b439cf67d90b0fb3acd08"}
Sep 16 14:40:21.363259 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.362863    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q" event={"ID":"696ae56f-a5a2-48a4-8ba7-f26332ca7c09","Type":"ContainerDied","Data":"07e6c7382082d1239d1ce63fa7b2e6a9aac34c9657f9ab13de18db26e0024147"}
Sep 16 14:40:21.363259 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.362883    2553 scope.go:117] "RemoveContainer" containerID="18c399cf2f22aaddd498b6ae24b34ff12db32069f88b439cf67d90b0fb3acd08"
Sep 16 14:40:21.364375 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.364354    2553 generic.go:356] "Generic (PLEG): container finished" podID="67217cc6-2a5b-4e5a-941a-67d5d5c1fff0" containerID="6f063678471b3d8958c482111c200f42af7efaa09054685d20551676f4f8b502" exitCode=0
Sep 16 14:40:21.364451 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.364388    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"
Sep 16 14:40:21.364451 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.364406    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8" event={"ID":"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0","Type":"ContainerDied","Data":"6f063678471b3d8958c482111c200f42af7efaa09054685d20551676f4f8b502"}
Sep 16 14:40:21.364451 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.364433    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8" event={"ID":"67217cc6-2a5b-4e5a-941a-67d5d5c1fff0","Type":"ContainerDied","Data":"7b8e08e1737bc90228f331a51ca22009fb9473f0a2e9e698f05d4959efc97fcd"}
Sep 16 14:40:21.397099 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.397079    2553 scope.go:117] "RemoveContainer" containerID="18c399cf2f22aaddd498b6ae24b34ff12db32069f88b439cf67d90b0fb3acd08"
Sep 16 14:40:21.397473 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:40:21.397417    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c399cf2f22aaddd498b6ae24b34ff12db32069f88b439cf67d90b0fb3acd08\": container with ID starting with 18c399cf2f22aaddd498b6ae24b34ff12db32069f88b439cf67d90b0fb3acd08 not found: ID does not exist" containerID="18c399cf2f22aaddd498b6ae24b34ff12db32069f88b439cf67d90b0fb3acd08"
Sep 16 14:40:21.397473 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.397463    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c399cf2f22aaddd498b6ae24b34ff12db32069f88b439cf67d90b0fb3acd08"} err="failed to get container status \"18c399cf2f22aaddd498b6ae24b34ff12db32069f88b439cf67d90b0fb3acd08\": rpc error: code = NotFound desc = could not find container \"18c399cf2f22aaddd498b6ae24b34ff12db32069f88b439cf67d90b0fb3acd08\": container with ID starting with 18c399cf2f22aaddd498b6ae24b34ff12db32069f88b439cf67d90b0fb3acd08 not found: ID does not exist"
Sep 16 14:40:21.397768 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.397487    2553 scope.go:117] "RemoveContainer" containerID="6f063678471b3d8958c482111c200f42af7efaa09054685d20551676f4f8b502"
Sep 16 14:40:21.399177 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.399158    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"]
Sep 16 14:40:21.401465 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.401439    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-924916/distinct-2-w0-924916-kserve-7f9bcd755-zc98q"]
Sep 16 14:40:21.406248 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.406222    2553 scope.go:117] "RemoveContainer" containerID="6f063678471b3d8958c482111c200f42af7efaa09054685d20551676f4f8b502"
Sep 16 14:40:21.406530 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:40:21.406511    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f063678471b3d8958c482111c200f42af7efaa09054685d20551676f4f8b502\": container with ID starting with 6f063678471b3d8958c482111c200f42af7efaa09054685d20551676f4f8b502 not found: ID does not exist" containerID="6f063678471b3d8958c482111c200f42af7efaa09054685d20551676f4f8b502"
Sep 16 14:40:21.406602 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.406539    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f063678471b3d8958c482111c200f42af7efaa09054685d20551676f4f8b502"} err="failed to get container status \"6f063678471b3d8958c482111c200f42af7efaa09054685d20551676f4f8b502\": rpc error: code = NotFound desc = could not find container \"6f063678471b3d8958c482111c200f42af7efaa09054685d20551676f4f8b502\": container with ID starting with 6f063678471b3d8958c482111c200f42af7efaa09054685d20551676f4f8b502 not found: ID does not exist"
Sep 16 14:40:21.411505 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.411478    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"]
Sep 16 14:40:21.417308 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.417286    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-924916/simulator-w0-924916-kserve-7b4b8cc696-mt7q8"]
Sep 16 14:40:21.564929 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.564885    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"
Sep 16 14:40:21.627529 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.627500    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmmdn\" (UniqueName: \"kubernetes.io/projected/fb3dc797-c169-49f0-a387-b6760e6a1f80-kube-api-access-qmmdn\") pod \"fb3dc797-c169-49f0-a387-b6760e6a1f80\" (UID: \"fb3dc797-c169-49f0-a387-b6760e6a1f80\") "
Sep 16 14:40:21.627688 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.627621    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3dc797-c169-49f0-a387-b6760e6a1f80-plugins-config-volume\") pod \"fb3dc797-c169-49f0-a387-b6760e6a1f80\" (UID: \"fb3dc797-c169-49f0-a387-b6760e6a1f80\") "
Sep 16 14:40:21.627961 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.627939    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb3dc797-c169-49f0-a387-b6760e6a1f80-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb3dc797-c169-49f0-a387-b6760e6a1f80" (UID: "fb3dc797-c169-49f0-a387-b6760e6a1f80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:40:21.629595 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.629572    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3dc797-c169-49f0-a387-b6760e6a1f80-kube-api-access-qmmdn" (OuterVolumeSpecName: "kube-api-access-qmmdn") pod "fb3dc797-c169-49f0-a387-b6760e6a1f80" (UID: "fb3dc797-c169-49f0-a387-b6760e6a1f80"). InnerVolumeSpecName "kube-api-access-qmmdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:40:21.729546 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.729515    2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3dc797-c169-49f0-a387-b6760e6a1f80-plugins-config-volume\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:40:21.729546 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.729546    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmmdn\" (UniqueName: \"kubernetes.io/projected/fb3dc797-c169-49f0-a387-b6760e6a1f80-kube-api-access-qmmdn\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:40:21.918141 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.918057    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67217cc6-2a5b-4e5a-941a-67d5d5c1fff0" path="/var/lib/kubelet/pods/67217cc6-2a5b-4e5a-941a-67d5d5c1fff0/volumes"
Sep 16 14:40:21.918495 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:21.918481    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="696ae56f-a5a2-48a4-8ba7-f26332ca7c09" path="/var/lib/kubelet/pods/696ae56f-a5a2-48a4-8ba7-f26332ca7c09/volumes"
Sep 16 14:40:22.372941 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:22.372883    2553 generic.go:356] "Generic (PLEG): container finished" podID="fb3dc797-c169-49f0-a387-b6760e6a1f80" containerID="c699ec3265b52db13c0125baff66ff92e196a95c2d1ce6628c2421e50c73d3d3" exitCode=0
Sep 16 14:40:22.372941 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:22.372942    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"
Sep 16 14:40:22.373455 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:22.372983    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6" event={"ID":"fb3dc797-c169-49f0-a387-b6760e6a1f80","Type":"ContainerDied","Data":"c699ec3265b52db13c0125baff66ff92e196a95c2d1ce6628c2421e50c73d3d3"}
Sep 16 14:40:22.373455 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:22.373031    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6" event={"ID":"fb3dc797-c169-49f0-a387-b6760e6a1f80","Type":"ContainerDied","Data":"648333514baded8bda6208e074e14ebc32a625bc2f6a4f4d590c04e633c5b5e9"}
Sep 16 14:40:22.373455 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:22.373049    2553 scope.go:117] "RemoveContainer" containerID="c699ec3265b52db13c0125baff66ff92e196a95c2d1ce6628c2421e50c73d3d3"
Sep 16 14:40:22.382294 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:22.382276    2553 scope.go:117] "RemoveContainer" containerID="c699ec3265b52db13c0125baff66ff92e196a95c2d1ce6628c2421e50c73d3d3"
Sep 16 14:40:22.382614 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:40:22.382584    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c699ec3265b52db13c0125baff66ff92e196a95c2d1ce6628c2421e50c73d3d3\": container with ID starting with c699ec3265b52db13c0125baff66ff92e196a95c2d1ce6628c2421e50c73d3d3 not found: ID does not exist" containerID="c699ec3265b52db13c0125baff66ff92e196a95c2d1ce6628c2421e50c73d3d3"
Sep 16 14:40:22.382710 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:22.382610    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c699ec3265b52db13c0125baff66ff92e196a95c2d1ce6628c2421e50c73d3d3"} err="failed to get container status \"c699ec3265b52db13c0125baff66ff92e196a95c2d1ce6628c2421e50c73d3d3\": rpc error: code = NotFound desc = could not find container \"c699ec3265b52db13c0125baff66ff92e196a95c2d1ce6628c2421e50c73d3d3\": container with ID starting with c699ec3265b52db13c0125baff66ff92e196a95c2d1ce6628c2421e50c73d3d3 not found: ID does not exist"
Sep 16 14:40:22.396129 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:22.396088    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"]
Sep 16 14:40:22.399127 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:22.399095    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-924916-75cdc8867c-j6gk6"]
Sep 16 14:40:23.918789 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:23.918754    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb3dc797-c169-49f0-a387-b6760e6a1f80" path="/var/lib/kubelet/pods/fb3dc797-c169-49f0-a387-b6760e6a1f80/volumes"
Sep 16 14:40:24.356288 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:24.356261    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:40:33.255609 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.255581    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"]
Sep 16 14:40:33.256257 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.256241    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67217cc6-2a5b-4e5a-941a-67d5d5c1fff0" containerName="main"
Sep 16 14:40:33.256305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.256262    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="67217cc6-2a5b-4e5a-941a-67d5d5c1fff0" containerName="main"
Sep 16 14:40:33.256340 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.256306    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="696ae56f-a5a2-48a4-8ba7-f26332ca7c09" containerName="main"
Sep 16 14:40:33.256340 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.256314    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="696ae56f-a5a2-48a4-8ba7-f26332ca7c09" containerName="main"
Sep 16 14:40:33.256340 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.256325    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb3dc797-c169-49f0-a387-b6760e6a1f80" containerName="payload-processing"
Sep 16 14:40:33.256340 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.256333    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb3dc797-c169-49f0-a387-b6760e6a1f80" containerName="payload-processing"
Sep 16 14:40:33.256466 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.256437    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="67217cc6-2a5b-4e5a-941a-67d5d5c1fff0" containerName="main"
Sep 16 14:40:33.256466 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.256449    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb3dc797-c169-49f0-a387-b6760e6a1f80" containerName="payload-processing"
Sep 16 14:40:33.256466 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.256456    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="696ae56f-a5a2-48a4-8ba7-f26332ca7c09" containerName="main"
Sep 16 14:40:33.261292 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.261276    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.265309 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.265287    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f891c2-openshift-default-dockercfg-v85h7\""
Sep 16 14:40:33.281554 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.281529    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"]
Sep 16 14:40:33.343334 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.343302    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istiod-ca-cert\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.343334 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.343337    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf67q\" (UniqueName: \"kubernetes.io/projected/8ce02c4a-beac-4039-a0f5-c72095b2bca4-kube-api-access-sf67q\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.343579 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.343363    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-workload-socket\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.343579 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.343442    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-data\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.343579 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.343522    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-credential-socket\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.343579 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.343561    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-workload-certs\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.343750 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.343594    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-token\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.343750 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.343687    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-envoy\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.343750 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.343721    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-podinfo\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.445151 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.445119    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-envoy\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.445334 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.445162    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-podinfo\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.445334 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.445203    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istiod-ca-cert\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.445334 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.445221    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf67q\" (UniqueName: \"kubernetes.io/projected/8ce02c4a-beac-4039-a0f5-c72095b2bca4-kube-api-access-sf67q\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.445334 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.445242    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-workload-socket\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.445334 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.445270    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-data\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.445334 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.445322    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-credential-socket\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.445671 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.445371    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-workload-certs\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.445671 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.445419    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-token\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.445671 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.445630    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-workload-socket\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.445853 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.445741    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-data\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.445853 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.445833    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-credential-socket\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.446052 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.446033    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istiod-ca-cert\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.446119 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.446097    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-workload-certs\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.447410 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.447381    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-envoy\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.447750 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.447727    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-podinfo\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.456611 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.456448    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-token\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.457081 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.457055    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf67q\" (UniqueName: \"kubernetes.io/projected/8ce02c4a-beac-4039-a0f5-c72095b2bca4-kube-api-access-sf67q\") pod \"e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") " pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.573562 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.573171    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:33.857221 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.856069    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"]
Sep 16 14:40:33.873196 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.872624    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:40:33.873196 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.872717    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:40:33.873196 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:33.872760    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:40:34.448938 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:34.448902    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt" event={"ID":"8ce02c4a-beac-4039-a0f5-c72095b2bca4","Type":"ContainerStarted","Data":"abd3abdd452b4caeb5e5a6159eba2459abb5abf5e14bf920618275301dc92f65"}
Sep 16 14:40:34.449326 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:34.448959    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt" event={"ID":"8ce02c4a-beac-4039-a0f5-c72095b2bca4","Type":"ContainerStarted","Data":"8659716817c426ed048e1684a58c8e712bb0e0e2f3bf970b42a176a3c4dc3165"}
Sep 16 14:40:34.509399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:34.509345    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt" podStartSLOduration=1.509328102 podStartE2EDuration="1.509328102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:40:34.506846344 +0000 UTC m=+1585.053588274" watchObservedRunningTime="2026-09-16 14:40:34.509328102 +0000 UTC m=+1585.056070033"
Sep 16 14:40:34.574044 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:34.574009    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:34.580979 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:34.580953    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:35.458348 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:35.458310    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:35.464561 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:35.464529    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:40:41.386424 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.386366    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"]
Sep 16 14:40:41.390938 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.390913    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:40:41.396168 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.396146    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f891c2-serving-cert\""
Sep 16 14:40:41.405587 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.405566    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"]
Sep 16 14:40:41.439427 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.439392    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b10c961-55a0-4284-94e4-cb5fa20a16c5-metrics-certs\") pod \"maas-api-e2e-api-b-f891c2-64f84d65d4-sh889\" (UID: \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:40:41.439560 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.439435    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glvz5\" (UniqueName: \"kubernetes.io/projected/9b10c961-55a0-4284-94e4-cb5fa20a16c5-kube-api-access-glvz5\") pod \"maas-api-e2e-api-b-f891c2-64f84d65d4-sh889\" (UID: \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:40:41.439560 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.439473    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b10c961-55a0-4284-94e4-cb5fa20a16c5-openshift-service-ca\") pod \"maas-api-e2e-api-b-f891c2-64f84d65d4-sh889\" (UID: \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:40:41.439560 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.439515    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10c961-55a0-4284-94e4-cb5fa20a16c5-maas-api-tls\") pod \"maas-api-e2e-api-b-f891c2-64f84d65d4-sh889\" (UID: \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:40:41.540455 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.540415    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b10c961-55a0-4284-94e4-cb5fa20a16c5-openshift-service-ca\") pod \"maas-api-e2e-api-b-f891c2-64f84d65d4-sh889\" (UID: \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:40:41.540644 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.540517    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10c961-55a0-4284-94e4-cb5fa20a16c5-maas-api-tls\") pod \"maas-api-e2e-api-b-f891c2-64f84d65d4-sh889\" (UID: \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:40:41.540717 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.540700    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b10c961-55a0-4284-94e4-cb5fa20a16c5-metrics-certs\") pod \"maas-api-e2e-api-b-f891c2-64f84d65d4-sh889\" (UID: \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:40:41.540782 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.540743    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glvz5\" (UniqueName: \"kubernetes.io/projected/9b10c961-55a0-4284-94e4-cb5fa20a16c5-kube-api-access-glvz5\") pod \"maas-api-e2e-api-b-f891c2-64f84d65d4-sh889\" (UID: \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:40:41.541149 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.541124    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b10c961-55a0-4284-94e4-cb5fa20a16c5-openshift-service-ca\") pod \"maas-api-e2e-api-b-f891c2-64f84d65d4-sh889\" (UID: \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:40:41.543021 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.543002    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10c961-55a0-4284-94e4-cb5fa20a16c5-maas-api-tls\") pod \"maas-api-e2e-api-b-f891c2-64f84d65d4-sh889\" (UID: \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:40:41.543136 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.543038    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b10c961-55a0-4284-94e4-cb5fa20a16c5-metrics-certs\") pod \"maas-api-e2e-api-b-f891c2-64f84d65d4-sh889\" (UID: \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:40:41.554272 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.554248    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glvz5\" (UniqueName: \"kubernetes.io/projected/9b10c961-55a0-4284-94e4-cb5fa20a16c5-kube-api-access-glvz5\") pod \"maas-api-e2e-api-b-f891c2-64f84d65d4-sh889\" (UID: \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:40:41.705627 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:41.705513    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:40:42.001201 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:42.001176    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"]
Sep 16 14:40:42.003394 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:40:42.003365    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b10c961_55a0_4284_94e4_cb5fa20a16c5.slice/crio-7de8d2ac2b5147f989aaf58a28191698836926e771f34329f6c5ff0b2ba4a73d WatchSource:0}: Error finding container 7de8d2ac2b5147f989aaf58a28191698836926e771f34329f6c5ff0b2ba4a73d: Status 404 returned error can't find the container with id 7de8d2ac2b5147f989aaf58a28191698836926e771f34329f6c5ff0b2ba4a73d
Sep 16 14:40:42.004938 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:42.004909    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:40:42.328012 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:42.327984    2553 patch_prober.go:28] interesting pod/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 14:40:42.328176 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:42.328042    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh" podUID="922143eb-a7d5-4910-b16b-a490824fdfed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 14:40:42.517197 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:42.517098    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889" event={"ID":"9b10c961-55a0-4284-94e4-cb5fa20a16c5","Type":"ContainerStarted","Data":"0a9d3ae83c2faecc9b5dcc26b84cb5d93403e6e62c2e798d1d8b14cf26611af1"}
Sep 16 14:40:42.517197 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:42.517176    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:40:42.517676 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:42.517206    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889" event={"ID":"9b10c961-55a0-4284-94e4-cb5fa20a16c5","Type":"ContainerStarted","Data":"7de8d2ac2b5147f989aaf58a28191698836926e771f34329f6c5ff0b2ba4a73d"}
Sep 16 14:40:42.518818 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:42.518792    2553 generic.go:356] "Generic (PLEG): container finished" podID="922143eb-a7d5-4910-b16b-a490824fdfed" containerID="9725ac2f26ffe82a4e1da1db8ef5cb6475211fb190ba6e344222d67c619e74d1" exitCode=0
Sep 16 14:40:42.518923 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:42.518863    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh" event={"ID":"922143eb-a7d5-4910-b16b-a490824fdfed","Type":"ContainerDied","Data":"9725ac2f26ffe82a4e1da1db8ef5cb6475211fb190ba6e344222d67c619e74d1"}
Sep 16 14:40:42.519241 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:42.519227    2553 scope.go:117] "RemoveContainer" containerID="9725ac2f26ffe82a4e1da1db8ef5cb6475211fb190ba6e344222d67c619e74d1"
Sep 16 14:40:42.520329 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:42.520303    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:40:42.520441 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:42.520381    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:40:42.520505 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:42.520456    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:40:42.542154 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:42.542094    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889" podStartSLOduration=1.230977395 podStartE2EDuration="1.542079612s" totalImagesPullingTime="311.102217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:40:42.540940891 +0000 UTC m=+1593.087682823" watchObservedRunningTime="2026-09-16 14:40:42.542079612 +0000 UTC m=+1593.088821575"
Sep 16 14:40:43.526504 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:43.526471    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh" event={"ID":"922143eb-a7d5-4910-b16b-a490824fdfed","Type":"ContainerStarted","Data":"fa09c807e70210a1b57975c62e5e7b6769f4b65ef174e23f7e0ba3e06478a9fc"}
Sep 16 14:40:44.050774 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:44.050733    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:44.051012 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:44.050972    2553 patch_prober.go:28] interesting pod/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 16 14:40:44.051098 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:44.051035    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh" podUID="922143eb-a7d5-4910-b16b-a490824fdfed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 16 14:40:45.051374 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:45.051332    2553 patch_prober.go:28] interesting pod/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 16 14:40:45.051839 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:45.051414    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh" podUID="922143eb-a7d5-4910-b16b-a490824fdfed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 16 14:40:46.050431 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:46.050398    2553 patch_prober.go:28] interesting pod/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 16 14:40:46.050595 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:46.050453    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh" podUID="922143eb-a7d5-4910-b16b-a490824fdfed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 16 14:40:47.051102 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:47.051054    2553 patch_prober.go:28] interesting pod/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 16 14:40:47.051630 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:47.051126    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh" podUID="922143eb-a7d5-4910-b16b-a490824fdfed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 16 14:40:48.050684 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:48.050649    2553 patch_prober.go:28] interesting pod/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 16 14:40:48.050859 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:48.050702    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh" podUID="922143eb-a7d5-4910-b16b-a490824fdfed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 16 14:40:48.532561 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:48.532529    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:40:49.050382 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:49.050350    2553 patch_prober.go:28] interesting pod/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 16 14:40:49.050551 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:49.050415    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh" podUID="922143eb-a7d5-4910-b16b-a490824fdfed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 16 14:40:50.054132 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:50.054102    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:50.054609 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:50.054549    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:40:50.055358 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:40:50.055337    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:41:06.500049 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:06.500009    2553 patch_prober.go:28] interesting pod/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 14:41:06.500469 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:06.500083    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt" podUID="8ce02c4a-beac-4039-a0f5-c72095b2bca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 14:41:07.676079 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:07.676044    2553 generic.go:356] "Generic (PLEG): container finished" podID="8ce02c4a-beac-4039-a0f5-c72095b2bca4" containerID="abd3abdd452b4caeb5e5a6159eba2459abb5abf5e14bf920618275301dc92f65" exitCode=0
Sep 16 14:41:07.676474 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:07.676122    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt" event={"ID":"8ce02c4a-beac-4039-a0f5-c72095b2bca4","Type":"ContainerDied","Data":"abd3abdd452b4caeb5e5a6159eba2459abb5abf5e14bf920618275301dc92f65"}
Sep 16 14:41:07.676556 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:07.676540    2553 scope.go:117] "RemoveContainer" containerID="abd3abdd452b4caeb5e5a6159eba2459abb5abf5e14bf920618275301dc92f65"
Sep 16 14:41:07.677911 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:07.677861    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:41:07.677986 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:07.677966    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:41:07.678037 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:07.678021    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:41:08.684876 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:08.684843    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt" event={"ID":"8ce02c4a-beac-4039-a0f5-c72095b2bca4","Type":"ContainerStarted","Data":"6bc2c77bc06dbed686ab2e514a81168168c9cf1f196c442c74d742bf6a5ea27c"}
Sep 16 14:41:09.574553 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:09.574510    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:41:09.574880 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:09.574671    2553 patch_prober.go:28] interesting pod/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 16 14:41:09.575101 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:09.575061    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt" podUID="8ce02c4a-beac-4039-a0f5-c72095b2bca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 16 14:41:10.574376 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:10.574337    2553 patch_prober.go:28] interesting pod/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 16 14:41:10.574851 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:10.574406    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt" podUID="8ce02c4a-beac-4039-a0f5-c72095b2bca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 16 14:41:11.574853 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:11.574800    2553 patch_prober.go:28] interesting pod/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 16 14:41:11.575392 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:11.574907    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt" podUID="8ce02c4a-beac-4039-a0f5-c72095b2bca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 16 14:41:12.574461 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:12.574414    2553 patch_prober.go:28] interesting pod/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 16 14:41:12.574660 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:12.574495    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt" podUID="8ce02c4a-beac-4039-a0f5-c72095b2bca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 16 14:41:13.574838 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:13.574792    2553 patch_prober.go:28] interesting pod/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 16 14:41:13.575360 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:13.574861    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt" podUID="8ce02c4a-beac-4039-a0f5-c72095b2bca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 16 14:41:14.578052 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:14.578021    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:41:14.578450 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:14.578428    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:41:14.579164 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:14.579142    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:41:25.632623 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:25.631938    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"]
Sep 16 14:41:25.632623 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:25.632389    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb" podUID="e09621d0-91c1-44da-9f84-9a4761ebc9a6" containerName="maas-api" containerID="cri-o://68d13025b5c110591558a358e632142b451b121cdca955ca3ef53272f4a1aa1f" gracePeriod=30
Sep 16 14:41:25.788516 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:25.788486    2553 generic.go:356] "Generic (PLEG): container finished" podID="e09621d0-91c1-44da-9f84-9a4761ebc9a6" containerID="68d13025b5c110591558a358e632142b451b121cdca955ca3ef53272f4a1aa1f" exitCode=0
Sep 16 14:41:25.788680 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:25.788549    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb" event={"ID":"e09621d0-91c1-44da-9f84-9a4761ebc9a6","Type":"ContainerDied","Data":"68d13025b5c110591558a358e632142b451b121cdca955ca3ef53272f4a1aa1f"}
Sep 16 14:41:25.889147 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:25.889088    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:41:25.943195 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:25.943161    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e09621d0-91c1-44da-9f84-9a4761ebc9a6-metrics-certs\") pod \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\" (UID: \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\") "
Sep 16 14:41:25.943849 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:25.943813    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckds8\" (UniqueName: \"kubernetes.io/projected/e09621d0-91c1-44da-9f84-9a4761ebc9a6-kube-api-access-ckds8\") pod \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\" (UID: \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\") "
Sep 16 14:41:25.944068 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:25.944052    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e09621d0-91c1-44da-9f84-9a4761ebc9a6-maas-api-tls\") pod \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\" (UID: \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\") "
Sep 16 14:41:25.944198 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:25.944184    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e09621d0-91c1-44da-9f84-9a4761ebc9a6-openshift-service-ca\") pod \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\" (UID: \"e09621d0-91c1-44da-9f84-9a4761ebc9a6\") "
Sep 16 14:41:25.945162 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:25.945139    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09621d0-91c1-44da-9f84-9a4761ebc9a6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e09621d0-91c1-44da-9f84-9a4761ebc9a6" (UID: "e09621d0-91c1-44da-9f84-9a4761ebc9a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:41:25.960881 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:25.959527    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09621d0-91c1-44da-9f84-9a4761ebc9a6-kube-api-access-ckds8" (OuterVolumeSpecName: "kube-api-access-ckds8") pod "e09621d0-91c1-44da-9f84-9a4761ebc9a6" (UID: "e09621d0-91c1-44da-9f84-9a4761ebc9a6"). InnerVolumeSpecName "kube-api-access-ckds8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:41:25.961422 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:25.961351    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e09621d0-91c1-44da-9f84-9a4761ebc9a6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e09621d0-91c1-44da-9f84-9a4761ebc9a6" (UID: "e09621d0-91c1-44da-9f84-9a4761ebc9a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:41:25.961953 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:25.961915    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e09621d0-91c1-44da-9f84-9a4761ebc9a6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e09621d0-91c1-44da-9f84-9a4761ebc9a6" (UID: "e09621d0-91c1-44da-9f84-9a4761ebc9a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:41:26.045726 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:26.045693    2553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e09621d0-91c1-44da-9f84-9a4761ebc9a6-metrics-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:26.045950 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:26.045932    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckds8\" (UniqueName: \"kubernetes.io/projected/e09621d0-91c1-44da-9f84-9a4761ebc9a6-kube-api-access-ckds8\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:26.046070 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:26.046056    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e09621d0-91c1-44da-9f84-9a4761ebc9a6-maas-api-tls\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:26.046157 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:26.046146    2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e09621d0-91c1-44da-9f84-9a4761ebc9a6-openshift-service-ca\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:26.795066 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:26.795039    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"
Sep 16 14:41:26.795066 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:26.795052    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb" event={"ID":"e09621d0-91c1-44da-9f84-9a4761ebc9a6","Type":"ContainerDied","Data":"99ee5e8b1a640c3e903210a76d0097aad2a0050f2eb6e13c2e7aee3d5fb0ec85"}
Sep 16 14:41:26.795520 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:26.795108    2553 scope.go:117] "RemoveContainer" containerID="68d13025b5c110591558a358e632142b451b121cdca955ca3ef53272f4a1aa1f"
Sep 16 14:41:26.819961 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:26.819933    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"]
Sep 16 14:41:26.822115 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:26.822095    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71d83f-677778d489-jmjmb"]
Sep 16 14:41:27.918325 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:27.918291    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e09621d0-91c1-44da-9f84-9a4761ebc9a6" path="/var/lib/kubelet/pods/e09621d0-91c1-44da-9f84-9a4761ebc9a6/volumes"
Sep 16 14:41:33.354641 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:33.353703    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"]
Sep 16 14:41:33.354641 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:33.353967    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh" podUID="922143eb-a7d5-4910-b16b-a490824fdfed" containerName="istio-proxy" containerID="cri-o://fa09c807e70210a1b57975c62e5e7b6769f4b65ef174e23f7e0ba3e06478a9fc" gracePeriod=30
Sep 16 14:41:35.057215 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:35.057182    2553 patch_prober.go:28] interesting pod/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 14:41:35.057654 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:35.057256    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh" podUID="922143eb-a7d5-4910-b16b-a490824fdfed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 14:41:38.612734 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.612706    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:41:38.780419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.779794    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922143eb-a7d5-4910-b16b-a490824fdfed-istio-token\") pod \"922143eb-a7d5-4910-b16b-a490824fdfed\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") "
Sep 16 14:41:38.780419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.779854    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-credential-socket\") pod \"922143eb-a7d5-4910-b16b-a490824fdfed\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") "
Sep 16 14:41:38.780419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.779992    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-istio-data\") pod \"922143eb-a7d5-4910-b16b-a490824fdfed\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") "
Sep 16 14:41:38.780419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.780018    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-istio-envoy\") pod \"922143eb-a7d5-4910-b16b-a490824fdfed\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") "
Sep 16 14:41:38.780419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.780048    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922143eb-a7d5-4910-b16b-a490824fdfed-istio-podinfo\") pod \"922143eb-a7d5-4910-b16b-a490824fdfed\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") "
Sep 16 14:41:38.780419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.780082    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922143eb-a7d5-4910-b16b-a490824fdfed-istiod-ca-cert\") pod \"922143eb-a7d5-4910-b16b-a490824fdfed\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") "
Sep 16 14:41:38.780419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.780111    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-workload-socket\") pod \"922143eb-a7d5-4910-b16b-a490824fdfed\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") "
Sep 16 14:41:38.780419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.780164    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8xt9\" (UniqueName: \"kubernetes.io/projected/922143eb-a7d5-4910-b16b-a490824fdfed-kube-api-access-x8xt9\") pod \"922143eb-a7d5-4910-b16b-a490824fdfed\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") "
Sep 16 14:41:38.780419 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.780163    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "922143eb-a7d5-4910-b16b-a490824fdfed" (UID: "922143eb-a7d5-4910-b16b-a490824fdfed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:38.781068 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.780448    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-istio-data" (OuterVolumeSpecName: "istio-data") pod "922143eb-a7d5-4910-b16b-a490824fdfed" (UID: "922143eb-a7d5-4910-b16b-a490824fdfed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:38.781068 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.780734    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "922143eb-a7d5-4910-b16b-a490824fdfed" (UID: "922143eb-a7d5-4910-b16b-a490824fdfed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:38.781068 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.780808    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922143eb-a7d5-4910-b16b-a490824fdfed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "922143eb-a7d5-4910-b16b-a490824fdfed" (UID: "922143eb-a7d5-4910-b16b-a490824fdfed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:41:38.781413 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.781394    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-workload-certs\") pod \"922143eb-a7d5-4910-b16b-a490824fdfed\" (UID: \"922143eb-a7d5-4910-b16b-a490824fdfed\") "
Sep 16 14:41:38.781998 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.781974    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922143eb-a7d5-4910-b16b-a490824fdfed-istiod-ca-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:38.782139 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.782126    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-workload-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:38.782235 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.782222    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-credential-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:38.782398 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.782383    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-istio-data\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:38.782565 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.782440    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "922143eb-a7d5-4910-b16b-a490824fdfed" (UID: "922143eb-a7d5-4910-b16b-a490824fdfed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:38.782880 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.782848    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922143eb-a7d5-4910-b16b-a490824fdfed-istio-token" (OuterVolumeSpecName: "istio-token") pod "922143eb-a7d5-4910-b16b-a490824fdfed" (UID: "922143eb-a7d5-4910-b16b-a490824fdfed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:41:38.783945 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.783907    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/922143eb-a7d5-4910-b16b-a490824fdfed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "922143eb-a7d5-4910-b16b-a490824fdfed" (UID: "922143eb-a7d5-4910-b16b-a490824fdfed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:41:38.784214 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.784187    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922143eb-a7d5-4910-b16b-a490824fdfed-kube-api-access-x8xt9" (OuterVolumeSpecName: "kube-api-access-x8xt9") pod "922143eb-a7d5-4910-b16b-a490824fdfed" (UID: "922143eb-a7d5-4910-b16b-a490824fdfed"). InnerVolumeSpecName "kube-api-access-x8xt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:41:38.784319 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.784288    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "922143eb-a7d5-4910-b16b-a490824fdfed" (UID: "922143eb-a7d5-4910-b16b-a490824fdfed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:38.864376 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.864344    2553 generic.go:356] "Generic (PLEG): container finished" podID="922143eb-a7d5-4910-b16b-a490824fdfed" containerID="fa09c807e70210a1b57975c62e5e7b6769f4b65ef174e23f7e0ba3e06478a9fc" exitCode=0
Sep 16 14:41:38.864511 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.864389    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"
Sep 16 14:41:38.864511 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.864429    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh" event={"ID":"922143eb-a7d5-4910-b16b-a490824fdfed","Type":"ContainerDied","Data":"fa09c807e70210a1b57975c62e5e7b6769f4b65ef174e23f7e0ba3e06478a9fc"}
Sep 16 14:41:38.864511 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.864504    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh" event={"ID":"922143eb-a7d5-4910-b16b-a490824fdfed","Type":"ContainerDied","Data":"a0f8819f3fcd6ac80cb4c8e81072ded72ad6a03bea82f1aa49c3ddc38c757015"}
Sep 16 14:41:38.864650 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.864526    2553 scope.go:117] "RemoveContainer" containerID="fa09c807e70210a1b57975c62e5e7b6769f4b65ef174e23f7e0ba3e06478a9fc"
Sep 16 14:41:38.874742 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.874727    2553 scope.go:117] "RemoveContainer" containerID="9725ac2f26ffe82a4e1da1db8ef5cb6475211fb190ba6e344222d67c619e74d1"
Sep 16 14:41:38.883014 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.882998    2553 scope.go:117] "RemoveContainer" containerID="fa09c807e70210a1b57975c62e5e7b6769f4b65ef174e23f7e0ba3e06478a9fc"
Sep 16 14:41:38.883295 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:41:38.883276    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa09c807e70210a1b57975c62e5e7b6769f4b65ef174e23f7e0ba3e06478a9fc\": container with ID starting with fa09c807e70210a1b57975c62e5e7b6769f4b65ef174e23f7e0ba3e06478a9fc not found: ID does not exist" containerID="fa09c807e70210a1b57975c62e5e7b6769f4b65ef174e23f7e0ba3e06478a9fc"
Sep 16 14:41:38.883351 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.883304    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa09c807e70210a1b57975c62e5e7b6769f4b65ef174e23f7e0ba3e06478a9fc"} err="failed to get container status \"fa09c807e70210a1b57975c62e5e7b6769f4b65ef174e23f7e0ba3e06478a9fc\": rpc error: code = NotFound desc = could not find container \"fa09c807e70210a1b57975c62e5e7b6769f4b65ef174e23f7e0ba3e06478a9fc\": container with ID starting with fa09c807e70210a1b57975c62e5e7b6769f4b65ef174e23f7e0ba3e06478a9fc not found: ID does not exist"
Sep 16 14:41:38.883351 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.883322    2553 scope.go:117] "RemoveContainer" containerID="9725ac2f26ffe82a4e1da1db8ef5cb6475211fb190ba6e344222d67c619e74d1"
Sep 16 14:41:38.883436 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.883402    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922143eb-a7d5-4910-b16b-a490824fdfed-istio-token\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:38.883436 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.883425    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-istio-envoy\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:38.883523 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.883438    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922143eb-a7d5-4910-b16b-a490824fdfed-istio-podinfo\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:38.883523 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.883450    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8xt9\" (UniqueName: \"kubernetes.io/projected/922143eb-a7d5-4910-b16b-a490824fdfed-kube-api-access-x8xt9\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:38.883523 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.883466    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922143eb-a7d5-4910-b16b-a490824fdfed-workload-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:38.883628 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:41:38.883562    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9725ac2f26ffe82a4e1da1db8ef5cb6475211fb190ba6e344222d67c619e74d1\": container with ID starting with 9725ac2f26ffe82a4e1da1db8ef5cb6475211fb190ba6e344222d67c619e74d1 not found: ID does not exist" containerID="9725ac2f26ffe82a4e1da1db8ef5cb6475211fb190ba6e344222d67c619e74d1"
Sep 16 14:41:38.883628 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.883583    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9725ac2f26ffe82a4e1da1db8ef5cb6475211fb190ba6e344222d67c619e74d1"} err="failed to get container status \"9725ac2f26ffe82a4e1da1db8ef5cb6475211fb190ba6e344222d67c619e74d1\": rpc error: code = NotFound desc = could not find container \"9725ac2f26ffe82a4e1da1db8ef5cb6475211fb190ba6e344222d67c619e74d1\": container with ID starting with 9725ac2f26ffe82a4e1da1db8ef5cb6475211fb190ba6e344222d67c619e74d1 not found: ID does not exist"
Sep 16 14:41:38.888310 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.888288    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"]
Sep 16 14:41:38.894601 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:38.894581    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-71d83f-openshift-default-5696c954cd-lkhwh"]
Sep 16 14:41:39.917724 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:39.917694    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="922143eb-a7d5-4910-b16b-a490824fdfed" path="/var/lib/kubelet/pods/922143eb-a7d5-4910-b16b-a490824fdfed/volumes"
Sep 16 14:41:42.760775 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:42.760736    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"]
Sep 16 14:41:52.605400 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.605363    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"]
Sep 16 14:41:52.605967 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.605610    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889" podUID="9b10c961-55a0-4284-94e4-cb5fa20a16c5" containerName="maas-api" containerID="cri-o://0a9d3ae83c2faecc9b5dcc26b84cb5d93403e6e62c2e798d1d8b14cf26611af1" gracePeriod=30
Sep 16 14:41:52.858950 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.858866    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:41:52.927375 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.927347    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10c961-55a0-4284-94e4-cb5fa20a16c5-maas-api-tls\") pod \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\" (UID: \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\") "
Sep 16 14:41:52.927546 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.927414    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b10c961-55a0-4284-94e4-cb5fa20a16c5-metrics-certs\") pod \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\" (UID: \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\") "
Sep 16 14:41:52.927546 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.927466    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glvz5\" (UniqueName: \"kubernetes.io/projected/9b10c961-55a0-4284-94e4-cb5fa20a16c5-kube-api-access-glvz5\") pod \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\" (UID: \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\") "
Sep 16 14:41:52.927546 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.927498    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b10c961-55a0-4284-94e4-cb5fa20a16c5-openshift-service-ca\") pod \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\" (UID: \"9b10c961-55a0-4284-94e4-cb5fa20a16c5\") "
Sep 16 14:41:52.927969 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.927936    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b10c961-55a0-4284-94e4-cb5fa20a16c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9b10c961-55a0-4284-94e4-cb5fa20a16c5" (UID: "9b10c961-55a0-4284-94e4-cb5fa20a16c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:41:52.929586 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.929444    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b10c961-55a0-4284-94e4-cb5fa20a16c5-kube-api-access-glvz5" (OuterVolumeSpecName: "kube-api-access-glvz5") pod "9b10c961-55a0-4284-94e4-cb5fa20a16c5" (UID: "9b10c961-55a0-4284-94e4-cb5fa20a16c5"). InnerVolumeSpecName "kube-api-access-glvz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:41:52.929586 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.929534    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b10c961-55a0-4284-94e4-cb5fa20a16c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9b10c961-55a0-4284-94e4-cb5fa20a16c5" (UID: "9b10c961-55a0-4284-94e4-cb5fa20a16c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:41:52.929759 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.929587    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b10c961-55a0-4284-94e4-cb5fa20a16c5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9b10c961-55a0-4284-94e4-cb5fa20a16c5" (UID: "9b10c961-55a0-4284-94e4-cb5fa20a16c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:41:52.946348 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.946319    2553 generic.go:356] "Generic (PLEG): container finished" podID="9b10c961-55a0-4284-94e4-cb5fa20a16c5" containerID="0a9d3ae83c2faecc9b5dcc26b84cb5d93403e6e62c2e798d1d8b14cf26611af1" exitCode=0
Sep 16 14:41:52.946447 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.946356    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"
Sep 16 14:41:52.946447 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.946364    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889" event={"ID":"9b10c961-55a0-4284-94e4-cb5fa20a16c5","Type":"ContainerDied","Data":"0a9d3ae83c2faecc9b5dcc26b84cb5d93403e6e62c2e798d1d8b14cf26611af1"}
Sep 16 14:41:52.946447 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.946420    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889" event={"ID":"9b10c961-55a0-4284-94e4-cb5fa20a16c5","Type":"ContainerDied","Data":"7de8d2ac2b5147f989aaf58a28191698836926e771f34329f6c5ff0b2ba4a73d"}
Sep 16 14:41:52.946555 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.946451    2553 scope.go:117] "RemoveContainer" containerID="0a9d3ae83c2faecc9b5dcc26b84cb5d93403e6e62c2e798d1d8b14cf26611af1"
Sep 16 14:41:52.957976 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.957941    2553 scope.go:117] "RemoveContainer" containerID="0a9d3ae83c2faecc9b5dcc26b84cb5d93403e6e62c2e798d1d8b14cf26611af1"
Sep 16 14:41:52.958297 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:41:52.958262    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9d3ae83c2faecc9b5dcc26b84cb5d93403e6e62c2e798d1d8b14cf26611af1\": container with ID starting with 0a9d3ae83c2faecc9b5dcc26b84cb5d93403e6e62c2e798d1d8b14cf26611af1 not found: ID does not exist" containerID="0a9d3ae83c2faecc9b5dcc26b84cb5d93403e6e62c2e798d1d8b14cf26611af1"
Sep 16 14:41:52.958399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.958305    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9d3ae83c2faecc9b5dcc26b84cb5d93403e6e62c2e798d1d8b14cf26611af1"} err="failed to get container status \"0a9d3ae83c2faecc9b5dcc26b84cb5d93403e6e62c2e798d1d8b14cf26611af1\": rpc error: code = NotFound desc = could not find container \"0a9d3ae83c2faecc9b5dcc26b84cb5d93403e6e62c2e798d1d8b14cf26611af1\": container with ID starting with 0a9d3ae83c2faecc9b5dcc26b84cb5d93403e6e62c2e798d1d8b14cf26611af1 not found: ID does not exist"
Sep 16 14:41:52.971567 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.971538    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"]
Sep 16 14:41:52.974049 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:52.974028    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f891c2-64f84d65d4-sh889"]
Sep 16 14:41:53.029328 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:53.029294    2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b10c961-55a0-4284-94e4-cb5fa20a16c5-openshift-service-ca\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:53.029328 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:53.029323    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10c961-55a0-4284-94e4-cb5fa20a16c5-maas-api-tls\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:53.029328 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:53.029333    2553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b10c961-55a0-4284-94e4-cb5fa20a16c5-metrics-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:53.029616 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:53.029341    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glvz5\" (UniqueName: \"kubernetes.io/projected/9b10c961-55a0-4284-94e4-cb5fa20a16c5-kube-api-access-glvz5\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:53.917986 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:53.917949    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b10c961-55a0-4284-94e4-cb5fa20a16c5" path="/var/lib/kubelet/pods/9b10c961-55a0-4284-94e4-cb5fa20a16c5/volumes"
Sep 16 14:41:57.780432 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:57.780386    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94" podUID="d68b8afe-5f69-468c-950e-264b03834708" containerName="main" containerID="cri-o://1c09d736ba8edcea13cc1250e452e1156e346e285c5a6661aec265cad47f4583" gracePeriod=45
Sep 16 14:41:57.980413 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:57.980378    2553 generic.go:356] "Generic (PLEG): container finished" podID="d68b8afe-5f69-468c-950e-264b03834708" containerID="1c09d736ba8edcea13cc1250e452e1156e346e285c5a6661aec265cad47f4583" exitCode=0
Sep 16 14:41:57.980570 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:57.980447    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94" event={"ID":"d68b8afe-5f69-468c-950e-264b03834708","Type":"ContainerDied","Data":"1c09d736ba8edcea13cc1250e452e1156e346e285c5a6661aec265cad47f4583"}
Sep 16 14:41:58.048594 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.048563    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:41:58.182341 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.182307    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-tmp-dir\") pod \"d68b8afe-5f69-468c-950e-264b03834708\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") "
Sep 16 14:41:58.182503 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.182421    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-model-cache\") pod \"d68b8afe-5f69-468c-950e-264b03834708\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") "
Sep 16 14:41:58.182503 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.182461    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsqkv\" (UniqueName: \"kubernetes.io/projected/d68b8afe-5f69-468c-950e-264b03834708-kube-api-access-hsqkv\") pod \"d68b8afe-5f69-468c-950e-264b03834708\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") "
Sep 16 14:41:58.182593 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.182540    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d68b8afe-5f69-468c-950e-264b03834708-tls-certs\") pod \"d68b8afe-5f69-468c-950e-264b03834708\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") "
Sep 16 14:41:58.182593 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.182568    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-home\") pod \"d68b8afe-5f69-468c-950e-264b03834708\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") "
Sep 16 14:41:58.182701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.182600    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-dshm\") pod \"d68b8afe-5f69-468c-950e-264b03834708\" (UID: \"d68b8afe-5f69-468c-950e-264b03834708\") "
Sep 16 14:41:58.182701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.182618    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d68b8afe-5f69-468c-950e-264b03834708" (UID: "d68b8afe-5f69-468c-950e-264b03834708"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:58.182701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.182654    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-model-cache" (OuterVolumeSpecName: "model-cache") pod "d68b8afe-5f69-468c-950e-264b03834708" (UID: "d68b8afe-5f69-468c-950e-264b03834708"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:58.182997 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.182830    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-home" (OuterVolumeSpecName: "home") pod "d68b8afe-5f69-468c-950e-264b03834708" (UID: "d68b8afe-5f69-468c-950e-264b03834708"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:58.182997 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.182987    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-home\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.183124 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.183026    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-tmp-dir\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.183124 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.183039    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-model-cache\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.184860 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.184838    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-dshm" (OuterVolumeSpecName: "dshm") pod "d68b8afe-5f69-468c-950e-264b03834708" (UID: "d68b8afe-5f69-468c-950e-264b03834708"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:58.185149 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.185132    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d68b8afe-5f69-468c-950e-264b03834708-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d68b8afe-5f69-468c-950e-264b03834708" (UID: "d68b8afe-5f69-468c-950e-264b03834708"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:41:58.185227 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.185162    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68b8afe-5f69-468c-950e-264b03834708-kube-api-access-hsqkv" (OuterVolumeSpecName: "kube-api-access-hsqkv") pod "d68b8afe-5f69-468c-950e-264b03834708" (UID: "d68b8afe-5f69-468c-950e-264b03834708"). InnerVolumeSpecName "kube-api-access-hsqkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:41:58.283871 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.283787    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d68b8afe-5f69-468c-950e-264b03834708-tls-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.283871 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.283815    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d68b8afe-5f69-468c-950e-264b03834708-dshm\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.283871 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.283823    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsqkv\" (UniqueName: \"kubernetes.io/projected/d68b8afe-5f69-468c-950e-264b03834708-kube-api-access-hsqkv\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.988202 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.988165    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94" event={"ID":"d68b8afe-5f69-468c-950e-264b03834708","Type":"ContainerDied","Data":"fafa674fac603a78d123d8ff9810394b34f9f7d0dcc15c4ddf0e7a75b3b9ec3e"}
Sep 16 14:41:58.988634 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.988214    2553 scope.go:117] "RemoveContainer" containerID="1c09d736ba8edcea13cc1250e452e1156e346e285c5a6661aec265cad47f4583"
Sep 16 14:41:58.988634 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:58.988175    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"
Sep 16 14:41:59.012687 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:59.012657    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"]
Sep 16 14:41:59.018311 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:59.018289    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/simulator-w4-5d484d-kserve-6879bf5f4-f2w94"]
Sep 16 14:41:59.920975 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:41:59.920945    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d68b8afe-5f69-468c-950e-264b03834708" path="/var/lib/kubelet/pods/d68b8afe-5f69-468c-950e-264b03834708/volumes"
Sep 16 14:42:00.233520 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:00.233490    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"]
Sep 16 14:42:00.234016 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:00.233702    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt" podUID="8ce02c4a-beac-4039-a0f5-c72095b2bca4" containerName="istio-proxy" containerID="cri-o://6bc2c77bc06dbed686ab2e514a81168168c9cf1f196c442c74d742bf6a5ea27c" gracePeriod=30
Sep 16 14:42:05.490270 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.490235    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:42:05.654365 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.654270    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-credential-socket\") pod \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") "
Sep 16 14:42:05.654365 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.654344    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-token\") pod \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") "
Sep 16 14:42:05.654593 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.654375    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istiod-ca-cert\") pod \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") "
Sep 16 14:42:05.654593 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.654448    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-workload-certs\") pod \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") "
Sep 16 14:42:05.654593 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.654475    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-envoy\") pod \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") "
Sep 16 14:42:05.654593 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.654514    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-workload-socket\") pod \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") "
Sep 16 14:42:05.654593 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.654564    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8ce02c4a-beac-4039-a0f5-c72095b2bca4" (UID: "8ce02c4a-beac-4039-a0f5-c72095b2bca4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:42:05.654593 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.654587    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-podinfo\") pod \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") "
Sep 16 14:42:05.654886 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.654620    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf67q\" (UniqueName: \"kubernetes.io/projected/8ce02c4a-beac-4039-a0f5-c72095b2bca4-kube-api-access-sf67q\") pod \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") "
Sep 16 14:42:05.654886 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.654675    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-data\") pod \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\" (UID: \"8ce02c4a-beac-4039-a0f5-c72095b2bca4\") "
Sep 16 14:42:05.655092 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.655065    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8ce02c4a-beac-4039-a0f5-c72095b2bca4" (UID: "8ce02c4a-beac-4039-a0f5-c72095b2bca4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:42:05.655169 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.655143    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-credential-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:05.655226 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.655192    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8ce02c4a-beac-4039-a0f5-c72095b2bca4" (UID: "8ce02c4a-beac-4039-a0f5-c72095b2bca4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:42:05.655311 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.655290    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8ce02c4a-beac-4039-a0f5-c72095b2bca4" (UID: "8ce02c4a-beac-4039-a0f5-c72095b2bca4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:42:05.655373 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.655307    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-data" (OuterVolumeSpecName: "istio-data") pod "8ce02c4a-beac-4039-a0f5-c72095b2bca4" (UID: "8ce02c4a-beac-4039-a0f5-c72095b2bca4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:42:05.656817 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.656788    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-token" (OuterVolumeSpecName: "istio-token") pod "8ce02c4a-beac-4039-a0f5-c72095b2bca4" (UID: "8ce02c4a-beac-4039-a0f5-c72095b2bca4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:42:05.657144 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.657125    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8ce02c4a-beac-4039-a0f5-c72095b2bca4" (UID: "8ce02c4a-beac-4039-a0f5-c72095b2bca4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:42:05.657351 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.657330    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8ce02c4a-beac-4039-a0f5-c72095b2bca4" (UID: "8ce02c4a-beac-4039-a0f5-c72095b2bca4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:42:05.657511 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.657496    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce02c4a-beac-4039-a0f5-c72095b2bca4-kube-api-access-sf67q" (OuterVolumeSpecName: "kube-api-access-sf67q") pod "8ce02c4a-beac-4039-a0f5-c72095b2bca4" (UID: "8ce02c4a-beac-4039-a0f5-c72095b2bca4"). InnerVolumeSpecName "kube-api-access-sf67q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:42:05.756157 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.756125    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-podinfo\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:05.756157 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.756151    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf67q\" (UniqueName: \"kubernetes.io/projected/8ce02c4a-beac-4039-a0f5-c72095b2bca4-kube-api-access-sf67q\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:05.756339 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.756166    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-data\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:05.756339 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.756175    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-token\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:05.756339 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.756182    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istiod-ca-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:05.756339 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.756190    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-workload-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:05.756339 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.756197    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-istio-envoy\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:05.756339 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:05.756204    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce02c4a-beac-4039-a0f5-c72095b2bca4-workload-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:06.030766 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:06.030732    2553 generic.go:356] "Generic (PLEG): container finished" podID="8ce02c4a-beac-4039-a0f5-c72095b2bca4" containerID="6bc2c77bc06dbed686ab2e514a81168168c9cf1f196c442c74d742bf6a5ea27c" exitCode=0
Sep 16 14:42:06.030930 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:06.030785    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"
Sep 16 14:42:06.030930 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:06.030811    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt" event={"ID":"8ce02c4a-beac-4039-a0f5-c72095b2bca4","Type":"ContainerDied","Data":"6bc2c77bc06dbed686ab2e514a81168168c9cf1f196c442c74d742bf6a5ea27c"}
Sep 16 14:42:06.030930 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:06.030865    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt" event={"ID":"8ce02c4a-beac-4039-a0f5-c72095b2bca4","Type":"ContainerDied","Data":"8659716817c426ed048e1684a58c8e712bb0e0e2f3bf970b42a176a3c4dc3165"}
Sep 16 14:42:06.030930 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:06.030911    2553 scope.go:117] "RemoveContainer" containerID="6bc2c77bc06dbed686ab2e514a81168168c9cf1f196c442c74d742bf6a5ea27c"
Sep 16 14:42:06.040139 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:06.040122    2553 scope.go:117] "RemoveContainer" containerID="abd3abdd452b4caeb5e5a6159eba2459abb5abf5e14bf920618275301dc92f65"
Sep 16 14:42:06.049305 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:06.049283    2553 scope.go:117] "RemoveContainer" containerID="6bc2c77bc06dbed686ab2e514a81168168c9cf1f196c442c74d742bf6a5ea27c"
Sep 16 14:42:06.049606 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:42:06.049585    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc2c77bc06dbed686ab2e514a81168168c9cf1f196c442c74d742bf6a5ea27c\": container with ID starting with 6bc2c77bc06dbed686ab2e514a81168168c9cf1f196c442c74d742bf6a5ea27c not found: ID does not exist" containerID="6bc2c77bc06dbed686ab2e514a81168168c9cf1f196c442c74d742bf6a5ea27c"
Sep 16 14:42:06.049701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:06.049619    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc2c77bc06dbed686ab2e514a81168168c9cf1f196c442c74d742bf6a5ea27c"} err="failed to get container status \"6bc2c77bc06dbed686ab2e514a81168168c9cf1f196c442c74d742bf6a5ea27c\": rpc error: code = NotFound desc = could not find container \"6bc2c77bc06dbed686ab2e514a81168168c9cf1f196c442c74d742bf6a5ea27c\": container with ID starting with 6bc2c77bc06dbed686ab2e514a81168168c9cf1f196c442c74d742bf6a5ea27c not found: ID does not exist"
Sep 16 14:42:06.049701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:06.049641    2553 scope.go:117] "RemoveContainer" containerID="abd3abdd452b4caeb5e5a6159eba2459abb5abf5e14bf920618275301dc92f65"
Sep 16 14:42:06.050038 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:42:06.050006    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd3abdd452b4caeb5e5a6159eba2459abb5abf5e14bf920618275301dc92f65\": container with ID starting with abd3abdd452b4caeb5e5a6159eba2459abb5abf5e14bf920618275301dc92f65 not found: ID does not exist" containerID="abd3abdd452b4caeb5e5a6159eba2459abb5abf5e14bf920618275301dc92f65"
Sep 16 14:42:06.050107 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:06.050041    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd3abdd452b4caeb5e5a6159eba2459abb5abf5e14bf920618275301dc92f65"} err="failed to get container status \"abd3abdd452b4caeb5e5a6159eba2459abb5abf5e14bf920618275301dc92f65\": rpc error: code = NotFound desc = could not find container \"abd3abdd452b4caeb5e5a6159eba2459abb5abf5e14bf920618275301dc92f65\": container with ID starting with abd3abdd452b4caeb5e5a6159eba2459abb5abf5e14bf920618275301dc92f65 not found: ID does not exist"
Sep 16 14:42:06.051456 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:06.051435    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"]
Sep 16 14:42:06.057227 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:06.057201    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f891c2-openshift-default-54c4dc76cb-fwmzt"]
Sep 16 14:42:07.917952 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:07.917918    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ce02c4a-beac-4039-a0f5-c72095b2bca4" path="/var/lib/kubelet/pods/8ce02c4a-beac-4039-a0f5-c72095b2bca4/volumes"
Sep 16 14:42:17.972152 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:17.972119    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"]
Sep 16 14:42:17.972599 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:17.972372    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm" podUID="e0d5f096-6536-44e4-b6f5-13719958d76e" containerName="istio-proxy" containerID="cri-o://a416539105a62c013c4f17cd158e07e9ed457413f0d63733719a9773055b445c" gracePeriod=30
Sep 16 14:42:23.122157 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.122125    2553 generic.go:356] "Generic (PLEG): container finished" podID="e0d5f096-6536-44e4-b6f5-13719958d76e" containerID="a416539105a62c013c4f17cd158e07e9ed457413f0d63733719a9773055b445c" exitCode=0
Sep 16 14:42:23.122504 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.122196    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm" event={"ID":"e0d5f096-6536-44e4-b6f5-13719958d76e","Type":"ContainerDied","Data":"a416539105a62c013c4f17cd158e07e9ed457413f0d63733719a9773055b445c"}
Sep 16 14:42:23.122504 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.122293    2553 scope.go:117] "RemoveContainer" containerID="1eaf5df6946d8a8a4af61c815cce50665f4f89f14e6497cd1d6849785efd50d3"
Sep 16 14:42:23.237982 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.237931    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:42:23.328546 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.328519    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-envoy\") pod \"e0d5f096-6536-44e4-b6f5-13719958d76e\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") "
Sep 16 14:42:23.328704 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.328572    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-workload-socket\") pod \"e0d5f096-6536-44e4-b6f5-13719958d76e\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") "
Sep 16 14:42:23.328704 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.328596    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-token\") pod \"e0d5f096-6536-44e4-b6f5-13719958d76e\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") "
Sep 16 14:42:23.328704 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.328638    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d5f096-6536-44e4-b6f5-13719958d76e-istiod-ca-cert\") pod \"e0d5f096-6536-44e4-b6f5-13719958d76e\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") "
Sep 16 14:42:23.328704 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.328674    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-data\") pod \"e0d5f096-6536-44e4-b6f5-13719958d76e\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") "
Sep 16 14:42:23.328926 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.328820    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e0d5f096-6536-44e4-b6f5-13719958d76e" (UID: "e0d5f096-6536-44e4-b6f5-13719958d76e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:42:23.328926 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.328886    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-podinfo\") pod \"e0d5f096-6536-44e4-b6f5-13719958d76e\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") "
Sep 16 14:42:23.329049 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.328949    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-credential-socket\") pod \"e0d5f096-6536-44e4-b6f5-13719958d76e\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") "
Sep 16 14:42:23.329049 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.328991    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4lh6\" (UniqueName: \"kubernetes.io/projected/e0d5f096-6536-44e4-b6f5-13719958d76e-kube-api-access-t4lh6\") pod \"e0d5f096-6536-44e4-b6f5-13719958d76e\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") "
Sep 16 14:42:23.329049 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.328994    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-data" (OuterVolumeSpecName: "istio-data") pod "e0d5f096-6536-44e4-b6f5-13719958d76e" (UID: "e0d5f096-6536-44e4-b6f5-13719958d76e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:42:23.329203 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.329057    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-workload-certs\") pod \"e0d5f096-6536-44e4-b6f5-13719958d76e\" (UID: \"e0d5f096-6536-44e4-b6f5-13719958d76e\") "
Sep 16 14:42:23.329203 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.329101    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d5f096-6536-44e4-b6f5-13719958d76e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e0d5f096-6536-44e4-b6f5-13719958d76e" (UID: "e0d5f096-6536-44e4-b6f5-13719958d76e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:42:23.329302 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.329201    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e0d5f096-6536-44e4-b6f5-13719958d76e" (UID: "e0d5f096-6536-44e4-b6f5-13719958d76e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:42:23.329421 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.329400    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e0d5f096-6536-44e4-b6f5-13719958d76e" (UID: "e0d5f096-6536-44e4-b6f5-13719958d76e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:42:23.329548 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.329528    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-workload-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:23.329595 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.329553    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d5f096-6536-44e4-b6f5-13719958d76e-istiod-ca-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:23.329595 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.329568    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-data\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:23.329595 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.329580    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-credential-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:23.329700 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.329593    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-workload-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:23.330948 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.330918    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e0d5f096-6536-44e4-b6f5-13719958d76e" (UID: "e0d5f096-6536-44e4-b6f5-13719958d76e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:42:23.331102 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.331015    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-token" (OuterVolumeSpecName: "istio-token") pod "e0d5f096-6536-44e4-b6f5-13719958d76e" (UID: "e0d5f096-6536-44e4-b6f5-13719958d76e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:42:23.331180 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.331161    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e0d5f096-6536-44e4-b6f5-13719958d76e" (UID: "e0d5f096-6536-44e4-b6f5-13719958d76e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:42:23.331327 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.331308    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d5f096-6536-44e4-b6f5-13719958d76e-kube-api-access-t4lh6" (OuterVolumeSpecName: "kube-api-access-t4lh6") pod "e0d5f096-6536-44e4-b6f5-13719958d76e" (UID: "e0d5f096-6536-44e4-b6f5-13719958d76e"). InnerVolumeSpecName "kube-api-access-t4lh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:42:23.430957 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.430923    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-envoy\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:23.430957 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.430952    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-token\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:23.430957 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.430960    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d5f096-6536-44e4-b6f5-13719958d76e-istio-podinfo\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:23.431180 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.430969    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4lh6\" (UniqueName: \"kubernetes.io/projected/e0d5f096-6536-44e4-b6f5-13719958d76e-kube-api-access-t4lh6\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:23.576601 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:23.576521    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"]
Sep 16 14:42:24.129326 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:24.129297    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"
Sep 16 14:42:24.129700 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:24.129335    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm" event={"ID":"e0d5f096-6536-44e4-b6f5-13719958d76e","Type":"ContainerDied","Data":"2a625970925c332c7e5ae27a299c6e222f357c3c1f101ba5d4632e6a3ba98971"}
Sep 16 14:42:24.129700 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:24.129392    2553 scope.go:117] "RemoveContainer" containerID="a416539105a62c013c4f17cd158e07e9ed457413f0d63733719a9773055b445c"
Sep 16 14:42:24.148054 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:24.148031    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"]
Sep 16 14:42:24.153225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:24.153200    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4c2787-openshift-default-5c5ffc94fb-h72cm"]
Sep 16 14:42:25.917333 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:25.917304    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0d5f096-6536-44e4-b6f5-13719958d76e" path="/var/lib/kubelet/pods/e0d5f096-6536-44e4-b6f5-13719958d76e/volumes"
Sep 16 14:42:28.815780 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:28.815746    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"]
Sep 16 14:42:28.816202 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:28.816068    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg" podUID="087f1e0c-e646-4ec5-a5fb-2d6df8b56572" containerName="maas-api" containerID="cri-o://850b36b338fe405379e3adaa203e72cfb17de105b6bc8dd9b60ec95daa0377b3" gracePeriod=30
Sep 16 14:42:29.069364 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.069298    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:42:29.159365 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.159330    2553 generic.go:356] "Generic (PLEG): container finished" podID="087f1e0c-e646-4ec5-a5fb-2d6df8b56572" containerID="850b36b338fe405379e3adaa203e72cfb17de105b6bc8dd9b60ec95daa0377b3" exitCode=0
Sep 16 14:42:29.159522 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.159388    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"
Sep 16 14:42:29.159522 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.159416    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg" event={"ID":"087f1e0c-e646-4ec5-a5fb-2d6df8b56572","Type":"ContainerDied","Data":"850b36b338fe405379e3adaa203e72cfb17de105b6bc8dd9b60ec95daa0377b3"}
Sep 16 14:42:29.159522 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.159476    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg" event={"ID":"087f1e0c-e646-4ec5-a5fb-2d6df8b56572","Type":"ContainerDied","Data":"94e5a4153cdc19c3f58377c6fa72fc5123d1cd83a45af0aea6a316b543363dbe"}
Sep 16 14:42:29.159522 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.159499    2553 scope.go:117] "RemoveContainer" containerID="850b36b338fe405379e3adaa203e72cfb17de105b6bc8dd9b60ec95daa0377b3"
Sep 16 14:42:29.168578 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.168555    2553 scope.go:117] "RemoveContainer" containerID="850b36b338fe405379e3adaa203e72cfb17de105b6bc8dd9b60ec95daa0377b3"
Sep 16 14:42:29.168876 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:42:29.168858    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850b36b338fe405379e3adaa203e72cfb17de105b6bc8dd9b60ec95daa0377b3\": container with ID starting with 850b36b338fe405379e3adaa203e72cfb17de105b6bc8dd9b60ec95daa0377b3 not found: ID does not exist" containerID="850b36b338fe405379e3adaa203e72cfb17de105b6bc8dd9b60ec95daa0377b3"
Sep 16 14:42:29.168984 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.168883    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850b36b338fe405379e3adaa203e72cfb17de105b6bc8dd9b60ec95daa0377b3"} err="failed to get container status \"850b36b338fe405379e3adaa203e72cfb17de105b6bc8dd9b60ec95daa0377b3\": rpc error: code = NotFound desc = could not find container \"850b36b338fe405379e3adaa203e72cfb17de105b6bc8dd9b60ec95daa0377b3\": container with ID starting with 850b36b338fe405379e3adaa203e72cfb17de105b6bc8dd9b60ec95daa0377b3 not found: ID does not exist"
Sep 16 14:42:29.191290 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.191261    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-maas-api-tls\") pod \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") "
Sep 16 14:42:29.191423 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.191302    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-metrics-certs\") pod \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") "
Sep 16 14:42:29.191423 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.191340    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-openshift-service-ca\") pod \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") "
Sep 16 14:42:29.191542 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.191434    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brm48\" (UniqueName: \"kubernetes.io/projected/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-kube-api-access-brm48\") pod \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\" (UID: \"087f1e0c-e646-4ec5-a5fb-2d6df8b56572\") "
Sep 16 14:42:29.191703 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.191677    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "087f1e0c-e646-4ec5-a5fb-2d6df8b56572" (UID: "087f1e0c-e646-4ec5-a5fb-2d6df8b56572"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:42:29.193488 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.193463    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "087f1e0c-e646-4ec5-a5fb-2d6df8b56572" (UID: "087f1e0c-e646-4ec5-a5fb-2d6df8b56572"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:42:29.194003 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.193974    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "087f1e0c-e646-4ec5-a5fb-2d6df8b56572" (UID: "087f1e0c-e646-4ec5-a5fb-2d6df8b56572"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:42:29.194003 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.193986    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-kube-api-access-brm48" (OuterVolumeSpecName: "kube-api-access-brm48") pod "087f1e0c-e646-4ec5-a5fb-2d6df8b56572" (UID: "087f1e0c-e646-4ec5-a5fb-2d6df8b56572"). InnerVolumeSpecName "kube-api-access-brm48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:42:29.292729 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.292698    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-maas-api-tls\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:29.292729 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.292727    2553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-metrics-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:29.292941 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.292736    2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-openshift-service-ca\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:29.292941 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.292747    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brm48\" (UniqueName: \"kubernetes.io/projected/087f1e0c-e646-4ec5-a5fb-2d6df8b56572-kube-api-access-brm48\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:29.483868 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.483838    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"]
Sep 16 14:42:29.490312 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.490285    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2539c0-5f6d44dd5d-v5wbg"]
Sep 16 14:42:29.917507 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:29.917474    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="087f1e0c-e646-4ec5-a5fb-2d6df8b56572" path="/var/lib/kubelet/pods/087f1e0c-e646-4ec5-a5fb-2d6df8b56572/volumes"
Sep 16 14:42:38.596488 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.596401    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg" podUID="cc8cd338-9b14-42ac-ac98-c315286ec8bf" containerName="main" containerID="cri-o://694f2300a9563085fe7e643d8885a2233f65d36838507a21eca349c06d4ccc9a" gracePeriod=45
Sep 16 14:42:38.852484 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.852421    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:42:38.981997 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.981966    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvxzc\" (UniqueName: \"kubernetes.io/projected/cc8cd338-9b14-42ac-ac98-c315286ec8bf-kube-api-access-zvxzc\") pod \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") "
Sep 16 14:42:38.982155 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.982022    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-model-cache\") pod \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") "
Sep 16 14:42:38.982155 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.982069    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-tmp-dir\") pod \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") "
Sep 16 14:42:38.982155 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.982137    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc8cd338-9b14-42ac-ac98-c315286ec8bf-tls-certs\") pod \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") "
Sep 16 14:42:38.982307 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.982182    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-dshm\") pod \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") "
Sep 16 14:42:38.982307 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.982206    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-home\") pod \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\" (UID: \"cc8cd338-9b14-42ac-ac98-c315286ec8bf\") "
Sep 16 14:42:38.982413 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.982341    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc8cd338-9b14-42ac-ac98-c315286ec8bf" (UID: "cc8cd338-9b14-42ac-ac98-c315286ec8bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:42:38.982413 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.982387    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc8cd338-9b14-42ac-ac98-c315286ec8bf" (UID: "cc8cd338-9b14-42ac-ac98-c315286ec8bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:42:38.982531 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.982467    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-home" (OuterVolumeSpecName: "home") pod "cc8cd338-9b14-42ac-ac98-c315286ec8bf" (UID: "cc8cd338-9b14-42ac-ac98-c315286ec8bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:42:38.982703 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.982684    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-tmp-dir\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:38.982748 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.982708    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-home\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:38.982748 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.982723    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-model-cache\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:38.984390 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.984366    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc8cd338-9b14-42ac-ac98-c315286ec8bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc8cd338-9b14-42ac-ac98-c315286ec8bf" (UID: "cc8cd338-9b14-42ac-ac98-c315286ec8bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:42:38.984531 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.984399    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-dshm" (OuterVolumeSpecName: "dshm") pod "cc8cd338-9b14-42ac-ac98-c315286ec8bf" (UID: "cc8cd338-9b14-42ac-ac98-c315286ec8bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:42:38.984531 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:38.984476    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8cd338-9b14-42ac-ac98-c315286ec8bf-kube-api-access-zvxzc" (OuterVolumeSpecName: "kube-api-access-zvxzc") pod "cc8cd338-9b14-42ac-ac98-c315286ec8bf" (UID: "cc8cd338-9b14-42ac-ac98-c315286ec8bf"). InnerVolumeSpecName "kube-api-access-zvxzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:42:39.084260 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:39.084226    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc8cd338-9b14-42ac-ac98-c315286ec8bf-tls-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:39.084260 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:39.084255    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd338-9b14-42ac-ac98-c315286ec8bf-dshm\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:39.084260 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:39.084269    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvxzc\" (UniqueName: \"kubernetes.io/projected/cc8cd338-9b14-42ac-ac98-c315286ec8bf-kube-api-access-zvxzc\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:42:39.214363 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:39.214276    2553 generic.go:356] "Generic (PLEG): container finished" podID="cc8cd338-9b14-42ac-ac98-c315286ec8bf" containerID="694f2300a9563085fe7e643d8885a2233f65d36838507a21eca349c06d4ccc9a" exitCode=0
Sep 16 14:42:39.214363 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:39.214320    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"
Sep 16 14:42:39.214363 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:39.214350    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg" event={"ID":"cc8cd338-9b14-42ac-ac98-c315286ec8bf","Type":"ContainerDied","Data":"694f2300a9563085fe7e643d8885a2233f65d36838507a21eca349c06d4ccc9a"}
Sep 16 14:42:39.214590 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:39.214401    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg" event={"ID":"cc8cd338-9b14-42ac-ac98-c315286ec8bf","Type":"ContainerDied","Data":"f601b94b41a560b18c40b831f19f541d3b831a61e67c882a74e9cd95dc188fc5"}
Sep 16 14:42:39.214590 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:39.214423    2553 scope.go:117] "RemoveContainer" containerID="694f2300a9563085fe7e643d8885a2233f65d36838507a21eca349c06d4ccc9a"
Sep 16 14:42:39.224058 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:39.224040    2553 scope.go:117] "RemoveContainer" containerID="694f2300a9563085fe7e643d8885a2233f65d36838507a21eca349c06d4ccc9a"
Sep 16 14:42:39.224318 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:42:39.224300    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694f2300a9563085fe7e643d8885a2233f65d36838507a21eca349c06d4ccc9a\": container with ID starting with 694f2300a9563085fe7e643d8885a2233f65d36838507a21eca349c06d4ccc9a not found: ID does not exist" containerID="694f2300a9563085fe7e643d8885a2233f65d36838507a21eca349c06d4ccc9a"
Sep 16 14:42:39.224365 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:39.224327    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694f2300a9563085fe7e643d8885a2233f65d36838507a21eca349c06d4ccc9a"} err="failed to get container status \"694f2300a9563085fe7e643d8885a2233f65d36838507a21eca349c06d4ccc9a\": rpc error: code = NotFound desc = could not find container \"694f2300a9563085fe7e643d8885a2233f65d36838507a21eca349c06d4ccc9a\": container with ID starting with 694f2300a9563085fe7e643d8885a2233f65d36838507a21eca349c06d4ccc9a not found: ID does not exist"
Sep 16 14:42:39.238343 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:39.238320    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"]
Sep 16 14:42:39.242393 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:39.242373    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/auth-test-model-055ce3-kserve-76fc74d67-g2fkg"]
Sep 16 14:42:39.918050 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:42:39.918020    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc8cd338-9b14-42ac-ac98-c315286ec8bf" path="/var/lib/kubelet/pods/cc8cd338-9b14-42ac-ac98-c315286ec8bf/volumes"
Sep 16 14:43:04.322815 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.322782    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"]
Sep 16 14:43:04.323235 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323224    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce02c4a-beac-4039-a0f5-c72095b2bca4" containerName="istio-proxy"
Sep 16 14:43:04.323287 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323237    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce02c4a-beac-4039-a0f5-c72095b2bca4" containerName="istio-proxy"
Sep 16 14:43:04.323287 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323247    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce02c4a-beac-4039-a0f5-c72095b2bca4" containerName="istio-proxy"
Sep 16 14:43:04.323287 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323253    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce02c4a-beac-4039-a0f5-c72095b2bca4" containerName="istio-proxy"
Sep 16 14:43:04.323287 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323260    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d68b8afe-5f69-468c-950e-264b03834708" containerName="main"
Sep 16 14:43:04.323287 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323266    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d68b8afe-5f69-468c-950e-264b03834708" containerName="main"
Sep 16 14:43:04.323287 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323274    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d5f096-6536-44e4-b6f5-13719958d76e" containerName="istio-proxy"
Sep 16 14:43:04.323287 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323280    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d5f096-6536-44e4-b6f5-13719958d76e" containerName="istio-proxy"
Sep 16 14:43:04.323287 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323287    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc8cd338-9b14-42ac-ac98-c315286ec8bf" containerName="main"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323293    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc8cd338-9b14-42ac-ac98-c315286ec8bf" containerName="main"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323300    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922143eb-a7d5-4910-b16b-a490824fdfed" containerName="istio-proxy"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323304    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="922143eb-a7d5-4910-b16b-a490824fdfed" containerName="istio-proxy"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323312    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="087f1e0c-e646-4ec5-a5fb-2d6df8b56572" containerName="maas-api"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323318    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="087f1e0c-e646-4ec5-a5fb-2d6df8b56572" containerName="maas-api"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323331    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d5f096-6536-44e4-b6f5-13719958d76e" containerName="istio-proxy"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323337    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d5f096-6536-44e4-b6f5-13719958d76e" containerName="istio-proxy"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323344    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e09621d0-91c1-44da-9f84-9a4761ebc9a6" containerName="maas-api"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323349    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e09621d0-91c1-44da-9f84-9a4761ebc9a6" containerName="maas-api"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323361    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b10c961-55a0-4284-94e4-cb5fa20a16c5" containerName="maas-api"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323367    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b10c961-55a0-4284-94e4-cb5fa20a16c5" containerName="maas-api"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323421    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce02c4a-beac-4039-a0f5-c72095b2bca4" containerName="istio-proxy"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323428    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e09621d0-91c1-44da-9f84-9a4761ebc9a6" containerName="maas-api"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323435    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0d5f096-6536-44e4-b6f5-13719958d76e" containerName="istio-proxy"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323441    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc8cd338-9b14-42ac-ac98-c315286ec8bf" containerName="main"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323448    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="922143eb-a7d5-4910-b16b-a490824fdfed" containerName="istio-proxy"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323453    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="087f1e0c-e646-4ec5-a5fb-2d6df8b56572" containerName="maas-api"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323459    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d68b8afe-5f69-468c-950e-264b03834708" containerName="main"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323464    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b10c961-55a0-4284-94e4-cb5fa20a16c5" containerName="maas-api"
Sep 16 14:43:04.323635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.323471    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0d5f096-6536-44e4-b6f5-13719958d76e" containerName="istio-proxy"
Sep 16 14:43:04.326408 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.326393    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.329222 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.329184    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-055ce3\"/\"openshift-service-ca.crt\""
Sep 16 14:43:04.329428 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.329245    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-055ce3\"/\"kube-root-ca.crt\""
Sep 16 14:43:04.330252 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.330232    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-055ce3\"/\"default-dockercfg-v45lm\""
Sep 16 14:43:04.330370 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.330285    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-055ce3\"/\"sub-test-model-055ce3-kserve-self-signed-certs\""
Sep 16 14:43:04.339370 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.339350    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"]
Sep 16 14:43:04.423503 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.423472    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcxf5\" (UniqueName: \"kubernetes.io/projected/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-kube-api-access-jcxf5\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.423680 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.423513    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-dshm\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.423680 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.423543    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-home\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.423680 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.423599    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-tls-certs\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.423806 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.423704    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-tmp-dir\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.423806 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.423755    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-model-cache\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.524770 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.524732    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-model-cache\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.524953 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.524777    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxf5\" (UniqueName: \"kubernetes.io/projected/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-kube-api-access-jcxf5\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.524953 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.524923    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-dshm\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.525056 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.524974    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-home\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.525056 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.525004    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-tls-certs\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.525170 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.525104    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-tmp-dir\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.525225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.525190    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-model-cache\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.525371 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.525352    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-home\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.525443 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.525414    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-tmp-dir\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.527000 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.526971    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-dshm\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.527312 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.527295    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-tls-certs\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.535864 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.535843    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxf5\" (UniqueName: \"kubernetes.io/projected/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-kube-api-access-jcxf5\") pod \"sub-test-model-055ce3-kserve-5688898f97-7wxss\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.637337 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.637241    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:04.918612 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:04.918585    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"]
Sep 16 14:43:04.920554 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:43:04.920525    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8d04198_c5f7_4680_9e28_c71a6cd7f3e4.slice/crio-8c9dd9101f1d8a7c0eb0771275556d401bf0b0e164d90d6f7672673324fee555 WatchSource:0}: Error finding container 8c9dd9101f1d8a7c0eb0771275556d401bf0b0e164d90d6f7672673324fee555: Status 404 returned error can't find the container with id 8c9dd9101f1d8a7c0eb0771275556d401bf0b0e164d90d6f7672673324fee555
Sep 16 14:43:05.352545 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:05.352505    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss" event={"ID":"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4","Type":"ContainerStarted","Data":"ea8056c4e268cb757824c4ee70622b07ebd014c3847a8a4fa9962e83bf4062b9"}
Sep 16 14:43:05.352545 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:05.352559    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss" event={"ID":"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4","Type":"ContainerStarted","Data":"8c9dd9101f1d8a7c0eb0771275556d401bf0b0e164d90d6f7672673324fee555"}
Sep 16 14:43:05.380657 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:05.380603    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss" podStartSLOduration=1.38058606 podStartE2EDuration="1.38058606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:43:05.378595978 +0000 UTC m=+1735.925337910" watchObservedRunningTime="2026-09-16 14:43:05.38058606 +0000 UTC m=+1735.927327990"
Sep 16 14:43:14.638030 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:14.637994    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:14.638452 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:14.638049    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:14.650233 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:14.650211    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:15.418642 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:15.418611    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:43:51.927922 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:43:51.925871    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"]
Sep 16 14:44:06.949564 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:06.949470    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss" podUID="d8d04198-c5f7-4680-9e28-c71a6cd7f3e4" containerName="main" containerID="cri-o://ea8056c4e268cb757824c4ee70622b07ebd014c3847a8a4fa9962e83bf4062b9" gracePeriod=45
Sep 16 14:44:07.199252 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.199225    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:44:07.296804 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.296776    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcxf5\" (UniqueName: \"kubernetes.io/projected/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-kube-api-access-jcxf5\") pod \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") "
Sep 16 14:44:07.296989 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.296823    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-tmp-dir\") pod \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") "
Sep 16 14:44:07.296989 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.296846    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-tls-certs\") pod \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") "
Sep 16 14:44:07.296989 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.296878    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-home\") pod \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") "
Sep 16 14:44:07.296989 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.296941    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-model-cache\") pod \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") "
Sep 16 14:44:07.296989 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.296982    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-dshm\") pod \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\" (UID: \"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4\") "
Sep 16 14:44:07.297213 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.297121    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8d04198-c5f7-4680-9e28-c71a6cd7f3e4" (UID: "d8d04198-c5f7-4680-9e28-c71a6cd7f3e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:44:07.297213 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.297163    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-home" (OuterVolumeSpecName: "home") pod "d8d04198-c5f7-4680-9e28-c71a6cd7f3e4" (UID: "d8d04198-c5f7-4680-9e28-c71a6cd7f3e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:44:07.297324 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.297245    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8d04198-c5f7-4680-9e28-c71a6cd7f3e4" (UID: "d8d04198-c5f7-4680-9e28-c71a6cd7f3e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:44:07.297410 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.297394    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-tmp-dir\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:44:07.297470 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.297410    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-home\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:44:07.297470 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.297419    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-model-cache\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:44:07.299127 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.299103    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-kube-api-access-jcxf5" (OuterVolumeSpecName: "kube-api-access-jcxf5") pod "d8d04198-c5f7-4680-9e28-c71a6cd7f3e4" (UID: "d8d04198-c5f7-4680-9e28-c71a6cd7f3e4"). InnerVolumeSpecName "kube-api-access-jcxf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:44:07.299127 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.299118    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8d04198-c5f7-4680-9e28-c71a6cd7f3e4" (UID: "d8d04198-c5f7-4680-9e28-c71a6cd7f3e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:44:07.299283 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.299144    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-dshm" (OuterVolumeSpecName: "dshm") pod "d8d04198-c5f7-4680-9e28-c71a6cd7f3e4" (UID: "d8d04198-c5f7-4680-9e28-c71a6cd7f3e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:44:07.398356 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.398319    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcxf5\" (UniqueName: \"kubernetes.io/projected/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-kube-api-access-jcxf5\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:44:07.398356 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.398350    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-tls-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:44:07.398356 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.398361    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4-dshm\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:44:07.679715 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.679612    2553 generic.go:356] "Generic (PLEG): container finished" podID="d8d04198-c5f7-4680-9e28-c71a6cd7f3e4" containerID="ea8056c4e268cb757824c4ee70622b07ebd014c3847a8a4fa9962e83bf4062b9" exitCode=0
Sep 16 14:44:07.679715 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.679660    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"
Sep 16 14:44:07.679967 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.679710    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss" event={"ID":"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4","Type":"ContainerDied","Data":"ea8056c4e268cb757824c4ee70622b07ebd014c3847a8a4fa9962e83bf4062b9"}
Sep 16 14:44:07.679967 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.679759    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss" event={"ID":"d8d04198-c5f7-4680-9e28-c71a6cd7f3e4","Type":"ContainerDied","Data":"8c9dd9101f1d8a7c0eb0771275556d401bf0b0e164d90d6f7672673324fee555"}
Sep 16 14:44:07.679967 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.679779    2553 scope.go:117] "RemoveContainer" containerID="ea8056c4e268cb757824c4ee70622b07ebd014c3847a8a4fa9962e83bf4062b9"
Sep 16 14:44:07.689484 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.689467    2553 scope.go:117] "RemoveContainer" containerID="ea8056c4e268cb757824c4ee70622b07ebd014c3847a8a4fa9962e83bf4062b9"
Sep 16 14:44:07.689727 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:44:07.689708    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8056c4e268cb757824c4ee70622b07ebd014c3847a8a4fa9962e83bf4062b9\": container with ID starting with ea8056c4e268cb757824c4ee70622b07ebd014c3847a8a4fa9962e83bf4062b9 not found: ID does not exist" containerID="ea8056c4e268cb757824c4ee70622b07ebd014c3847a8a4fa9962e83bf4062b9"
Sep 16 14:44:07.689782 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.689734    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8056c4e268cb757824c4ee70622b07ebd014c3847a8a4fa9962e83bf4062b9"} err="failed to get container status \"ea8056c4e268cb757824c4ee70622b07ebd014c3847a8a4fa9962e83bf4062b9\": rpc error: code = NotFound desc = could not find container \"ea8056c4e268cb757824c4ee70622b07ebd014c3847a8a4fa9962e83bf4062b9\": container with ID starting with ea8056c4e268cb757824c4ee70622b07ebd014c3847a8a4fa9962e83bf4062b9 not found: ID does not exist"
Sep 16 14:44:07.704138 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.704106    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"]
Sep 16 14:44:07.710815 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.710790    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/sub-test-model-055ce3-kserve-5688898f97-7wxss"]
Sep 16 14:44:07.918323 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:07.918293    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8d04198-c5f7-4680-9e28-c71a6cd7f3e4" path="/var/lib/kubelet/pods/d8d04198-c5f7-4680-9e28-c71a6cd7f3e4/volumes"
Sep 16 14:44:09.995860 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:09.995833    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:44:10.001162 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:44:10.001142    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:45:00.134634 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:00.134603    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f88e1-29826165-d7jnd"]
Sep 16 14:45:00.135136 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:00.135040    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922143eb-a7d5-4910-b16b-a490824fdfed" containerName="istio-proxy"
Sep 16 14:45:00.135136 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:00.135053    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="922143eb-a7d5-4910-b16b-a490824fdfed" containerName="istio-proxy"
Sep 16 14:45:00.135136 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:00.135065    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8d04198-c5f7-4680-9e28-c71a6cd7f3e4" containerName="main"
Sep 16 14:45:00.135136 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:00.135070    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8d04198-c5f7-4680-9e28-c71a6cd7f3e4" containerName="main"
Sep 16 14:45:00.135136 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:00.135133    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce02c4a-beac-4039-a0f5-c72095b2bca4" containerName="istio-proxy"
Sep 16 14:45:00.135301 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:00.135144    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="922143eb-a7d5-4910-b16b-a490824fdfed" containerName="istio-proxy"
Sep 16 14:45:00.135301 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:00.135152    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8d04198-c5f7-4680-9e28-c71a6cd7f3e4" containerName="main"
Sep 16 14:45:00.138195 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:00.138180    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f88e1-29826165-d7jnd"
Sep 16 14:45:00.141042 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:00.141017    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:45:00.149808 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:00.149758    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f88e1-29826165-d7jnd"]
Sep 16 14:45:00.150423 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:00.150392    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f88e1-29826165-d7jnd"
Sep 16 14:45:00.426786 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:00.426759    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f88e1-29826165-d7jnd"]
Sep 16 14:45:00.429181 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:45:00.429154    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e34253e_0fc4_42fe_9268_aacdcc6a9114.slice/crio-f3b4a9883a39f3e481a4f935f688f431d55815f814bc4fb24e203df902bafb0d WatchSource:0}: Error finding container f3b4a9883a39f3e481a4f935f688f431d55815f814bc4fb24e203df902bafb0d: Status 404 returned error can't find the container with id f3b4a9883a39f3e481a4f935f688f431d55815f814bc4fb24e203df902bafb0d
Sep 16 14:45:00.953664 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:00.953571    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f88e1-29826165-d7jnd" event={"ID":"5e34253e-0fc4-42fe-9268-aacdcc6a9114","Type":"ContainerStarted","Data":"5b8c854eda7e5b6c54fe05e6e3adf077fe5a61259c399a695ae5e37ddfb51d4b"}
Sep 16 14:45:00.953664 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:00.953635    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f88e1-29826165-d7jnd" event={"ID":"5e34253e-0fc4-42fe-9268-aacdcc6a9114","Type":"ContainerStarted","Data":"f3b4a9883a39f3e481a4f935f688f431d55815f814bc4fb24e203df902bafb0d"}
Sep 16 14:45:00.969516 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:00.969456    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f88e1-29826165-d7jnd" podStartSLOduration=0.969440761 podStartE2EDuration="969.440761ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:45:00.96923945 +0000 UTC m=+1851.515981381" watchObservedRunningTime="2026-09-16 14:45:00.969440761 +0000 UTC m=+1851.516182691"
Sep 16 14:45:01.961383 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:01.961349    2553 generic.go:356] "Generic (PLEG): container finished" podID="5e34253e-0fc4-42fe-9268-aacdcc6a9114" containerID="5b8c854eda7e5b6c54fe05e6e3adf077fe5a61259c399a695ae5e37ddfb51d4b" exitCode=0
Sep 16 14:45:01.961796 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:01.961403    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f88e1-29826165-d7jnd" event={"ID":"5e34253e-0fc4-42fe-9268-aacdcc6a9114","Type":"ContainerDied","Data":"5b8c854eda7e5b6c54fe05e6e3adf077fe5a61259c399a695ae5e37ddfb51d4b"}
Sep 16 14:45:03.095701 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:03.095678    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f88e1-29826165-d7jnd"
Sep 16 14:45:03.973209 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:03.973173    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f88e1-29826165-d7jnd" event={"ID":"5e34253e-0fc4-42fe-9268-aacdcc6a9114","Type":"ContainerDied","Data":"f3b4a9883a39f3e481a4f935f688f431d55815f814bc4fb24e203df902bafb0d"}
Sep 16 14:45:03.973338 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:03.973227    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3b4a9883a39f3e481a4f935f688f431d55815f814bc4fb24e203df902bafb0d"
Sep 16 14:45:03.973338 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:03.973192    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f88e1-29826165-d7jnd"
Sep 16 14:45:12.882908 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:12.882863    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"]
Sep 16 14:45:12.883460 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:12.883443    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e34253e-0fc4-42fe-9268-aacdcc6a9114" containerName="cleanup"
Sep 16 14:45:12.883511 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:12.883462    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e34253e-0fc4-42fe-9268-aacdcc6a9114" containerName="cleanup"
Sep 16 14:45:12.883554 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:12.883522    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e34253e-0fc4-42fe-9268-aacdcc6a9114" containerName="cleanup"
Sep 16 14:45:12.886507 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:12.886490    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:12.889087 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:12.889062    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a70ddb-openshift-default-dockercfg-shkdd\""
Sep 16 14:45:12.896585 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:12.896558    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"]
Sep 16 14:45:13.024950 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.024919    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-credential-socket\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.025147 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.024959    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-data\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.025147 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.024994    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdvmn\" (UniqueName: \"kubernetes.io/projected/558668d8-3d9b-4626-b2e7-250c5b45ad1a-kube-api-access-wdvmn\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.025147 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.025050    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istiod-ca-cert\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.025147 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.025096    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-workload-certs\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.025147 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.025111    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-token\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.025147 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.025137    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-workload-socket\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.025374 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.025161    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-podinfo\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.025374 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.025233    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-envoy\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.126238 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.126206    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-envoy\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.126393 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.126243    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-credential-socket\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.126393 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.126268    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-data\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.126393 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.126300    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdvmn\" (UniqueName: \"kubernetes.io/projected/558668d8-3d9b-4626-b2e7-250c5b45ad1a-kube-api-access-wdvmn\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.126393 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.126324    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istiod-ca-cert\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.126604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.126416    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-workload-certs\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.126604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.126460    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-token\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.126604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.126499    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-workload-socket\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.126604 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.126542    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-podinfo\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.126826 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.126717    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-credential-socket\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.126826 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.126809    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-workload-certs\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.126976 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.126945    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-workload-socket\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.127072 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.127054    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-data\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.127129 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.127056    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istiod-ca-cert\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.128566 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.128536    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-envoy\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.128845 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.128828    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-podinfo\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.134114 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.134059    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-token\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.134114 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.134097    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdvmn\" (UniqueName: \"kubernetes.io/projected/558668d8-3d9b-4626-b2e7-250c5b45ad1a-kube-api-access-wdvmn\") pod \"e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") " pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.198042 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.198009    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:13.475294 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.475267    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"]
Sep 16 14:45:13.476562 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:45:13.476538    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558668d8_3d9b_4626_b2e7_250c5b45ad1a.slice/crio-0d01a9a20ebdec2282752cd97894b3b852e7b0bf7b0e70fc2eb6c4fcc5dfd274 WatchSource:0}: Error finding container 0d01a9a20ebdec2282752cd97894b3b852e7b0bf7b0e70fc2eb6c4fcc5dfd274: Status 404 returned error can't find the container with id 0d01a9a20ebdec2282752cd97894b3b852e7b0bf7b0e70fc2eb6c4fcc5dfd274
Sep 16 14:45:13.478986 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.478950    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:45:13.479061 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.479044    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:45:13.479100 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:13.479090    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:45:14.026754 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:14.026722    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86" event={"ID":"558668d8-3d9b-4626-b2e7-250c5b45ad1a","Type":"ContainerStarted","Data":"15c30995d3786c6a5651440fe3d41c7abf37b83d14e14224bed9fc7ac0738295"}
Sep 16 14:45:14.027150 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:14.026771    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86" event={"ID":"558668d8-3d9b-4626-b2e7-250c5b45ad1a","Type":"ContainerStarted","Data":"0d01a9a20ebdec2282752cd97894b3b852e7b0bf7b0e70fc2eb6c4fcc5dfd274"}
Sep 16 14:45:14.049123 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:14.049078    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86" podStartSLOduration=2.049062817 podStartE2EDuration="2.049062817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:45:14.046947408 +0000 UTC m=+1864.593689339" watchObservedRunningTime="2026-09-16 14:45:14.049062817 +0000 UTC m=+1864.595804747"
Sep 16 14:45:14.198543 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:14.198507    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:14.203262 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:14.203236    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:15.032448 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:15.032418    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:15.033226 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:15.033205    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:31.074049 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:31.074009    2553 patch_prober.go:28] interesting pod/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 14:45:31.074516 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:31.074087    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86" podUID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 14:45:32.126115 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:32.126083    2553 generic.go:356] "Generic (PLEG): container finished" podID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerID="15c30995d3786c6a5651440fe3d41c7abf37b83d14e14224bed9fc7ac0738295" exitCode=0
Sep 16 14:45:32.126115 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:32.126113    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86" event={"ID":"558668d8-3d9b-4626-b2e7-250c5b45ad1a","Type":"ContainerDied","Data":"15c30995d3786c6a5651440fe3d41c7abf37b83d14e14224bed9fc7ac0738295"}
Sep 16 14:45:32.126687 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:32.126665    2553 scope.go:117] "RemoveContainer" containerID="15c30995d3786c6a5651440fe3d41c7abf37b83d14e14224bed9fc7ac0738295"
Sep 16 14:45:32.127833 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:32.127800    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:45:32.127949 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:32.127880    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:45:32.127949 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:32.127932    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:45:33.133678 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:33.133644    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86" event={"ID":"558668d8-3d9b-4626-b2e7-250c5b45ad1a","Type":"ContainerStarted","Data":"566624d8cb7a69464ad51b0f7af0a7a08e2666f04e4e96f93e5344cccd7532ad"}
Sep 16 14:45:33.198819 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:33.198793    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:33.199025 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:33.199004    2553 patch_prober.go:28] interesting pod/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 16 14:45:33.199083 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:33.199049    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86" podUID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 16 14:45:34.199280 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:34.199240    2553 patch_prober.go:28] interesting pod/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 16 14:45:34.199667 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:34.199305    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86" podUID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 16 14:45:35.198432 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:35.198396    2553 patch_prober.go:28] interesting pod/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 16 14:45:35.198618 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:35.198466    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86" podUID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 16 14:45:36.198358 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:36.198321    2553 patch_prober.go:28] interesting pod/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 16 14:45:36.198728 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:36.198375    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86" podUID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 16 14:45:37.198532 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:37.198496    2553 patch_prober.go:28] interesting pod/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 16 14:45:37.198938 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:37.198561    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86" podUID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 16 14:45:38.199056 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:38.199023    2553 patch_prober.go:28] interesting pod/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 16 14:45:38.199434 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:38.199081    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86" podUID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 16 14:45:39.202435 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:39.202400    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:39.202840 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:39.202671    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:39.203364 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:39.203343    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:45:42.003637 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.003601    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"]
Sep 16 14:45:42.006765 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.006745    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.011123 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.011098    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3eaa3c-openshift-default-dockercfg-lv85h\""
Sep 16 14:45:42.023361 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.023336    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"]
Sep 16 14:45:42.109001 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.108956    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-workload-certs\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.109210 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.109011    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-envoy\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.109210 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.109084    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2w6c\" (UniqueName: \"kubernetes.io/projected/20069523-9db2-4a2f-a346-25dfddb2ce0e-kube-api-access-h2w6c\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.109210 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.109113    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20069523-9db2-4a2f-a346-25dfddb2ce0e-istiod-ca-cert\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.109210 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.109145    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-token\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.109210 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.109175    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-podinfo\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.109479 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.109292    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-workload-socket\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.109479 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.109332    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-credential-socket\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.109479 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.109379    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-data\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.210555 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.210519    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-podinfo\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.210765 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.210619    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-workload-socket\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.210765 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.210653    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-credential-socket\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.210765 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.210698    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-data\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.210765 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.210730    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-workload-certs\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.210765 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.210758    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-envoy\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.211079 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.210798    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2w6c\" (UniqueName: \"kubernetes.io/projected/20069523-9db2-4a2f-a346-25dfddb2ce0e-kube-api-access-h2w6c\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.211079 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.210828    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20069523-9db2-4a2f-a346-25dfddb2ce0e-istiod-ca-cert\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.211079 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.210876    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-token\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.211244 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.211120    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-workload-socket\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.211244 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.211149    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-data\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.211640 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.211617    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-credential-socket\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.211861 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.211839    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-workload-certs\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.212160 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.212142    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20069523-9db2-4a2f-a346-25dfddb2ce0e-istiod-ca-cert\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.213394 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.213374    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-podinfo\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.213487 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.213378    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-envoy\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.221379 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.221361    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-token\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.221669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.221645    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2w6c\" (UniqueName: \"kubernetes.io/projected/20069523-9db2-4a2f-a346-25dfddb2ce0e-kube-api-access-h2w6c\") pod \"e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") " pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.317851 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.317763    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:42.600015 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.599983    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"]
Sep 16 14:45:42.601574 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:45:42.601548    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20069523_9db2_4a2f_a346_25dfddb2ce0e.slice/crio-26e33a17d07fd833290f8826b2c25bd9cdd162adaf2bf5f96339d4c56835292d WatchSource:0}: Error finding container 26e33a17d07fd833290f8826b2c25bd9cdd162adaf2bf5f96339d4c56835292d: Status 404 returned error can't find the container with id 26e33a17d07fd833290f8826b2c25bd9cdd162adaf2bf5f96339d4c56835292d
Sep 16 14:45:42.604200 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.604170    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:45:42.604321 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.604228    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:45:42.604321 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:42.604264    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:45:43.195779 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:43.195744    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn" event={"ID":"20069523-9db2-4a2f-a346-25dfddb2ce0e","Type":"ContainerStarted","Data":"d9306a60f85a3af241b731038c39b8185ea1c207970ac0cc4a6ee3a98154abbe"}
Sep 16 14:45:43.196206 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:43.195797    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn" event={"ID":"20069523-9db2-4a2f-a346-25dfddb2ce0e","Type":"ContainerStarted","Data":"26e33a17d07fd833290f8826b2c25bd9cdd162adaf2bf5f96339d4c56835292d"}
Sep 16 14:45:43.217135 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:43.217090    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn" podStartSLOduration=2.217073461 podStartE2EDuration="2.217073461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:45:43.215032318 +0000 UTC m=+1893.761774290" watchObservedRunningTime="2026-09-16 14:45:43.217073461 +0000 UTC m=+1893.763815391"
Sep 16 14:45:43.318492 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:43.318457    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:43.323343 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:43.323320    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:44.201674 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:44.201644    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:45:44.202624 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:45:44.202606    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:46:03.318673 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:03.318636    2553 generic.go:356] "Generic (PLEG): container finished" podID="20069523-9db2-4a2f-a346-25dfddb2ce0e" containerID="d9306a60f85a3af241b731038c39b8185ea1c207970ac0cc4a6ee3a98154abbe" exitCode=0
Sep 16 14:46:03.319089 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:03.318703    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn" event={"ID":"20069523-9db2-4a2f-a346-25dfddb2ce0e","Type":"ContainerDied","Data":"d9306a60f85a3af241b731038c39b8185ea1c207970ac0cc4a6ee3a98154abbe"}
Sep 16 14:46:03.319230 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:03.319212    2553 scope.go:117] "RemoveContainer" containerID="d9306a60f85a3af241b731038c39b8185ea1c207970ac0cc4a6ee3a98154abbe"
Sep 16 14:46:03.320338 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:03.320312    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:46:03.320399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:03.320378    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:46:03.320439 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:03.320416    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:46:04.325987 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:04.325953    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn" event={"ID":"20069523-9db2-4a2f-a346-25dfddb2ce0e","Type":"ContainerStarted","Data":"7fb15a71ab69063022d9dcb2036f11e4e7d88f869dc68de7c2c0cc0e2246eb08"}
Sep 16 14:46:05.318347 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:05.318310    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:46:05.318614 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:05.318589    2553 patch_prober.go:28] interesting pod/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 16 14:46:05.318713 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:05.318645    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn" podUID="20069523-9db2-4a2f-a346-25dfddb2ce0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 16 14:46:06.318411 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:06.318376    2553 patch_prober.go:28] interesting pod/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 16 14:46:06.318793 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:06.318439    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn" podUID="20069523-9db2-4a2f-a346-25dfddb2ce0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 16 14:46:07.319020 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:07.318984    2553 patch_prober.go:28] interesting pod/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 16 14:46:07.319400 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:07.319039    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn" podUID="20069523-9db2-4a2f-a346-25dfddb2ce0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 16 14:46:08.319138 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:08.319103    2553 patch_prober.go:28] interesting pod/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 16 14:46:08.319524 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:08.319194    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn" podUID="20069523-9db2-4a2f-a346-25dfddb2ce0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 16 14:46:09.471814 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:09.471775    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:46:09.472420 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:09.472124    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:46:09.475075 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:09.475052    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:46:15.333415 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:15.333388    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-g9lgv_2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b/payload-processing/0.log"
Sep 16 14:46:51.446693 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:51.446658    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-g9lgv_2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b/payload-processing/0.log"
Sep 16 14:46:52.174744 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.174713    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"]
Sep 16 14:46:52.178608 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.178589    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.181326 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.181307    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-a26a50-openshift-default-dockercfg-n5swx\""
Sep 16 14:46:52.191383 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.191361    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"]
Sep 16 14:46:52.232136 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.232105    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-data\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.232393 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.232367    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dng9q\" (UniqueName: \"kubernetes.io/projected/dfbf848f-1191-46e4-b74d-84b3a9c9e247-kube-api-access-dng9q\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.232495 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.232419    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.232495 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.232477    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-envoy\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.232597 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.232503    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-podinfo\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.232597 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.232551    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-credential-socket\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.232597 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.232575    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-workload-socket\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.232722 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.232633    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-token\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.232722 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.232709    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-workload-certs\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.334160 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.334124    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-data\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.334327 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.334185    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dng9q\" (UniqueName: \"kubernetes.io/projected/dfbf848f-1191-46e4-b74d-84b3a9c9e247-kube-api-access-dng9q\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.334327 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.334216    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.334327 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.334252    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-envoy\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.334327 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.334279    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-podinfo\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.334327 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.334309    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-credential-socket\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.334599 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.334338    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-workload-socket\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.334599 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.334497    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-token\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.334599 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.334540    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-data\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.334762 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.334608    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-workload-certs\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.335036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.334998    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-credential-socket\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.335036 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.335027    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-workload-socket\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.335191 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.335116    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-workload-certs\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.335358 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.335336    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.336565 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.336547    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-envoy\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.337160 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.337127    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-podinfo\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.361951 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.361926    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dng9q\" (UniqueName: \"kubernetes.io/projected/dfbf848f-1191-46e4-b74d-84b3a9c9e247-kube-api-access-dng9q\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.363129 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.363107    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-token\") pod \"e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") " pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.490525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.490491    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:52.773303 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.773274    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"]
Sep 16 14:46:52.775244 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:46:52.775221    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfbf848f_1191_46e4_b74d_84b3a9c9e247.slice/crio-d99085b2b3f823ea072170eaa21a74e73b00b40f4d855ec878c6622720adb26a WatchSource:0}: Error finding container d99085b2b3f823ea072170eaa21a74e73b00b40f4d855ec878c6622720adb26a: Status 404 returned error can't find the container with id d99085b2b3f823ea072170eaa21a74e73b00b40f4d855ec878c6622720adb26a
Sep 16 14:46:52.777373 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.777346    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:46:52.777430 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.777406    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:46:52.777466 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:52.777434    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:46:53.592823 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:53.592786    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh" event={"ID":"dfbf848f-1191-46e4-b74d-84b3a9c9e247","Type":"ContainerStarted","Data":"88f5870227a4780ea47326a13c9707f22081b1a09935c03e5948b9e755ba4360"}
Sep 16 14:46:53.593293 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:53.592879    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh" event={"ID":"dfbf848f-1191-46e4-b74d-84b3a9c9e247","Type":"ContainerStarted","Data":"d99085b2b3f823ea072170eaa21a74e73b00b40f4d855ec878c6622720adb26a"}
Sep 16 14:46:53.644040 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:53.643994    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh" podStartSLOduration=1.643980553 podStartE2EDuration="1.643980553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:46:53.642811101 +0000 UTC m=+1964.189553032" watchObservedRunningTime="2026-09-16 14:46:53.643980553 +0000 UTC m=+1964.190722483"
Sep 16 14:46:54.490816 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:54.490783    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:54.495539 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:54.495510    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:54.598249 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:54.598219    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:54.599101 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:54.599081    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:46:59.004484 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.004455    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"]
Sep 16 14:46:59.008121 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.008104    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:46:59.011539 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.011515    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-a26a50-serving-cert\""
Sep 16 14:46:59.025266 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.025237    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"]
Sep 16 14:46:59.103287 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.103225    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53fe12fd-e9ec-4642-8dde-3618e994e37b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:46:59.103464 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.103323    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9r9q\" (UniqueName: \"kubernetes.io/projected/53fe12fd-e9ec-4642-8dde-3618e994e37b-kube-api-access-c9r9q\") pod \"maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:46:59.103464 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.103395    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53fe12fd-e9ec-4642-8dde-3618e994e37b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:46:59.103464 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.103437    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53fe12fd-e9ec-4642-8dde-3618e994e37b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:46:59.204642 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.204608    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53fe12fd-e9ec-4642-8dde-3618e994e37b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:46:59.204814 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.204653    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53fe12fd-e9ec-4642-8dde-3618e994e37b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:46:59.204814 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.204723    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53fe12fd-e9ec-4642-8dde-3618e994e37b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:46:59.204814 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.204756    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9r9q\" (UniqueName: \"kubernetes.io/projected/53fe12fd-e9ec-4642-8dde-3618e994e37b-kube-api-access-c9r9q\") pod \"maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:46:59.204814 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:46:59.204772    2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-serving-cert: secret "maas-api-e2e-ipp-cleanup-a26a50-serving-cert" not found
Sep 16 14:46:59.205004 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:46:59.204862    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53fe12fd-e9ec-4642-8dde-3618e994e37b-maas-api-tls podName:53fe12fd-e9ec-4642-8dde-3618e994e37b nodeName:}" failed. No retries permitted until 2026-09-16 14:46:59.704839591 +0000 UTC m=+1970.251581502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/53fe12fd-e9ec-4642-8dde-3618e994e37b-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t" (UID: "53fe12fd-e9ec-4642-8dde-3618e994e37b") : secret "maas-api-e2e-ipp-cleanup-a26a50-serving-cert" not found
Sep 16 14:46:59.205426 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.205408    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53fe12fd-e9ec-4642-8dde-3618e994e37b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:46:59.207158 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.207143    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53fe12fd-e9ec-4642-8dde-3618e994e37b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:46:59.217045 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.217018    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9r9q\" (UniqueName: \"kubernetes.io/projected/53fe12fd-e9ec-4642-8dde-3618e994e37b-kube-api-access-c9r9q\") pod \"maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:46:59.711606 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.711568    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53fe12fd-e9ec-4642-8dde-3618e994e37b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:46:59.713999 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.713971    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53fe12fd-e9ec-4642-8dde-3618e994e37b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:46:59.918577 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:46:59.918546    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:47:00.194118 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:00.194093    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"]
Sep 16 14:47:00.195838 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:47:00.195812    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fe12fd_e9ec_4642_8dde_3618e994e37b.slice/crio-8793cc730d874d5a72613c077215fc5b03f3ecd07e45fbd4e0cb963186abef99 WatchSource:0}: Error finding container 8793cc730d874d5a72613c077215fc5b03f3ecd07e45fbd4e0cb963186abef99: Status 404 returned error can't find the container with id 8793cc730d874d5a72613c077215fc5b03f3ecd07e45fbd4e0cb963186abef99
Sep 16 14:47:00.197265 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:00.197248    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:47:00.629612 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:00.629582    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t" event={"ID":"53fe12fd-e9ec-4642-8dde-3618e994e37b","Type":"ContainerStarted","Data":"8793cc730d874d5a72613c077215fc5b03f3ecd07e45fbd4e0cb963186abef99"}
Sep 16 14:47:01.637230 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:01.637192    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t" event={"ID":"53fe12fd-e9ec-4642-8dde-3618e994e37b","Type":"ContainerStarted","Data":"0f83c4c9edea5a654da86bc3d6bb5ff8c34b3a6e8dfe6b3bc9d16951434675df"}
Sep 16 14:47:01.637615 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:01.637356    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:47:01.658296 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:01.658239    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t" podStartSLOduration=3.24735218 podStartE2EDuration="3.658218047s" totalImagesPullingTime="410.865867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:47:01.655228433 +0000 UTC m=+1972.201970363" watchObservedRunningTime="2026-09-16 14:47:01.658218047 +0000 UTC m=+1972.204959978"
Sep 16 14:47:07.648854 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:07.648778    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:47:14.720586 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:14.720555    2553 generic.go:356] "Generic (PLEG): container finished" podID="dfbf848f-1191-46e4-b74d-84b3a9c9e247" containerID="88f5870227a4780ea47326a13c9707f22081b1a09935c03e5948b9e755ba4360" exitCode=0
Sep 16 14:47:14.720999 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:14.720586    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh" event={"ID":"dfbf848f-1191-46e4-b74d-84b3a9c9e247","Type":"ContainerDied","Data":"88f5870227a4780ea47326a13c9707f22081b1a09935c03e5948b9e755ba4360"}
Sep 16 14:47:14.721047 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:14.721024    2553 scope.go:117] "RemoveContainer" containerID="88f5870227a4780ea47326a13c9707f22081b1a09935c03e5948b9e755ba4360"
Sep 16 14:47:14.722153 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:14.722129    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:47:14.722233 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:14.722201    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:47:14.722270 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:14.722237    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 14:47:15.728356 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:15.728320    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh" event={"ID":"dfbf848f-1191-46e4-b74d-84b3a9c9e247","Type":"ContainerStarted","Data":"134fc0694e4c19b8e74aa09ada7a1c2603cce1b6f5ef71567806ab3ec4bfd418"}
Sep 16 14:47:16.491175 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:16.491135    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:47:16.491425 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:16.491345    2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 16 14:47:16.491425 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:16.491403    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh" podUID="dfbf848f-1191-46e4-b74d-84b3a9c9e247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 16 14:47:17.491325 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:17.491288    2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 16 14:47:17.491710 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:17.491350    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh" podUID="dfbf848f-1191-46e4-b74d-84b3a9c9e247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 16 14:47:18.491774 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:18.491737    2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 16 14:47:18.492211 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:18.491806    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh" podUID="dfbf848f-1191-46e4-b74d-84b3a9c9e247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 16 14:47:19.491035 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:19.490998    2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 16 14:47:19.491241 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:19.491076    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh" podUID="dfbf848f-1191-46e4-b74d-84b3a9c9e247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 16 14:47:20.491194 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:20.491158    2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 16 14:47:20.491574 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:20.491211    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh" podUID="dfbf848f-1191-46e4-b74d-84b3a9c9e247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 16 14:47:21.495041 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:21.495006    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:47:21.495435 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:21.495267    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:47:21.496122 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:21.496102    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:47:37.297390 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.297356    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"]
Sep 16 14:47:37.297936 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.297615    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t" podUID="53fe12fd-e9ec-4642-8dde-3618e994e37b" containerName="maas-api" containerID="cri-o://0f83c4c9edea5a654da86bc3d6bb5ff8c34b3a6e8dfe6b3bc9d16951434675df" gracePeriod=30
Sep 16 14:47:37.551714 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.551643    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:47:37.690934 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.690904    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9r9q\" (UniqueName: \"kubernetes.io/projected/53fe12fd-e9ec-4642-8dde-3618e994e37b-kube-api-access-c9r9q\") pod \"53fe12fd-e9ec-4642-8dde-3618e994e37b\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") "
Sep 16 14:47:37.691128 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.690947    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53fe12fd-e9ec-4642-8dde-3618e994e37b-metrics-certs\") pod \"53fe12fd-e9ec-4642-8dde-3618e994e37b\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") "
Sep 16 14:47:37.691128 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.690994    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53fe12fd-e9ec-4642-8dde-3618e994e37b-openshift-service-ca\") pod \"53fe12fd-e9ec-4642-8dde-3618e994e37b\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") "
Sep 16 14:47:37.691128 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.691011    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53fe12fd-e9ec-4642-8dde-3618e994e37b-maas-api-tls\") pod \"53fe12fd-e9ec-4642-8dde-3618e994e37b\" (UID: \"53fe12fd-e9ec-4642-8dde-3618e994e37b\") "
Sep 16 14:47:37.691375 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.691338    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53fe12fd-e9ec-4642-8dde-3618e994e37b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "53fe12fd-e9ec-4642-8dde-3618e994e37b" (UID: "53fe12fd-e9ec-4642-8dde-3618e994e37b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:47:37.693151 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.693129    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53fe12fd-e9ec-4642-8dde-3618e994e37b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "53fe12fd-e9ec-4642-8dde-3618e994e37b" (UID: "53fe12fd-e9ec-4642-8dde-3618e994e37b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:47:37.693255 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.693216    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fe12fd-e9ec-4642-8dde-3618e994e37b-kube-api-access-c9r9q" (OuterVolumeSpecName: "kube-api-access-c9r9q") pod "53fe12fd-e9ec-4642-8dde-3618e994e37b" (UID: "53fe12fd-e9ec-4642-8dde-3618e994e37b"). InnerVolumeSpecName "kube-api-access-c9r9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:47:37.693319 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.693269    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53fe12fd-e9ec-4642-8dde-3618e994e37b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "53fe12fd-e9ec-4642-8dde-3618e994e37b" (UID: "53fe12fd-e9ec-4642-8dde-3618e994e37b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:47:37.791963 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.791935    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9r9q\" (UniqueName: \"kubernetes.io/projected/53fe12fd-e9ec-4642-8dde-3618e994e37b-kube-api-access-c9r9q\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:47:37.791963 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.791961    2553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53fe12fd-e9ec-4642-8dde-3618e994e37b-metrics-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:47:37.791963 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.791971    2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53fe12fd-e9ec-4642-8dde-3618e994e37b-openshift-service-ca\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:47:37.791963 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.791980    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53fe12fd-e9ec-4642-8dde-3618e994e37b-maas-api-tls\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:47:37.846466 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.846385    2553 generic.go:356] "Generic (PLEG): container finished" podID="53fe12fd-e9ec-4642-8dde-3618e994e37b" containerID="0f83c4c9edea5a654da86bc3d6bb5ff8c34b3a6e8dfe6b3bc9d16951434675df" exitCode=0
Sep 16 14:47:37.846466 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.846437    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"
Sep 16 14:47:37.846466 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.846454    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t" event={"ID":"53fe12fd-e9ec-4642-8dde-3618e994e37b","Type":"ContainerDied","Data":"0f83c4c9edea5a654da86bc3d6bb5ff8c34b3a6e8dfe6b3bc9d16951434675df"}
Sep 16 14:47:37.846766 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.846495    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t" event={"ID":"53fe12fd-e9ec-4642-8dde-3618e994e37b","Type":"ContainerDied","Data":"8793cc730d874d5a72613c077215fc5b03f3ecd07e45fbd4e0cb963186abef99"}
Sep 16 14:47:37.846766 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.846517    2553 scope.go:117] "RemoveContainer" containerID="0f83c4c9edea5a654da86bc3d6bb5ff8c34b3a6e8dfe6b3bc9d16951434675df"
Sep 16 14:47:37.861991 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.861969    2553 scope.go:117] "RemoveContainer" containerID="0f83c4c9edea5a654da86bc3d6bb5ff8c34b3a6e8dfe6b3bc9d16951434675df"
Sep 16 14:47:37.862303 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:47:37.862283    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f83c4c9edea5a654da86bc3d6bb5ff8c34b3a6e8dfe6b3bc9d16951434675df\": container with ID starting with 0f83c4c9edea5a654da86bc3d6bb5ff8c34b3a6e8dfe6b3bc9d16951434675df not found: ID does not exist" containerID="0f83c4c9edea5a654da86bc3d6bb5ff8c34b3a6e8dfe6b3bc9d16951434675df"
Sep 16 14:47:37.862382 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.862313    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f83c4c9edea5a654da86bc3d6bb5ff8c34b3a6e8dfe6b3bc9d16951434675df"} err="failed to get container status \"0f83c4c9edea5a654da86bc3d6bb5ff8c34b3a6e8dfe6b3bc9d16951434675df\": rpc error: code = NotFound desc = could not find container \"0f83c4c9edea5a654da86bc3d6bb5ff8c34b3a6e8dfe6b3bc9d16951434675df\": container with ID starting with 0f83c4c9edea5a654da86bc3d6bb5ff8c34b3a6e8dfe6b3bc9d16951434675df not found: ID does not exist"
Sep 16 14:47:37.870346 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.870323    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"]
Sep 16 14:47:37.876584 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.876556    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a26a50-578cb56cff-xdr5t"]
Sep 16 14:47:37.918676 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:37.918648    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53fe12fd-e9ec-4642-8dde-3618e994e37b" path="/var/lib/kubelet/pods/53fe12fd-e9ec-4642-8dde-3618e994e37b/volumes"
Sep 16 14:47:44.120391 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:44.120352    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"]
Sep 16 14:47:44.121014 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:44.120584    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh" podUID="dfbf848f-1191-46e4-b74d-84b3a9c9e247" containerName="istio-proxy" containerID="cri-o://134fc0694e4c19b8e74aa09ada7a1c2603cce1b6f5ef71567806ab3ec4bfd418" gracePeriod=30
Sep 16 14:47:49.396826 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.396767    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:47:49.510670 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.510631    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-data\") pod \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") "
Sep 16 14:47:49.510670 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.510677    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-podinfo\") pod \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") "
Sep 16 14:47:49.510965 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.510717    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-token\") pod \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") "
Sep 16 14:47:49.510965 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.510832    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-envoy\") pod \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") "
Sep 16 14:47:49.510965 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.510880    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istiod-ca-cert\") pod \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") "
Sep 16 14:47:49.510965 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.510949    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-credential-socket\") pod \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") "
Sep 16 14:47:49.511181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.510966    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-data" (OuterVolumeSpecName: "istio-data") pod "dfbf848f-1191-46e4-b74d-84b3a9c9e247" (UID: "dfbf848f-1191-46e4-b74d-84b3a9c9e247"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:47:49.511181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.510984    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-workload-socket\") pod \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") "
Sep 16 14:47:49.511181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.511076    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dng9q\" (UniqueName: \"kubernetes.io/projected/dfbf848f-1191-46e4-b74d-84b3a9c9e247-kube-api-access-dng9q\") pod \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") "
Sep 16 14:47:49.511328 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.511204    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dfbf848f-1191-46e4-b74d-84b3a9c9e247" (UID: "dfbf848f-1191-46e4-b74d-84b3a9c9e247"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:47:49.511328 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.511219    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dfbf848f-1191-46e4-b74d-84b3a9c9e247" (UID: "dfbf848f-1191-46e4-b74d-84b3a9c9e247"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:47:49.511328 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.511287    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dfbf848f-1191-46e4-b74d-84b3a9c9e247" (UID: "dfbf848f-1191-46e4-b74d-84b3a9c9e247"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:47:49.511463 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.511105    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-workload-certs\") pod \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\" (UID: \"dfbf848f-1191-46e4-b74d-84b3a9c9e247\") "
Sep 16 14:47:49.511515 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.511474    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dfbf848f-1191-46e4-b74d-84b3a9c9e247" (UID: "dfbf848f-1191-46e4-b74d-84b3a9c9e247"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:47:49.511853 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.511832    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istiod-ca-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:47:49.511937 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.511860    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-credential-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:47:49.511937 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.511876    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-workload-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:47:49.511937 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.511906    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-workload-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:47:49.511937 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.511920    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-data\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:47:49.513142 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.513118    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dfbf848f-1191-46e4-b74d-84b3a9c9e247" (UID: "dfbf848f-1191-46e4-b74d-84b3a9c9e247"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:47:49.513248 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.513140    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbf848f-1191-46e4-b74d-84b3a9c9e247-kube-api-access-dng9q" (OuterVolumeSpecName: "kube-api-access-dng9q") pod "dfbf848f-1191-46e4-b74d-84b3a9c9e247" (UID: "dfbf848f-1191-46e4-b74d-84b3a9c9e247"). InnerVolumeSpecName "kube-api-access-dng9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:47:49.513415 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.513393    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dfbf848f-1191-46e4-b74d-84b3a9c9e247" (UID: "dfbf848f-1191-46e4-b74d-84b3a9c9e247"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:47:49.513478 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.513394    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-token" (OuterVolumeSpecName: "istio-token") pod "dfbf848f-1191-46e4-b74d-84b3a9c9e247" (UID: "dfbf848f-1191-46e4-b74d-84b3a9c9e247"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:47:49.613415 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.613380    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dng9q\" (UniqueName: \"kubernetes.io/projected/dfbf848f-1191-46e4-b74d-84b3a9c9e247-kube-api-access-dng9q\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:47:49.613415 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.613409    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-podinfo\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:47:49.613415 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.613418    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-token\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:47:49.613639 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.613426    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbf848f-1191-46e4-b74d-84b3a9c9e247-istio-envoy\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:47:49.916039 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.916000    2553 generic.go:356] "Generic (PLEG): container finished" podID="dfbf848f-1191-46e4-b74d-84b3a9c9e247" containerID="134fc0694e4c19b8e74aa09ada7a1c2603cce1b6f5ef71567806ab3ec4bfd418" exitCode=0
Sep 16 14:47:49.916196 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.916077    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"
Sep 16 14:47:49.919250 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.919223    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh" event={"ID":"dfbf848f-1191-46e4-b74d-84b3a9c9e247","Type":"ContainerDied","Data":"134fc0694e4c19b8e74aa09ada7a1c2603cce1b6f5ef71567806ab3ec4bfd418"}
Sep 16 14:47:49.919365 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.919265    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh" event={"ID":"dfbf848f-1191-46e4-b74d-84b3a9c9e247","Type":"ContainerDied","Data":"d99085b2b3f823ea072170eaa21a74e73b00b40f4d855ec878c6622720adb26a"}
Sep 16 14:47:49.919365 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.919287    2553 scope.go:117] "RemoveContainer" containerID="134fc0694e4c19b8e74aa09ada7a1c2603cce1b6f5ef71567806ab3ec4bfd418"
Sep 16 14:47:49.929157 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.929140    2553 scope.go:117] "RemoveContainer" containerID="88f5870227a4780ea47326a13c9707f22081b1a09935c03e5948b9e755ba4360"
Sep 16 14:47:49.937028 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.937008    2553 scope.go:117] "RemoveContainer" containerID="134fc0694e4c19b8e74aa09ada7a1c2603cce1b6f5ef71567806ab3ec4bfd418"
Sep 16 14:47:49.937271 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:47:49.937248    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134fc0694e4c19b8e74aa09ada7a1c2603cce1b6f5ef71567806ab3ec4bfd418\": container with ID starting with 134fc0694e4c19b8e74aa09ada7a1c2603cce1b6f5ef71567806ab3ec4bfd418 not found: ID does not exist" containerID="134fc0694e4c19b8e74aa09ada7a1c2603cce1b6f5ef71567806ab3ec4bfd418"
Sep 16 14:47:49.937347 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.937276    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134fc0694e4c19b8e74aa09ada7a1c2603cce1b6f5ef71567806ab3ec4bfd418"} err="failed to get container status \"134fc0694e4c19b8e74aa09ada7a1c2603cce1b6f5ef71567806ab3ec4bfd418\": rpc error: code = NotFound desc = could not find container \"134fc0694e4c19b8e74aa09ada7a1c2603cce1b6f5ef71567806ab3ec4bfd418\": container with ID starting with 134fc0694e4c19b8e74aa09ada7a1c2603cce1b6f5ef71567806ab3ec4bfd418 not found: ID does not exist"
Sep 16 14:47:49.937347 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.937295    2553 scope.go:117] "RemoveContainer" containerID="88f5870227a4780ea47326a13c9707f22081b1a09935c03e5948b9e755ba4360"
Sep 16 14:47:49.937563 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:47:49.937539    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f5870227a4780ea47326a13c9707f22081b1a09935c03e5948b9e755ba4360\": container with ID starting with 88f5870227a4780ea47326a13c9707f22081b1a09935c03e5948b9e755ba4360 not found: ID does not exist" containerID="88f5870227a4780ea47326a13c9707f22081b1a09935c03e5948b9e755ba4360"
Sep 16 14:47:49.937664 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.937568    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f5870227a4780ea47326a13c9707f22081b1a09935c03e5948b9e755ba4360"} err="failed to get container status \"88f5870227a4780ea47326a13c9707f22081b1a09935c03e5948b9e755ba4360\": rpc error: code = NotFound desc = could not find container \"88f5870227a4780ea47326a13c9707f22081b1a09935c03e5948b9e755ba4360\": container with ID starting with 88f5870227a4780ea47326a13c9707f22081b1a09935c03e5948b9e755ba4360 not found: ID does not exist"
Sep 16 14:47:49.942267 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.942245    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"]
Sep 16 14:47:49.945822 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:49.945800    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a26a50-openshift-default-7f6b8d74d6-jtmqh"]
Sep 16 14:47:51.918227 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:47:51.918194    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfbf848f-1191-46e4-b74d-84b3a9c9e247" path="/var/lib/kubelet/pods/dfbf848f-1191-46e4-b74d-84b3a9c9e247/volumes"
Sep 16 14:48:52.325516 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:52.325484    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"]
Sep 16 14:48:52.326044 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:52.325723    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86" podUID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerName="istio-proxy" containerID="cri-o://566624d8cb7a69464ad51b0f7af0a7a08e2666f04e4e96f93e5344cccd7532ad" gracePeriod=30
Sep 16 14:48:54.205355 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:54.205310    2553 patch_prober.go:28] interesting pod/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 14:48:54.205714 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:54.205378    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86" podUID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 14:48:57.573731 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.573707    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:48:57.658954 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.658864    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istiod-ca-cert\") pod \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") "
Sep 16 14:48:57.658954 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.658919    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-workload-certs\") pod \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") "
Sep 16 14:48:57.658954 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.658938    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-podinfo\") pod \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") "
Sep 16 14:48:57.659192 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.658991    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdvmn\" (UniqueName: \"kubernetes.io/projected/558668d8-3d9b-4626-b2e7-250c5b45ad1a-kube-api-access-wdvmn\") pod \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") "
Sep 16 14:48:57.659192 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.659017    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-credential-socket\") pod \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") "
Sep 16 14:48:57.659192 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.659051    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-workload-socket\") pod \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") "
Sep 16 14:48:57.659192 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.659066    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-envoy\") pod \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") "
Sep 16 14:48:57.659192 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.659091    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-token\") pod \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") "
Sep 16 14:48:57.659192 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.659157    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-data\") pod \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\" (UID: \"558668d8-3d9b-4626-b2e7-250c5b45ad1a\") "
Sep 16 14:48:57.659540 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.659342    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "558668d8-3d9b-4626-b2e7-250c5b45ad1a" (UID: "558668d8-3d9b-4626-b2e7-250c5b45ad1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:48:57.659540 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.659351    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "558668d8-3d9b-4626-b2e7-250c5b45ad1a" (UID: "558668d8-3d9b-4626-b2e7-250c5b45ad1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:48:57.659540 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.659495    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istiod-ca-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:48:57.659540 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.659512    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-workload-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:48:57.659725 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.659560    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-data" (OuterVolumeSpecName: "istio-data") pod "558668d8-3d9b-4626-b2e7-250c5b45ad1a" (UID: "558668d8-3d9b-4626-b2e7-250c5b45ad1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:48:57.659725 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.659578    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "558668d8-3d9b-4626-b2e7-250c5b45ad1a" (UID: "558668d8-3d9b-4626-b2e7-250c5b45ad1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:48:57.659804 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.659750    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "558668d8-3d9b-4626-b2e7-250c5b45ad1a" (UID: "558668d8-3d9b-4626-b2e7-250c5b45ad1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:48:57.661222 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.661200    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "558668d8-3d9b-4626-b2e7-250c5b45ad1a" (UID: "558668d8-3d9b-4626-b2e7-250c5b45ad1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:48:57.661383 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.661364    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-token" (OuterVolumeSpecName: "istio-token") pod "558668d8-3d9b-4626-b2e7-250c5b45ad1a" (UID: "558668d8-3d9b-4626-b2e7-250c5b45ad1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:48:57.661635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.661609    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "558668d8-3d9b-4626-b2e7-250c5b45ad1a" (UID: "558668d8-3d9b-4626-b2e7-250c5b45ad1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:48:57.661635 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.661615    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558668d8-3d9b-4626-b2e7-250c5b45ad1a-kube-api-access-wdvmn" (OuterVolumeSpecName: "kube-api-access-wdvmn") pod "558668d8-3d9b-4626-b2e7-250c5b45ad1a" (UID: "558668d8-3d9b-4626-b2e7-250c5b45ad1a"). InnerVolumeSpecName "kube-api-access-wdvmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:48:57.760652 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.760624    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-data\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:48:57.760652 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.760649    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-podinfo\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:48:57.760829 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.760659    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdvmn\" (UniqueName: \"kubernetes.io/projected/558668d8-3d9b-4626-b2e7-250c5b45ad1a-kube-api-access-wdvmn\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:48:57.760829 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.760669    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-credential-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:48:57.760829 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.760678    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-workload-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:48:57.760829 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.760685    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-envoy\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:48:57.760829 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:57.760694    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558668d8-3d9b-4626-b2e7-250c5b45ad1a-istio-token\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:48:58.271544 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:58.271512    2553 generic.go:356] "Generic (PLEG): container finished" podID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerID="566624d8cb7a69464ad51b0f7af0a7a08e2666f04e4e96f93e5344cccd7532ad" exitCode=0
Sep 16 14:48:58.271751 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:58.271567    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"
Sep 16 14:48:58.271751 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:58.271599    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86" event={"ID":"558668d8-3d9b-4626-b2e7-250c5b45ad1a","Type":"ContainerDied","Data":"566624d8cb7a69464ad51b0f7af0a7a08e2666f04e4e96f93e5344cccd7532ad"}
Sep 16 14:48:58.271751 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:58.271653    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86" event={"ID":"558668d8-3d9b-4626-b2e7-250c5b45ad1a","Type":"ContainerDied","Data":"0d01a9a20ebdec2282752cd97894b3b852e7b0bf7b0e70fc2eb6c4fcc5dfd274"}
Sep 16 14:48:58.271751 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:58.271679    2553 scope.go:117] "RemoveContainer" containerID="566624d8cb7a69464ad51b0f7af0a7a08e2666f04e4e96f93e5344cccd7532ad"
Sep 16 14:48:58.281013 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:58.280968    2553 scope.go:117] "RemoveContainer" containerID="15c30995d3786c6a5651440fe3d41c7abf37b83d14e14224bed9fc7ac0738295"
Sep 16 14:48:58.289720 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:58.289700    2553 scope.go:117] "RemoveContainer" containerID="566624d8cb7a69464ad51b0f7af0a7a08e2666f04e4e96f93e5344cccd7532ad"
Sep 16 14:48:58.290044 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:48:58.290026    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566624d8cb7a69464ad51b0f7af0a7a08e2666f04e4e96f93e5344cccd7532ad\": container with ID starting with 566624d8cb7a69464ad51b0f7af0a7a08e2666f04e4e96f93e5344cccd7532ad not found: ID does not exist" containerID="566624d8cb7a69464ad51b0f7af0a7a08e2666f04e4e96f93e5344cccd7532ad"
Sep 16 14:48:58.290129 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:58.290051    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566624d8cb7a69464ad51b0f7af0a7a08e2666f04e4e96f93e5344cccd7532ad"} err="failed to get container status \"566624d8cb7a69464ad51b0f7af0a7a08e2666f04e4e96f93e5344cccd7532ad\": rpc error: code = NotFound desc = could not find container \"566624d8cb7a69464ad51b0f7af0a7a08e2666f04e4e96f93e5344cccd7532ad\": container with ID starting with 566624d8cb7a69464ad51b0f7af0a7a08e2666f04e4e96f93e5344cccd7532ad not found: ID does not exist"
Sep 16 14:48:58.290129 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:58.290067    2553 scope.go:117] "RemoveContainer" containerID="15c30995d3786c6a5651440fe3d41c7abf37b83d14e14224bed9fc7ac0738295"
Sep 16 14:48:58.290290 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:48:58.290274    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c30995d3786c6a5651440fe3d41c7abf37b83d14e14224bed9fc7ac0738295\": container with ID starting with 15c30995d3786c6a5651440fe3d41c7abf37b83d14e14224bed9fc7ac0738295 not found: ID does not exist" containerID="15c30995d3786c6a5651440fe3d41c7abf37b83d14e14224bed9fc7ac0738295"
Sep 16 14:48:58.290330 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:58.290293    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c30995d3786c6a5651440fe3d41c7abf37b83d14e14224bed9fc7ac0738295"} err="failed to get container status \"15c30995d3786c6a5651440fe3d41c7abf37b83d14e14224bed9fc7ac0738295\": rpc error: code = NotFound desc = could not find container \"15c30995d3786c6a5651440fe3d41c7abf37b83d14e14224bed9fc7ac0738295\": container with ID starting with 15c30995d3786c6a5651440fe3d41c7abf37b83d14e14224bed9fc7ac0738295 not found: ID does not exist"
Sep 16 14:48:58.290872 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:58.290833    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"]
Sep 16 14:48:58.294023 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:58.294002    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a70ddb-openshift-default-7c7778968b-qwk86"]
Sep 16 14:48:59.917720 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:48:59.917688    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" path="/var/lib/kubelet/pods/558668d8-3d9b-4626-b2e7-250c5b45ad1a/volumes"
Sep 16 14:49:10.047113 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:10.047079    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"]
Sep 16 14:49:10.047577 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:10.047338    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn" podUID="20069523-9db2-4a2f-a346-25dfddb2ce0e" containerName="istio-proxy" containerID="cri-o://7fb15a71ab69063022d9dcb2036f11e4e7d88f869dc68de7c2c0cc0e2246eb08" gracePeriod=30
Sep 16 14:49:10.051839 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:10.051815    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:49:10.056050 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:10.056031    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:49:15.304582 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.304556    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:49:15.334773 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.334705    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-credential-socket\") pod \"20069523-9db2-4a2f-a346-25dfddb2ce0e\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") "
Sep 16 14:49:15.334773 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.334747    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-data\") pod \"20069523-9db2-4a2f-a346-25dfddb2ce0e\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") "
Sep 16 14:49:15.334968 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.334783    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-workload-certs\") pod \"20069523-9db2-4a2f-a346-25dfddb2ce0e\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") "
Sep 16 14:49:15.334968 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.334827    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-envoy\") pod \"20069523-9db2-4a2f-a346-25dfddb2ce0e\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") "
Sep 16 14:49:15.334968 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.334852    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-workload-socket\") pod \"20069523-9db2-4a2f-a346-25dfddb2ce0e\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") "
Sep 16 14:49:15.334968 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.334916    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20069523-9db2-4a2f-a346-25dfddb2ce0e-istiod-ca-cert\") pod \"20069523-9db2-4a2f-a346-25dfddb2ce0e\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") "
Sep 16 14:49:15.335181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.334967    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "20069523-9db2-4a2f-a346-25dfddb2ce0e" (UID: "20069523-9db2-4a2f-a346-25dfddb2ce0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:49:15.335181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.334976    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2w6c\" (UniqueName: \"kubernetes.io/projected/20069523-9db2-4a2f-a346-25dfddb2ce0e-kube-api-access-h2w6c\") pod \"20069523-9db2-4a2f-a346-25dfddb2ce0e\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") "
Sep 16 14:49:15.335181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.335051    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-token\") pod \"20069523-9db2-4a2f-a346-25dfddb2ce0e\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") "
Sep 16 14:49:15.335181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.335069    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-data" (OuterVolumeSpecName: "istio-data") pod "20069523-9db2-4a2f-a346-25dfddb2ce0e" (UID: "20069523-9db2-4a2f-a346-25dfddb2ce0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:49:15.335181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.335093    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-podinfo\") pod \"20069523-9db2-4a2f-a346-25dfddb2ce0e\" (UID: \"20069523-9db2-4a2f-a346-25dfddb2ce0e\") "
Sep 16 14:49:15.335181 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.335118    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "20069523-9db2-4a2f-a346-25dfddb2ce0e" (UID: "20069523-9db2-4a2f-a346-25dfddb2ce0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:49:15.335507 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.335323    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20069523-9db2-4a2f-a346-25dfddb2ce0e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "20069523-9db2-4a2f-a346-25dfddb2ce0e" (UID: "20069523-9db2-4a2f-a346-25dfddb2ce0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:49:15.335566 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.335549    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-credential-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:15.335626 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.335560    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "20069523-9db2-4a2f-a346-25dfddb2ce0e" (UID: "20069523-9db2-4a2f-a346-25dfddb2ce0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:49:15.335626 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.335569    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-data\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:15.335626 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.335594    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-workload-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:15.335626 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.335609    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20069523-9db2-4a2f-a346-25dfddb2ce0e-istiod-ca-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:15.337273 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.337250    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "20069523-9db2-4a2f-a346-25dfddb2ce0e" (UID: "20069523-9db2-4a2f-a346-25dfddb2ce0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:49:15.337437 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.337416    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "20069523-9db2-4a2f-a346-25dfddb2ce0e" (UID: "20069523-9db2-4a2f-a346-25dfddb2ce0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:49:15.337488 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.337438    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-token" (OuterVolumeSpecName: "istio-token") pod "20069523-9db2-4a2f-a346-25dfddb2ce0e" (UID: "20069523-9db2-4a2f-a346-25dfddb2ce0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:49:15.337540 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.337526    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20069523-9db2-4a2f-a346-25dfddb2ce0e-kube-api-access-h2w6c" (OuterVolumeSpecName: "kube-api-access-h2w6c") pod "20069523-9db2-4a2f-a346-25dfddb2ce0e" (UID: "20069523-9db2-4a2f-a346-25dfddb2ce0e"). InnerVolumeSpecName "kube-api-access-h2w6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:49:15.363302 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.363276    2553 generic.go:356] "Generic (PLEG): container finished" podID="20069523-9db2-4a2f-a346-25dfddb2ce0e" containerID="7fb15a71ab69063022d9dcb2036f11e4e7d88f869dc68de7c2c0cc0e2246eb08" exitCode=0
Sep 16 14:49:15.363406 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.363306    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn" event={"ID":"20069523-9db2-4a2f-a346-25dfddb2ce0e","Type":"ContainerDied","Data":"7fb15a71ab69063022d9dcb2036f11e4e7d88f869dc68de7c2c0cc0e2246eb08"}
Sep 16 14:49:15.363406 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.363329    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"
Sep 16 14:49:15.363406 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.363383    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn" event={"ID":"20069523-9db2-4a2f-a346-25dfddb2ce0e","Type":"ContainerDied","Data":"26e33a17d07fd833290f8826b2c25bd9cdd162adaf2bf5f96339d4c56835292d"}
Sep 16 14:49:15.363406 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.363406    2553 scope.go:117] "RemoveContainer" containerID="7fb15a71ab69063022d9dcb2036f11e4e7d88f869dc68de7c2c0cc0e2246eb08"
Sep 16 14:49:15.372447 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.372432    2553 scope.go:117] "RemoveContainer" containerID="d9306a60f85a3af241b731038c39b8185ea1c207970ac0cc4a6ee3a98154abbe"
Sep 16 14:49:15.379983 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.379963    2553 scope.go:117] "RemoveContainer" containerID="7fb15a71ab69063022d9dcb2036f11e4e7d88f869dc68de7c2c0cc0e2246eb08"
Sep 16 14:49:15.380251 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:49:15.380232    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb15a71ab69063022d9dcb2036f11e4e7d88f869dc68de7c2c0cc0e2246eb08\": container with ID starting with 7fb15a71ab69063022d9dcb2036f11e4e7d88f869dc68de7c2c0cc0e2246eb08 not found: ID does not exist" containerID="7fb15a71ab69063022d9dcb2036f11e4e7d88f869dc68de7c2c0cc0e2246eb08"
Sep 16 14:49:15.380318 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.380258    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb15a71ab69063022d9dcb2036f11e4e7d88f869dc68de7c2c0cc0e2246eb08"} err="failed to get container status \"7fb15a71ab69063022d9dcb2036f11e4e7d88f869dc68de7c2c0cc0e2246eb08\": rpc error: code = NotFound desc = could not find container \"7fb15a71ab69063022d9dcb2036f11e4e7d88f869dc68de7c2c0cc0e2246eb08\": container with ID starting with 7fb15a71ab69063022d9dcb2036f11e4e7d88f869dc68de7c2c0cc0e2246eb08 not found: ID does not exist"
Sep 16 14:49:15.380318 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.380275    2553 scope.go:117] "RemoveContainer" containerID="d9306a60f85a3af241b731038c39b8185ea1c207970ac0cc4a6ee3a98154abbe"
Sep 16 14:49:15.380535 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:49:15.380518    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9306a60f85a3af241b731038c39b8185ea1c207970ac0cc4a6ee3a98154abbe\": container with ID starting with d9306a60f85a3af241b731038c39b8185ea1c207970ac0cc4a6ee3a98154abbe not found: ID does not exist" containerID="d9306a60f85a3af241b731038c39b8185ea1c207970ac0cc4a6ee3a98154abbe"
Sep 16 14:49:15.380581 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.380541    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9306a60f85a3af241b731038c39b8185ea1c207970ac0cc4a6ee3a98154abbe"} err="failed to get container status \"d9306a60f85a3af241b731038c39b8185ea1c207970ac0cc4a6ee3a98154abbe\": rpc error: code = NotFound desc = could not find container \"d9306a60f85a3af241b731038c39b8185ea1c207970ac0cc4a6ee3a98154abbe\": container with ID starting with d9306a60f85a3af241b731038c39b8185ea1c207970ac0cc4a6ee3a98154abbe not found: ID does not exist"
Sep 16 14:49:15.387631 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.387609    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"]
Sep 16 14:49:15.389617 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.389597    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3eaa3c-openshift-default-7d78b5bb96-jnrxn"]
Sep 16 14:49:15.436475 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.436455    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-envoy\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:15.436475 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.436474    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20069523-9db2-4a2f-a346-25dfddb2ce0e-workload-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:15.436592 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.436483    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2w6c\" (UniqueName: \"kubernetes.io/projected/20069523-9db2-4a2f-a346-25dfddb2ce0e-kube-api-access-h2w6c\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:15.436592 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.436493    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-token\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:15.436592 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.436501    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20069523-9db2-4a2f-a346-25dfddb2ce0e-istio-podinfo\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:15.917453 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:15.917419    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20069523-9db2-4a2f-a346-25dfddb2ce0e" path="/var/lib/kubelet/pods/20069523-9db2-4a2f-a346-25dfddb2ce0e/volumes"
Sep 16 14:49:20.827367 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:20.827330    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"]
Sep 16 14:49:20.827831 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:20.827588    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv" podUID="95f00493-e2a6-44af-987f-6d07769594dc" containerName="maas-api" containerID="cri-o://f9c43237a06600dbe11cc00399358112ac3d0974b1441a41aae5afaf75adb091" gracePeriod=30
Sep 16 14:49:20.859744 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:20.859710    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f88e1-29826165-d7jnd"]
Sep 16 14:49:20.861783 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:20.861756    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f88e1-29826165-d7jnd"]
Sep 16 14:49:21.079654 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.079599    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:49:21.186709 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.186673    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95f00493-e2a6-44af-987f-6d07769594dc-metrics-certs\") pod \"95f00493-e2a6-44af-987f-6d07769594dc\" (UID: \"95f00493-e2a6-44af-987f-6d07769594dc\") "
Sep 16 14:49:21.186709 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.186710    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95f00493-e2a6-44af-987f-6d07769594dc-maas-api-tls\") pod \"95f00493-e2a6-44af-987f-6d07769594dc\" (UID: \"95f00493-e2a6-44af-987f-6d07769594dc\") "
Sep 16 14:49:21.186953 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.186769    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f00493-e2a6-44af-987f-6d07769594dc-openshift-service-ca\") pod \"95f00493-e2a6-44af-987f-6d07769594dc\" (UID: \"95f00493-e2a6-44af-987f-6d07769594dc\") "
Sep 16 14:49:21.186953 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.186825    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxkfc\" (UniqueName: \"kubernetes.io/projected/95f00493-e2a6-44af-987f-6d07769594dc-kube-api-access-pxkfc\") pod \"95f00493-e2a6-44af-987f-6d07769594dc\" (UID: \"95f00493-e2a6-44af-987f-6d07769594dc\") "
Sep 16 14:49:21.187200 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.187175    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f00493-e2a6-44af-987f-6d07769594dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "95f00493-e2a6-44af-987f-6d07769594dc" (UID: "95f00493-e2a6-44af-987f-6d07769594dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:49:21.189068 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.189024    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95f00493-e2a6-44af-987f-6d07769594dc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "95f00493-e2a6-44af-987f-6d07769594dc" (UID: "95f00493-e2a6-44af-987f-6d07769594dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:49:21.189199 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.189084    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f00493-e2a6-44af-987f-6d07769594dc-kube-api-access-pxkfc" (OuterVolumeSpecName: "kube-api-access-pxkfc") pod "95f00493-e2a6-44af-987f-6d07769594dc" (UID: "95f00493-e2a6-44af-987f-6d07769594dc"). InnerVolumeSpecName "kube-api-access-pxkfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:49:21.189199 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.189169    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95f00493-e2a6-44af-987f-6d07769594dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "95f00493-e2a6-44af-987f-6d07769594dc" (UID: "95f00493-e2a6-44af-987f-6d07769594dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:49:21.288499 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.288461    2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f00493-e2a6-44af-987f-6d07769594dc-openshift-service-ca\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:21.288499 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.288493    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxkfc\" (UniqueName: \"kubernetes.io/projected/95f00493-e2a6-44af-987f-6d07769594dc-kube-api-access-pxkfc\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:21.288499 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.288503    2553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95f00493-e2a6-44af-987f-6d07769594dc-metrics-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:21.288743 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.288513    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95f00493-e2a6-44af-987f-6d07769594dc-maas-api-tls\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:21.395404 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.395311    2553 generic.go:356] "Generic (PLEG): container finished" podID="95f00493-e2a6-44af-987f-6d07769594dc" containerID="f9c43237a06600dbe11cc00399358112ac3d0974b1441a41aae5afaf75adb091" exitCode=0
Sep 16 14:49:21.395404 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.395368    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"
Sep 16 14:49:21.395404 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.395392    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv" event={"ID":"95f00493-e2a6-44af-987f-6d07769594dc","Type":"ContainerDied","Data":"f9c43237a06600dbe11cc00399358112ac3d0974b1441a41aae5afaf75adb091"}
Sep 16 14:49:21.395683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.395474    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv" event={"ID":"95f00493-e2a6-44af-987f-6d07769594dc","Type":"ContainerDied","Data":"606276da53c4ffd98fc1ce3a5a706ca1bd065c3bb2ae41a0af6d0923e08a780a"}
Sep 16 14:49:21.395683 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.395498    2553 scope.go:117] "RemoveContainer" containerID="f9c43237a06600dbe11cc00399358112ac3d0974b1441a41aae5afaf75adb091"
Sep 16 14:49:21.405204 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.405185    2553 scope.go:117] "RemoveContainer" containerID="f9c43237a06600dbe11cc00399358112ac3d0974b1441a41aae5afaf75adb091"
Sep 16 14:49:21.405496 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:49:21.405477    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c43237a06600dbe11cc00399358112ac3d0974b1441a41aae5afaf75adb091\": container with ID starting with f9c43237a06600dbe11cc00399358112ac3d0974b1441a41aae5afaf75adb091 not found: ID does not exist" containerID="f9c43237a06600dbe11cc00399358112ac3d0974b1441a41aae5afaf75adb091"
Sep 16 14:49:21.405562 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.405503    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c43237a06600dbe11cc00399358112ac3d0974b1441a41aae5afaf75adb091"} err="failed to get container status \"f9c43237a06600dbe11cc00399358112ac3d0974b1441a41aae5afaf75adb091\": rpc error: code = NotFound desc = could not find container \"f9c43237a06600dbe11cc00399358112ac3d0974b1441a41aae5afaf75adb091\": container with ID starting with f9c43237a06600dbe11cc00399358112ac3d0974b1441a41aae5afaf75adb091 not found: ID does not exist"
Sep 16 14:49:21.419871 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.419845    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"]
Sep 16 14:49:21.423904 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.423871    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f88e1-84bdf495dc-jhmjv"]
Sep 16 14:49:21.918796 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.918760    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e34253e-0fc4-42fe-9268-aacdcc6a9114" path="/var/lib/kubelet/pods/5e34253e-0fc4-42fe-9268-aacdcc6a9114/volumes"
Sep 16 14:49:21.919184 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:21.919141    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95f00493-e2a6-44af-987f-6d07769594dc" path="/var/lib/kubelet/pods/95f00493-e2a6-44af-987f-6d07769594dc/volumes"
Sep 16 14:49:46.380983 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:46.380943    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"]
Sep 16 14:49:46.381488 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:46.381174    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz" podUID="242319a3-bd8b-4739-afc9-d885f8d7d65d" containerName="istio-proxy" containerID="cri-o://bd08f2d16605e0aa13e0071bf9f5c73de63e92df1cf819150c6684acfeae6a0c" gracePeriod=30
Sep 16 14:49:51.560352 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.560110    2553 generic.go:356] "Generic (PLEG): container finished" podID="242319a3-bd8b-4739-afc9-d885f8d7d65d" containerID="bd08f2d16605e0aa13e0071bf9f5c73de63e92df1cf819150c6684acfeae6a0c" exitCode=0
Sep 16 14:49:51.560352 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.560211    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz" event={"ID":"242319a3-bd8b-4739-afc9-d885f8d7d65d","Type":"ContainerDied","Data":"bd08f2d16605e0aa13e0071bf9f5c73de63e92df1cf819150c6684acfeae6a0c"}
Sep 16 14:49:51.560352 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.560274    2553 scope.go:117] "RemoveContainer" containerID="d35afeea70ee758e77fb98608684650650fb9cae841cd3831579e6a7a6a62da2"
Sep 16 14:49:51.630076 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.630053    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:49:51.668564 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.668504    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-workload-certs\") pod \"242319a3-bd8b-4739-afc9-d885f8d7d65d\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") "
Sep 16 14:49:51.668564 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.668542    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-token\") pod \"242319a3-bd8b-4739-afc9-d885f8d7d65d\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") "
Sep 16 14:49:51.668770 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.668575    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-data\") pod \"242319a3-bd8b-4739-afc9-d885f8d7d65d\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") "
Sep 16 14:49:51.668770 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.668593    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-envoy\") pod \"242319a3-bd8b-4739-afc9-d885f8d7d65d\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") "
Sep 16 14:49:51.668770 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.668618    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62b6h\" (UniqueName: \"kubernetes.io/projected/242319a3-bd8b-4739-afc9-d885f8d7d65d-kube-api-access-62b6h\") pod \"242319a3-bd8b-4739-afc9-d885f8d7d65d\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") "
Sep 16 14:49:51.668770 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.668672    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242319a3-bd8b-4739-afc9-d885f8d7d65d-istiod-ca-cert\") pod \"242319a3-bd8b-4739-afc9-d885f8d7d65d\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") "
Sep 16 14:49:51.668770 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.668690    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-podinfo\") pod \"242319a3-bd8b-4739-afc9-d885f8d7d65d\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") "
Sep 16 14:49:51.668770 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.668730    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-workload-socket\") pod \"242319a3-bd8b-4739-afc9-d885f8d7d65d\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") "
Sep 16 14:49:51.668770 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.668757    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-credential-socket\") pod \"242319a3-bd8b-4739-afc9-d885f8d7d65d\" (UID: \"242319a3-bd8b-4739-afc9-d885f8d7d65d\") "
Sep 16 14:49:51.669160 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.668789    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "242319a3-bd8b-4739-afc9-d885f8d7d65d" (UID: "242319a3-bd8b-4739-afc9-d885f8d7d65d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:49:51.669160 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.668965    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-data" (OuterVolumeSpecName: "istio-data") pod "242319a3-bd8b-4739-afc9-d885f8d7d65d" (UID: "242319a3-bd8b-4739-afc9-d885f8d7d65d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:49:51.669160 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.669110    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-workload-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:51.669160 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.669128    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-data\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:51.669372 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.669197    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "242319a3-bd8b-4739-afc9-d885f8d7d65d" (UID: "242319a3-bd8b-4739-afc9-d885f8d7d65d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:49:51.669372 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.669207    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "242319a3-bd8b-4739-afc9-d885f8d7d65d" (UID: "242319a3-bd8b-4739-afc9-d885f8d7d65d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:49:51.669372 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.669342    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242319a3-bd8b-4739-afc9-d885f8d7d65d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "242319a3-bd8b-4739-afc9-d885f8d7d65d" (UID: "242319a3-bd8b-4739-afc9-d885f8d7d65d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:49:51.671150 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.671102    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-token" (OuterVolumeSpecName: "istio-token") pod "242319a3-bd8b-4739-afc9-d885f8d7d65d" (UID: "242319a3-bd8b-4739-afc9-d885f8d7d65d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:49:51.671251 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.671194    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "242319a3-bd8b-4739-afc9-d885f8d7d65d" (UID: "242319a3-bd8b-4739-afc9-d885f8d7d65d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:49:51.671251 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.671199    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242319a3-bd8b-4739-afc9-d885f8d7d65d-kube-api-access-62b6h" (OuterVolumeSpecName: "kube-api-access-62b6h") pod "242319a3-bd8b-4739-afc9-d885f8d7d65d" (UID: "242319a3-bd8b-4739-afc9-d885f8d7d65d"). InnerVolumeSpecName "kube-api-access-62b6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:49:51.671415 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.671389    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "242319a3-bd8b-4739-afc9-d885f8d7d65d" (UID: "242319a3-bd8b-4739-afc9-d885f8d7d65d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:49:51.770441 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.770419    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/242319a3-bd8b-4739-afc9-d885f8d7d65d-istiod-ca-cert\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:51.770441 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.770441    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-podinfo\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:51.770591 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.770450    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-workload-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:51.770591 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.770458    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-credential-socket\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:51.770591 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.770467    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-token\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:51.770591 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.770475    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/242319a3-bd8b-4739-afc9-d885f8d7d65d-istio-envoy\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:51.770591 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:51.770483    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62b6h\" (UniqueName: \"kubernetes.io/projected/242319a3-bd8b-4739-afc9-d885f8d7d65d-kube-api-access-62b6h\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:49:52.567117 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:52.567092    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"
Sep 16 14:49:52.567591 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:52.567084    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz" event={"ID":"242319a3-bd8b-4739-afc9-d885f8d7d65d","Type":"ContainerDied","Data":"bd8861283855dcf7624da99353f76cbead0235a46112ceda690e08e67d3d4842"}
Sep 16 14:49:52.567591 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:52.567231    2553 scope.go:117] "RemoveContainer" containerID="bd08f2d16605e0aa13e0071bf9f5c73de63e92df1cf819150c6684acfeae6a0c"
Sep 16 14:49:52.587676 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:52.587650    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"]
Sep 16 14:49:52.591153 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:52.591129    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-055ce3-openshift-default-5bc88d64-6dnpz"]
Sep 16 14:49:53.916993 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:49:53.916958    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="242319a3-bd8b-4739-afc9-d885f8d7d65d" path="/var/lib/kubelet/pods/242319a3-bd8b-4739-afc9-d885f8d7d65d/volumes"
Sep 16 14:51:10.321867 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:51:10.321832    2553 scope.go:117] "RemoveContainer" containerID="5b8c854eda7e5b6c54fe05e6e3adf077fe5a61259c399a695ae5e37ddfb51d4b"
Sep 16 14:52:25.309516 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:25.309475    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"]
Sep 16 14:52:25.312135 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:25.309796    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p" podUID="5df2edc4-d6c5-4c5c-95b7-399f90a35586" containerName="manager" containerID="cri-o://9556fcca9065bc06eff744bf265ed3f80f3fd437b3f4ea134c23dedcd6f7652c" gracePeriod=10
Sep 16 14:52:25.972293 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:25.972272    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"
Sep 16 14:52:26.086327 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:26.086245    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5df2edc4-d6c5-4c5c-95b7-399f90a35586-extensions-socket-volume\") pod \"5df2edc4-d6c5-4c5c-95b7-399f90a35586\" (UID: \"5df2edc4-d6c5-4c5c-95b7-399f90a35586\") "
Sep 16 14:52:26.086327 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:26.086315    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl9jl\" (UniqueName: \"kubernetes.io/projected/5df2edc4-d6c5-4c5c-95b7-399f90a35586-kube-api-access-cl9jl\") pod \"5df2edc4-d6c5-4c5c-95b7-399f90a35586\" (UID: \"5df2edc4-d6c5-4c5c-95b7-399f90a35586\") "
Sep 16 14:52:26.086697 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:26.086665    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df2edc4-d6c5-4c5c-95b7-399f90a35586-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5df2edc4-d6c5-4c5c-95b7-399f90a35586" (UID: "5df2edc4-d6c5-4c5c-95b7-399f90a35586"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:52:26.088361 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:26.088339    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df2edc4-d6c5-4c5c-95b7-399f90a35586-kube-api-access-cl9jl" (OuterVolumeSpecName: "kube-api-access-cl9jl") pod "5df2edc4-d6c5-4c5c-95b7-399f90a35586" (UID: "5df2edc4-d6c5-4c5c-95b7-399f90a35586"). InnerVolumeSpecName "kube-api-access-cl9jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:52:26.187271 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:26.187241    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl9jl\" (UniqueName: \"kubernetes.io/projected/5df2edc4-d6c5-4c5c-95b7-399f90a35586-kube-api-access-cl9jl\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:52:26.187271 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:26.187266    2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5df2edc4-d6c5-4c5c-95b7-399f90a35586-extensions-socket-volume\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:52:26.337475 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:26.337380    2553 generic.go:356] "Generic (PLEG): container finished" podID="5df2edc4-d6c5-4c5c-95b7-399f90a35586" containerID="9556fcca9065bc06eff744bf265ed3f80f3fd437b3f4ea134c23dedcd6f7652c" exitCode=0
Sep 16 14:52:26.337475 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:26.337426    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"
Sep 16 14:52:26.337475 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:26.337431    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p" event={"ID":"5df2edc4-d6c5-4c5c-95b7-399f90a35586","Type":"ContainerDied","Data":"9556fcca9065bc06eff744bf265ed3f80f3fd437b3f4ea134c23dedcd6f7652c"}
Sep 16 14:52:26.338074 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:26.337501    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p" event={"ID":"5df2edc4-d6c5-4c5c-95b7-399f90a35586","Type":"ContainerDied","Data":"4dac47b79eaee8e7559b7daa176aad505be7642621dd3318d10072fcf970f30c"}
Sep 16 14:52:26.338074 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:26.337534    2553 scope.go:117] "RemoveContainer" containerID="9556fcca9065bc06eff744bf265ed3f80f3fd437b3f4ea134c23dedcd6f7652c"
Sep 16 14:52:26.346528 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:26.346510    2553 scope.go:117] "RemoveContainer" containerID="9556fcca9065bc06eff744bf265ed3f80f3fd437b3f4ea134c23dedcd6f7652c"
Sep 16 14:52:26.346784 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:52:26.346767    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9556fcca9065bc06eff744bf265ed3f80f3fd437b3f4ea134c23dedcd6f7652c\": container with ID starting with 9556fcca9065bc06eff744bf265ed3f80f3fd437b3f4ea134c23dedcd6f7652c not found: ID does not exist" containerID="9556fcca9065bc06eff744bf265ed3f80f3fd437b3f4ea134c23dedcd6f7652c"
Sep 16 14:52:26.346838 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:26.346793    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9556fcca9065bc06eff744bf265ed3f80f3fd437b3f4ea134c23dedcd6f7652c"} err="failed to get container status \"9556fcca9065bc06eff744bf265ed3f80f3fd437b3f4ea134c23dedcd6f7652c\": rpc error: code = NotFound desc = could not find container \"9556fcca9065bc06eff744bf265ed3f80f3fd437b3f4ea134c23dedcd6f7652c\": container with ID starting with 9556fcca9065bc06eff744bf265ed3f80f3fd437b3f4ea134c23dedcd6f7652c not found: ID does not exist"
Sep 16 14:52:26.364647 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:26.364620    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"]
Sep 16 14:52:26.369469 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:26.369446    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-nr88p"]
Sep 16 14:52:27.916861 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:52:27.916827    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5df2edc4-d6c5-4c5c-95b7-399f90a35586" path="/var/lib/kubelet/pods/5df2edc4-d6c5-4c5c-95b7-399f90a35586/volumes"
Sep 16 14:53:26.013652 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.013623    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkgzm"]
Sep 16 14:53:26.014086 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014034    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95f00493-e2a6-44af-987f-6d07769594dc" containerName="maas-api"
Sep 16 14:53:26.014086 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014048    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="95f00493-e2a6-44af-987f-6d07769594dc" containerName="maas-api"
Sep 16 14:53:26.014086 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014059    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53fe12fd-e9ec-4642-8dde-3618e994e37b" containerName="maas-api"
Sep 16 14:53:26.014086 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014064    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="53fe12fd-e9ec-4642-8dde-3618e994e37b" containerName="maas-api"
Sep 16 14:53:26.014086 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014079    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerName="istio-proxy"
Sep 16 14:53:26.014086 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014085    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014091    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20069523-9db2-4a2f-a346-25dfddb2ce0e" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014097    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="20069523-9db2-4a2f-a346-25dfddb2ce0e" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014102    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014107    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014112    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242319a3-bd8b-4739-afc9-d885f8d7d65d" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014117    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="242319a3-bd8b-4739-afc9-d885f8d7d65d" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014125    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242319a3-bd8b-4739-afc9-d885f8d7d65d" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014130    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="242319a3-bd8b-4739-afc9-d885f8d7d65d" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014139    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfbf848f-1191-46e4-b74d-84b3a9c9e247" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014144    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfbf848f-1191-46e4-b74d-84b3a9c9e247" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014151    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5df2edc4-d6c5-4c5c-95b7-399f90a35586" containerName="manager"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014156    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5df2edc4-d6c5-4c5c-95b7-399f90a35586" containerName="manager"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014165    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfbf848f-1191-46e4-b74d-84b3a9c9e247" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014170    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfbf848f-1191-46e4-b74d-84b3a9c9e247" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014180    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20069523-9db2-4a2f-a346-25dfddb2ce0e" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014185    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="20069523-9db2-4a2f-a346-25dfddb2ce0e" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014237    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014244    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfbf848f-1191-46e4-b74d-84b3a9c9e247" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014250    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="95f00493-e2a6-44af-987f-6d07769594dc" containerName="maas-api"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014255    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="242319a3-bd8b-4739-afc9-d885f8d7d65d" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014261    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="20069523-9db2-4a2f-a346-25dfddb2ce0e" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014268    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="20069523-9db2-4a2f-a346-25dfddb2ce0e" containerName="istio-proxy"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014272    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5df2edc4-d6c5-4c5c-95b7-399f90a35586" containerName="manager"
Sep 16 14:53:26.014295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.014280    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="53fe12fd-e9ec-4642-8dde-3618e994e37b" containerName="maas-api"
Sep 16 14:53:26.017542 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.017527    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkgzm"
Sep 16 14:53:26.020096 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.020075    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wt2h7\""
Sep 16 14:53:26.028529 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.028506    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkgzm"]
Sep 16 14:53:26.112734 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.112707    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6da1bce6-0a84-4740-a1ab-2d4452a9e2a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mkgzm\" (UID: \"6da1bce6-0a84-4740-a1ab-2d4452a9e2a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkgzm"
Sep 16 14:53:26.112905 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.112740    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2xm8\" (UniqueName: \"kubernetes.io/projected/6da1bce6-0a84-4740-a1ab-2d4452a9e2a1-kube-api-access-z2xm8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mkgzm\" (UID: \"6da1bce6-0a84-4740-a1ab-2d4452a9e2a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkgzm"
Sep 16 14:53:26.213958 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.213925    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6da1bce6-0a84-4740-a1ab-2d4452a9e2a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mkgzm\" (UID: \"6da1bce6-0a84-4740-a1ab-2d4452a9e2a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkgzm"
Sep 16 14:53:26.213958 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.213960    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2xm8\" (UniqueName: \"kubernetes.io/projected/6da1bce6-0a84-4740-a1ab-2d4452a9e2a1-kube-api-access-z2xm8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mkgzm\" (UID: \"6da1bce6-0a84-4740-a1ab-2d4452a9e2a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkgzm"
Sep 16 14:53:26.214311 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.214292    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6da1bce6-0a84-4740-a1ab-2d4452a9e2a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mkgzm\" (UID: \"6da1bce6-0a84-4740-a1ab-2d4452a9e2a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkgzm"
Sep 16 14:53:26.222986 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.222961    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2xm8\" (UniqueName: \"kubernetes.io/projected/6da1bce6-0a84-4740-a1ab-2d4452a9e2a1-kube-api-access-z2xm8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mkgzm\" (UID: \"6da1bce6-0a84-4740-a1ab-2d4452a9e2a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkgzm"
Sep 16 14:53:26.328380 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.328314    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkgzm"
Sep 16 14:53:26.633593 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:53:26.633562    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da1bce6_0a84_4740_a1ab_2d4452a9e2a1.slice/crio-2b3b38808024fcb601192a868d18b7439de66f301340204fbe7c00ccfe425435 WatchSource:0}: Error finding container 2b3b38808024fcb601192a868d18b7439de66f301340204fbe7c00ccfe425435: Status 404 returned error can't find the container with id 2b3b38808024fcb601192a868d18b7439de66f301340204fbe7c00ccfe425435
Sep 16 14:53:26.634993 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:26.634966    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkgzm"]
Sep 16 14:53:27.643732 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:27.643696    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkgzm" event={"ID":"6da1bce6-0a84-4740-a1ab-2d4452a9e2a1","Type":"ContainerStarted","Data":"432ed733904714edad0c3b054edf7bc70d4ea5a9ddb8e65c6d4742d5b842c361"}
Sep 16 14:53:27.644224 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:27.643777    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkgzm"
Sep 16 14:53:27.644224 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:27.643795    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkgzm" event={"ID":"6da1bce6-0a84-4740-a1ab-2d4452a9e2a1","Type":"ContainerStarted","Data":"2b3b38808024fcb601192a868d18b7439de66f301340204fbe7c00ccfe425435"}
Sep 16 14:53:27.693689 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:27.693642    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkgzm" podStartSLOduration=2.693626454 podStartE2EDuration="2.693626454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:53:27.690427243 +0000 UTC m=+2358.237169167" watchObservedRunningTime="2026-09-16 14:53:27.693626454 +0000 UTC m=+2358.240368385"
Sep 16 14:53:38.651312 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:53:38.651282    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkgzm"
Sep 16 14:54:10.088312 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:10.088282    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:54:10.094619 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:10.094594    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:54:17.623944 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.623908    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"]
Sep 16 14:54:17.624525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.624355    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="558668d8-3d9b-4626-b2e7-250c5b45ad1a" containerName="istio-proxy"
Sep 16 14:54:17.624525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.624368    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfbf848f-1191-46e4-b74d-84b3a9c9e247" containerName="istio-proxy"
Sep 16 14:54:17.624525 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.624375    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="242319a3-bd8b-4739-afc9-d885f8d7d65d" containerName="istio-proxy"
Sep 16 14:54:17.627167 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.627148    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.630163 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.630142    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f0c0912d-kserve-self-signed-certs\""
Sep 16 14:54:17.638247 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.638227    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"]
Sep 16 14:54:17.682840 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.682806    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-model-cache\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.683014 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.682846    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-tls-certs\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.683014 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.682884    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-dshm\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.683014 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.682956    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-home\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.683014 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.682978    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-tmp-dir\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.683014 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.683009    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wdgm\" (UniqueName: \"kubernetes.io/projected/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-kube-api-access-2wdgm\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.784288 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.784253    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-model-cache\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.784288 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.784296    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-tls-certs\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.784516 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.784325    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-dshm\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.784577 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.784517    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-home\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.784638 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.784574    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-tmp-dir\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.784696 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.784635    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wdgm\" (UniqueName: \"kubernetes.io/projected/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-kube-api-access-2wdgm\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.784756 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.784712    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-model-cache\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.784872 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.784848    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-home\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.785007 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.784982    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-tmp-dir\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.786717 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.786696    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-dshm\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.787372 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.787350    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-tls-certs\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.795187 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.795165    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wdgm\" (UniqueName: \"kubernetes.io/projected/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-kube-api-access-2wdgm\") pod \"e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") " pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:17.937614 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:17.937525    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:18.216927 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:18.216883    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"]
Sep 16 14:54:18.219268 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:54:18.219245    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08ddcb2a_43cb_4efc_ad01_0cd937b850e3.slice/crio-360f1bba30147779492faf250a89825163fc93eced7c3f1f88b73d4a4c0d9dfa WatchSource:0}: Error finding container 360f1bba30147779492faf250a89825163fc93eced7c3f1f88b73d4a4c0d9dfa: Status 404 returned error can't find the container with id 360f1bba30147779492faf250a89825163fc93eced7c3f1f88b73d4a4c0d9dfa
Sep 16 14:54:18.910292 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:18.910256    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw" event={"ID":"08ddcb2a-43cb-4efc-ad01-0cd937b850e3","Type":"ContainerStarted","Data":"9408ae7e8b61fefdef0cd23686dee89335740f66e9028667e2bef58702dae218"}
Sep 16 14:54:18.910292 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:18.910295    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw" event={"ID":"08ddcb2a-43cb-4efc-ad01-0cd937b850e3","Type":"ContainerStarted","Data":"360f1bba30147779492faf250a89825163fc93eced7c3f1f88b73d4a4c0d9dfa"}
Sep 16 14:54:18.935460 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:18.935409    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw" podStartSLOduration=1.935393194 podStartE2EDuration="1.935393194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:54:18.932473421 +0000 UTC m=+2409.479215353" watchObservedRunningTime="2026-09-16 14:54:18.935393194 +0000 UTC m=+2409.482135125"
Sep 16 14:54:27.938414 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:27.938385    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:27.938831 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:27.938440    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:27.950673 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:27.950645    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:27.977826 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:27.977801    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:54:53.022432 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:54:53.022402    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"]
Sep 16 14:55:02.007706 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.007670    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"]
Sep 16 14:55:02.011485 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.011464    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:02.015752 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.015730    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 14:55:02.015867 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.015770    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qjgdq\""
Sep 16 14:55:02.015867 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.015789    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 14:55:02.027333 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.027312    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"]
Sep 16 14:55:02.099886 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.099855    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7436723-1660-4b8b-9809-7d37af51420e-maas-api-tls\") pod \"maas-api-8488dbc8b7-c654w\" (UID: \"d7436723-1660-4b8b-9809-7d37af51420e\") " pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:02.100082 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.099908    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7436723-1660-4b8b-9809-7d37af51420e-openshift-service-ca\") pod \"maas-api-8488dbc8b7-c654w\" (UID: \"d7436723-1660-4b8b-9809-7d37af51420e\") " pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:02.100082 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.099975    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7436723-1660-4b8b-9809-7d37af51420e-metrics-certs\") pod \"maas-api-8488dbc8b7-c654w\" (UID: \"d7436723-1660-4b8b-9809-7d37af51420e\") " pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:02.100206 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.100102    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4bdq\" (UniqueName: \"kubernetes.io/projected/d7436723-1660-4b8b-9809-7d37af51420e-kube-api-access-v4bdq\") pod \"maas-api-8488dbc8b7-c654w\" (UID: \"d7436723-1660-4b8b-9809-7d37af51420e\") " pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:02.200745 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.200708    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7436723-1660-4b8b-9809-7d37af51420e-maas-api-tls\") pod \"maas-api-8488dbc8b7-c654w\" (UID: \"d7436723-1660-4b8b-9809-7d37af51420e\") " pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:02.200961 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.200754    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7436723-1660-4b8b-9809-7d37af51420e-openshift-service-ca\") pod \"maas-api-8488dbc8b7-c654w\" (UID: \"d7436723-1660-4b8b-9809-7d37af51420e\") " pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:02.200961 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.200781    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7436723-1660-4b8b-9809-7d37af51420e-metrics-certs\") pod \"maas-api-8488dbc8b7-c654w\" (UID: \"d7436723-1660-4b8b-9809-7d37af51420e\") " pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:02.200961 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.200862    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4bdq\" (UniqueName: \"kubernetes.io/projected/d7436723-1660-4b8b-9809-7d37af51420e-kube-api-access-v4bdq\") pod \"maas-api-8488dbc8b7-c654w\" (UID: \"d7436723-1660-4b8b-9809-7d37af51420e\") " pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:02.201490 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.201465    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7436723-1660-4b8b-9809-7d37af51420e-openshift-service-ca\") pod \"maas-api-8488dbc8b7-c654w\" (UID: \"d7436723-1660-4b8b-9809-7d37af51420e\") " pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:02.203381 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.203357    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7436723-1660-4b8b-9809-7d37af51420e-maas-api-tls\") pod \"maas-api-8488dbc8b7-c654w\" (UID: \"d7436723-1660-4b8b-9809-7d37af51420e\") " pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:02.203475 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.203357    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7436723-1660-4b8b-9809-7d37af51420e-metrics-certs\") pod \"maas-api-8488dbc8b7-c654w\" (UID: \"d7436723-1660-4b8b-9809-7d37af51420e\") " pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:02.209150 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.209133    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4bdq\" (UniqueName: \"kubernetes.io/projected/d7436723-1660-4b8b-9809-7d37af51420e-kube-api-access-v4bdq\") pod \"maas-api-8488dbc8b7-c654w\" (UID: \"d7436723-1660-4b8b-9809-7d37af51420e\") " pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:02.322363 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.322281    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:02.612521 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.612489    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"]
Sep 16 14:55:02.614161 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:55:02.614125    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7436723_1660_4b8b_9809_7d37af51420e.slice/crio-3fe6c1576592787a220df7cd284038fe4b17a488a385c9550e9a4079988c5ef4 WatchSource:0}: Error finding container 3fe6c1576592787a220df7cd284038fe4b17a488a385c9550e9a4079988c5ef4: Status 404 returned error can't find the container with id 3fe6c1576592787a220df7cd284038fe4b17a488a385c9550e9a4079988c5ef4
Sep 16 14:55:02.616024 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:02.615999    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:55:03.137028 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:03.136991    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w" event={"ID":"d7436723-1660-4b8b-9809-7d37af51420e","Type":"ContainerStarted","Data":"3fe6c1576592787a220df7cd284038fe4b17a488a385c9550e9a4079988c5ef4"}
Sep 16 14:55:04.143566 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:04.143527    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w" event={"ID":"d7436723-1660-4b8b-9809-7d37af51420e","Type":"ContainerStarted","Data":"eea1a32daeeb0ee91de3d35aac1bfb9270de67e89621130aa3c6f7f2e739954c"}
Sep 16 14:55:04.144052 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:04.143600    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:04.166064 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:04.166022    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w" podStartSLOduration=2.18160256 podStartE2EDuration="3.166007517s" totalImagesPullingTime="984.404957ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:55:04.163937516 +0000 UTC m=+2454.710679447" watchObservedRunningTime="2026-09-16 14:55:04.166007517 +0000 UTC m=+2454.712749448"
Sep 16 14:55:08.042338 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.042297    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw" podUID="08ddcb2a-43cb-4efc-ad01-0cd937b850e3" containerName="main" containerID="cri-o://9408ae7e8b61fefdef0cd23686dee89335740f66e9028667e2bef58702dae218" gracePeriod=45
Sep 16 14:55:08.165662 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.165635    2553 generic.go:356] "Generic (PLEG): container finished" podID="08ddcb2a-43cb-4efc-ad01-0cd937b850e3" containerID="9408ae7e8b61fefdef0cd23686dee89335740f66e9028667e2bef58702dae218" exitCode=0
Sep 16 14:55:08.165804 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.165704    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw" event={"ID":"08ddcb2a-43cb-4efc-ad01-0cd937b850e3","Type":"ContainerDied","Data":"9408ae7e8b61fefdef0cd23686dee89335740f66e9028667e2bef58702dae218"}
Sep 16 14:55:08.302136 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.302076    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:55:08.365083 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.365054    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-tmp-dir\") pod \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") "
Sep 16 14:55:08.365252 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.365089    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-dshm\") pod \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") "
Sep 16 14:55:08.365252 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.365128    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-home\") pod \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") "
Sep 16 14:55:08.365252 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.365198    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-model-cache\") pod \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") "
Sep 16 14:55:08.365252 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.365241    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wdgm\" (UniqueName: \"kubernetes.io/projected/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-kube-api-access-2wdgm\") pod \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") "
Sep 16 14:55:08.365481 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.365268    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-tls-certs\") pod \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\" (UID: \"08ddcb2a-43cb-4efc-ad01-0cd937b850e3\") "
Sep 16 14:55:08.365481 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.365432    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08ddcb2a-43cb-4efc-ad01-0cd937b850e3" (UID: "08ddcb2a-43cb-4efc-ad01-0cd937b850e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:55:08.365597 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.365477    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "08ddcb2a-43cb-4efc-ad01-0cd937b850e3" (UID: "08ddcb2a-43cb-4efc-ad01-0cd937b850e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:55:08.365597 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.365589    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-model-cache\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:55:08.365724 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.365605    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-tmp-dir\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:55:08.365724 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.365694    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-home" (OuterVolumeSpecName: "home") pod "08ddcb2a-43cb-4efc-ad01-0cd937b850e3" (UID: "08ddcb2a-43cb-4efc-ad01-0cd937b850e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:55:08.367399 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.367372    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-dshm" (OuterVolumeSpecName: "dshm") pod "08ddcb2a-43cb-4efc-ad01-0cd937b850e3" (UID: "08ddcb2a-43cb-4efc-ad01-0cd937b850e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:55:08.367498 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.367442    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-kube-api-access-2wdgm" (OuterVolumeSpecName: "kube-api-access-2wdgm") pod "08ddcb2a-43cb-4efc-ad01-0cd937b850e3" (UID: "08ddcb2a-43cb-4efc-ad01-0cd937b850e3"). InnerVolumeSpecName "kube-api-access-2wdgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:55:08.367831 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.367815    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08ddcb2a-43cb-4efc-ad01-0cd937b850e3" (UID: "08ddcb2a-43cb-4efc-ad01-0cd937b850e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:55:08.466301 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.466268    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wdgm\" (UniqueName: \"kubernetes.io/projected/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-kube-api-access-2wdgm\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:55:08.466301 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.466292    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-tls-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:55:08.466301 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.466302    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-dshm\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:55:08.466555 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:08.466310    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ddcb2a-43cb-4efc-ad01-0cd937b850e3-home\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:55:09.172106 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:09.172073    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"
Sep 16 14:55:09.172106 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:09.172090    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw" event={"ID":"08ddcb2a-43cb-4efc-ad01-0cd937b850e3","Type":"ContainerDied","Data":"360f1bba30147779492faf250a89825163fc93eced7c3f1f88b73d4a4c0d9dfa"}
Sep 16 14:55:09.172612 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:09.172147    2553 scope.go:117] "RemoveContainer" containerID="9408ae7e8b61fefdef0cd23686dee89335740f66e9028667e2bef58702dae218"
Sep 16 14:55:09.196850 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:09.196821    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"]
Sep 16 14:55:09.200368 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:09.200346    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f0c0912d-kserve-596769c647-mclhw"]
Sep 16 14:55:09.917453 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:09.917422    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08ddcb2a-43cb-4efc-ad01-0cd937b850e3" path="/var/lib/kubelet/pods/08ddcb2a-43cb-4efc-ad01-0cd937b850e3/volumes"
Sep 16 14:55:10.154176 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.154150    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:10.614019 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.613989    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"]
Sep 16 14:55:10.614396 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.614383    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08ddcb2a-43cb-4efc-ad01-0cd937b850e3" containerName="main"
Sep 16 14:55:10.614396 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.614397    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="08ddcb2a-43cb-4efc-ad01-0cd937b850e3" containerName="main"
Sep 16 14:55:10.614479 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.614475    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="08ddcb2a-43cb-4efc-ad01-0cd937b850e3" containerName="main"
Sep 16 14:55:10.618991 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.618971    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"
Sep 16 14:55:10.642593 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.642560    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"]
Sep 16 14:55:10.686605 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.686569    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6bada29-1967-4510-9bd6-ecbb8987c471-maas-api-tls\") pod \"maas-api-6976c56b87-m7tsw\" (UID: \"e6bada29-1967-4510-9bd6-ecbb8987c471\") " pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"
Sep 16 14:55:10.686745 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.686606    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxlgd\" (UniqueName: \"kubernetes.io/projected/e6bada29-1967-4510-9bd6-ecbb8987c471-kube-api-access-hxlgd\") pod \"maas-api-6976c56b87-m7tsw\" (UID: \"e6bada29-1967-4510-9bd6-ecbb8987c471\") " pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"
Sep 16 14:55:10.686745 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.686683    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6bada29-1967-4510-9bd6-ecbb8987c471-metrics-certs\") pod \"maas-api-6976c56b87-m7tsw\" (UID: \"e6bada29-1967-4510-9bd6-ecbb8987c471\") " pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"
Sep 16 14:55:10.686745 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.686728    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6bada29-1967-4510-9bd6-ecbb8987c471-openshift-service-ca\") pod \"maas-api-6976c56b87-m7tsw\" (UID: \"e6bada29-1967-4510-9bd6-ecbb8987c471\") " pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"
Sep 16 14:55:10.787905 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.787871    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6bada29-1967-4510-9bd6-ecbb8987c471-maas-api-tls\") pod \"maas-api-6976c56b87-m7tsw\" (UID: \"e6bada29-1967-4510-9bd6-ecbb8987c471\") " pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"
Sep 16 14:55:10.788058 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.787952    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxlgd\" (UniqueName: \"kubernetes.io/projected/e6bada29-1967-4510-9bd6-ecbb8987c471-kube-api-access-hxlgd\") pod \"maas-api-6976c56b87-m7tsw\" (UID: \"e6bada29-1967-4510-9bd6-ecbb8987c471\") " pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"
Sep 16 14:55:10.788058 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.788017    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6bada29-1967-4510-9bd6-ecbb8987c471-metrics-certs\") pod \"maas-api-6976c56b87-m7tsw\" (UID: \"e6bada29-1967-4510-9bd6-ecbb8987c471\") " pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"
Sep 16 14:55:10.788171 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.788071    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6bada29-1967-4510-9bd6-ecbb8987c471-openshift-service-ca\") pod \"maas-api-6976c56b87-m7tsw\" (UID: \"e6bada29-1967-4510-9bd6-ecbb8987c471\") " pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"
Sep 16 14:55:10.788720 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.788699    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6bada29-1967-4510-9bd6-ecbb8987c471-openshift-service-ca\") pod \"maas-api-6976c56b87-m7tsw\" (UID: \"e6bada29-1967-4510-9bd6-ecbb8987c471\") " pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"
Sep 16 14:55:10.790424 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.790405    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6bada29-1967-4510-9bd6-ecbb8987c471-metrics-certs\") pod \"maas-api-6976c56b87-m7tsw\" (UID: \"e6bada29-1967-4510-9bd6-ecbb8987c471\") " pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"
Sep 16 14:55:10.790515 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.790426    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6bada29-1967-4510-9bd6-ecbb8987c471-maas-api-tls\") pod \"maas-api-6976c56b87-m7tsw\" (UID: \"e6bada29-1967-4510-9bd6-ecbb8987c471\") " pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"
Sep 16 14:55:10.798689 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.798666    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxlgd\" (UniqueName: \"kubernetes.io/projected/e6bada29-1967-4510-9bd6-ecbb8987c471-kube-api-access-hxlgd\") pod \"maas-api-6976c56b87-m7tsw\" (UID: \"e6bada29-1967-4510-9bd6-ecbb8987c471\") " pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"
Sep 16 14:55:10.928948 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:10.928843    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"
Sep 16 14:55:11.223295 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:11.223265    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"]
Sep 16 14:55:11.226405 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:55:11.226380    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6bada29_1967_4510_9bd6_ecbb8987c471.slice/crio-a4ef346bbf78d3e36a9a8436f3884ef99eeec8aa20ef01fb463e2e972c39d354 WatchSource:0}: Error finding container a4ef346bbf78d3e36a9a8436f3884ef99eeec8aa20ef01fb463e2e972c39d354: Status 404 returned error can't find the container with id a4ef346bbf78d3e36a9a8436f3884ef99eeec8aa20ef01fb463e2e972c39d354
Sep 16 14:55:12.191067 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:12.191033    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw" event={"ID":"e6bada29-1967-4510-9bd6-ecbb8987c471","Type":"ContainerStarted","Data":"86edcca8871a850f136e89a6694322f9fe8efd2f8ba931d0f98f573dd1329a38"}
Sep 16 14:55:12.191476 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:12.191088    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw" event={"ID":"e6bada29-1967-4510-9bd6-ecbb8987c471","Type":"ContainerStarted","Data":"a4ef346bbf78d3e36a9a8436f3884ef99eeec8aa20ef01fb463e2e972c39d354"}
Sep 16 14:55:12.191476 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:12.191110    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"
Sep 16 14:55:12.217068 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:12.217021    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw" podStartSLOduration=1.810986773 podStartE2EDuration="2.217003272s" totalImagesPullingTime="406.016499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:55:12.215417453 +0000 UTC m=+2462.762159384" watchObservedRunningTime="2026-09-16 14:55:12.217003272 +0000 UTC m=+2462.763745204"
Sep 16 14:55:18.201775 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:18.201745    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6976c56b87-m7tsw"
Sep 16 14:55:18.256594 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:18.256555    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"]
Sep 16 14:55:18.256785 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:18.256759    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w" podUID="d7436723-1660-4b8b-9809-7d37af51420e" containerName="maas-api" containerID="cri-o://eea1a32daeeb0ee91de3d35aac1bfb9270de67e89621130aa3c6f7f2e739954c" gracePeriod=30
Sep 16 14:55:19.508478 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:19.508453    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:19.674065 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:19.673972    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4bdq\" (UniqueName: \"kubernetes.io/projected/d7436723-1660-4b8b-9809-7d37af51420e-kube-api-access-v4bdq\") pod \"d7436723-1660-4b8b-9809-7d37af51420e\" (UID: \"d7436723-1660-4b8b-9809-7d37af51420e\") "
Sep 16 14:55:19.674065 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:19.674056    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7436723-1660-4b8b-9809-7d37af51420e-metrics-certs\") pod \"d7436723-1660-4b8b-9809-7d37af51420e\" (UID: \"d7436723-1660-4b8b-9809-7d37af51420e\") "
Sep 16 14:55:19.674312 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:19.674088    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7436723-1660-4b8b-9809-7d37af51420e-maas-api-tls\") pod \"d7436723-1660-4b8b-9809-7d37af51420e\" (UID: \"d7436723-1660-4b8b-9809-7d37af51420e\") "
Sep 16 14:55:19.674312 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:19.674107    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7436723-1660-4b8b-9809-7d37af51420e-openshift-service-ca\") pod \"d7436723-1660-4b8b-9809-7d37af51420e\" (UID: \"d7436723-1660-4b8b-9809-7d37af51420e\") "
Sep 16 14:55:19.674530 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:19.674498    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7436723-1660-4b8b-9809-7d37af51420e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d7436723-1660-4b8b-9809-7d37af51420e" (UID: "d7436723-1660-4b8b-9809-7d37af51420e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:55:19.676282 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:19.676252    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7436723-1660-4b8b-9809-7d37af51420e-kube-api-access-v4bdq" (OuterVolumeSpecName: "kube-api-access-v4bdq") pod "d7436723-1660-4b8b-9809-7d37af51420e" (UID: "d7436723-1660-4b8b-9809-7d37af51420e"). InnerVolumeSpecName "kube-api-access-v4bdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:55:19.676644 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:19.676618    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7436723-1660-4b8b-9809-7d37af51420e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d7436723-1660-4b8b-9809-7d37af51420e" (UID: "d7436723-1660-4b8b-9809-7d37af51420e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:55:19.676733 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:19.676639    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7436723-1660-4b8b-9809-7d37af51420e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d7436723-1660-4b8b-9809-7d37af51420e" (UID: "d7436723-1660-4b8b-9809-7d37af51420e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:55:19.775367 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:19.775334    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4bdq\" (UniqueName: \"kubernetes.io/projected/d7436723-1660-4b8b-9809-7d37af51420e-kube-api-access-v4bdq\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:55:19.775367 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:19.775363    2553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7436723-1660-4b8b-9809-7d37af51420e-metrics-certs\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:55:19.775367 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:19.775372    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7436723-1660-4b8b-9809-7d37af51420e-maas-api-tls\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:55:19.775595 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:19.775380    2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7436723-1660-4b8b-9809-7d37af51420e-openshift-service-ca\") on node \"ip-10-0-128-50.ec2.internal\" DevicePath \"\""
Sep 16 14:55:20.234826 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:20.234788    2553 generic.go:356] "Generic (PLEG): container finished" podID="d7436723-1660-4b8b-9809-7d37af51420e" containerID="eea1a32daeeb0ee91de3d35aac1bfb9270de67e89621130aa3c6f7f2e739954c" exitCode=0
Sep 16 14:55:20.234981 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:20.234836    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"
Sep 16 14:55:20.234981 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:20.234865    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w" event={"ID":"d7436723-1660-4b8b-9809-7d37af51420e","Type":"ContainerDied","Data":"eea1a32daeeb0ee91de3d35aac1bfb9270de67e89621130aa3c6f7f2e739954c"}
Sep 16 14:55:20.234981 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:20.234938    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w" event={"ID":"d7436723-1660-4b8b-9809-7d37af51420e","Type":"ContainerDied","Data":"3fe6c1576592787a220df7cd284038fe4b17a488a385c9550e9a4079988c5ef4"}
Sep 16 14:55:20.234981 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:20.234964    2553 scope.go:117] "RemoveContainer" containerID="eea1a32daeeb0ee91de3d35aac1bfb9270de67e89621130aa3c6f7f2e739954c"
Sep 16 14:55:20.243673 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:20.243654    2553 scope.go:117] "RemoveContainer" containerID="eea1a32daeeb0ee91de3d35aac1bfb9270de67e89621130aa3c6f7f2e739954c"
Sep 16 14:55:20.243976 ip-10-0-128-50 kubenswrapper[2553]: E0916 14:55:20.243953    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea1a32daeeb0ee91de3d35aac1bfb9270de67e89621130aa3c6f7f2e739954c\": container with ID starting with eea1a32daeeb0ee91de3d35aac1bfb9270de67e89621130aa3c6f7f2e739954c not found: ID does not exist" containerID="eea1a32daeeb0ee91de3d35aac1bfb9270de67e89621130aa3c6f7f2e739954c"
Sep 16 14:55:20.244023 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:20.243987    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea1a32daeeb0ee91de3d35aac1bfb9270de67e89621130aa3c6f7f2e739954c"} err="failed to get container status \"eea1a32daeeb0ee91de3d35aac1bfb9270de67e89621130aa3c6f7f2e739954c\": rpc error: code = NotFound desc = could not find container \"eea1a32daeeb0ee91de3d35aac1bfb9270de67e89621130aa3c6f7f2e739954c\": container with ID starting with eea1a32daeeb0ee91de3d35aac1bfb9270de67e89621130aa3c6f7f2e739954c not found: ID does not exist"
Sep 16 14:55:20.253017 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:20.252995    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"]
Sep 16 14:55:20.257613 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:20.257591    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8488dbc8b7-c654w"]
Sep 16 14:55:21.917403 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:21.917367    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7436723-1660-4b8b-9809-7d37af51420e" path="/var/lib/kubelet/pods/d7436723-1660-4b8b-9809-7d37af51420e/volumes"
Sep 16 14:55:39.568959 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:39.568928    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-z9p4t_20018af2-d3a0-46a0-bafb-c88d059786d0/manager/0.log"
Sep 16 14:55:39.694195 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:39.694168    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mqkwd_8bba3386-b1f4-4638-b6e0-c052a04f207a/manager/0.log"
Sep 16 14:55:40.391498 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:40.391460    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b85944db-lvhcd_7c930415-76ce-4fc9-b21e-c3e7dee47a44/manager/0.log"
Sep 16 14:55:40.415459 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:40.415439    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b85944db-lvhcd_7c930415-76ce-4fc9-b21e-c3e7dee47a44/copy-manifests/0.log"
Sep 16 14:55:41.090609 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:41.090575    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6976c56b87-m7tsw_e6bada29-1967-4510-9bd6-ecbb8987c471/maas-api/0.log"
Sep 16 14:55:41.209624 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:41.209600    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826150-z4pdg_fd519550-0e8b-4fe0-8010-777d0b5859aa/cleanup/0.log"
Sep 16 14:55:41.440651 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:41.440563    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-qxpfl_f8bf5d21-2659-4472-a72f-da2f20e6428a/postgres/0.log"
Sep 16 14:55:42.367482 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:42.367454    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-7np9h_436a2c5e-3287-408c-8111-0bf29d3777f7/manager/0.log"
Sep 16 14:55:42.727541 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:42.727513    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-mkgzm_6da1bce6-0a84-4740-a1ab-2d4452a9e2a1/manager/0.log"
Sep 16 14:55:43.340069 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:43.340042    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5_63adf8da-3abd-458d-aae0-1a70b362c4f4/istio-proxy/0.log"
Sep 16 14:55:44.075310 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:44.075276    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-g9lgv_2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b/payload-processing/0.log"
Sep 16 14:55:44.794848 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:44.794817    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-p6j8z_4ef12922-dcf0-4d90-8c04-0f63d2c36375/main/0.log"
Sep 16 14:55:45.028198 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:45.028168    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpbw2b_4a009633-059b-4702-bbdd-ad7d8c083a04/main/0.log"
Sep 16 14:55:53.730637 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:53.730608    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kdnvs_c23242c8-2ee7-4041-b310-6538da68c470/global-pull-secret-syncer/0.log"
Sep 16 14:55:53.879124 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:53.879092    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kswqv_345b6071-6885-4fa9-8a8f-cc1b40368688/konnectivity-agent/0.log"
Sep 16 14:55:53.943195 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:53.943168    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-50.ec2.internal_cc9a0b5b683b9efc648017b7df9cf169/haproxy/0.log"
Sep 16 14:55:58.181022 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:58.180996    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-7np9h_436a2c5e-3287-408c-8111-0bf29d3777f7/manager/0.log"
Sep 16 14:55:58.289071 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:58.289037    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-mkgzm_6da1bce6-0a84-4740-a1ab-2d4452a9e2a1/manager/0.log"
Sep 16 14:55:59.980843 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:55:59.980815    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_984dce34-c05d-48c1-9686-055dabd6364e/alertmanager/0.log"
Sep 16 14:56:00.025082 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:00.025054    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_984dce34-c05d-48c1-9686-055dabd6364e/config-reloader/0.log"
Sep 16 14:56:00.056242 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:00.056217    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_984dce34-c05d-48c1-9686-055dabd6364e/kube-rbac-proxy-web/0.log"
Sep 16 14:56:00.090540 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:00.090518    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_984dce34-c05d-48c1-9686-055dabd6364e/kube-rbac-proxy/0.log"
Sep 16 14:56:00.125794 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:00.125768    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_984dce34-c05d-48c1-9686-055dabd6364e/kube-rbac-proxy-metric/0.log"
Sep 16 14:56:00.168488 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:00.168467    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_984dce34-c05d-48c1-9686-055dabd6364e/prom-label-proxy/0.log"
Sep 16 14:56:00.215493 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:00.215472    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_984dce34-c05d-48c1-9686-055dabd6364e/init-config-reloader/0.log"
Sep 16 14:56:00.301785 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:00.301751    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-5l2fd_136f6c0b-4717-4e2b-b788-fa72b68f6eb8/cluster-monitoring-operator/0.log"
Sep 16 14:56:00.369865 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:00.369832    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-5pxl9_62ac91a1-838a-4136-b993-c630a0fde35a/kube-state-metrics/0.log"
Sep 16 14:56:00.403663 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:00.403638    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-5pxl9_62ac91a1-838a-4136-b993-c630a0fde35a/kube-rbac-proxy-main/0.log"
Sep 16 14:56:00.455375 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:00.455346    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-5pxl9_62ac91a1-838a-4136-b993-c630a0fde35a/kube-rbac-proxy-self/0.log"
Sep 16 14:56:00.756171 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:00.756148    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dkdrg_a2de9462-1851-4e5e-90c6-3fe7fb19b91c/node-exporter/0.log"
Sep 16 14:56:00.825577 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:00.825550    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dkdrg_a2de9462-1851-4e5e-90c6-3fe7fb19b91c/kube-rbac-proxy/0.log"
Sep 16 14:56:00.858631 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:00.858612    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dkdrg_a2de9462-1851-4e5e-90c6-3fe7fb19b91c/init-textfile/0.log"
Sep 16 14:56:01.232962 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:01.232934    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fbc3a51b-ec10-43c1-acf9-4ccd064500be/prometheus/0.log"
Sep 16 14:56:01.273351 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:01.273324    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fbc3a51b-ec10-43c1-acf9-4ccd064500be/config-reloader/0.log"
Sep 16 14:56:01.317314 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:01.317279    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fbc3a51b-ec10-43c1-acf9-4ccd064500be/thanos-sidecar/0.log"
Sep 16 14:56:01.358772 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:01.358748    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fbc3a51b-ec10-43c1-acf9-4ccd064500be/kube-rbac-proxy-web/0.log"
Sep 16 14:56:01.415904 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:01.415863    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fbc3a51b-ec10-43c1-acf9-4ccd064500be/kube-rbac-proxy/0.log"
Sep 16 14:56:01.463254 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:01.463229    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fbc3a51b-ec10-43c1-acf9-4ccd064500be/kube-rbac-proxy-thanos/0.log"
Sep 16 14:56:01.537278 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:01.537209    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fbc3a51b-ec10-43c1-acf9-4ccd064500be/init-config-reloader/0.log"
Sep 16 14:56:01.727243 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:01.727217    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-54q4d_dd7160a7-b3ca-4104-a986-2ad932a3874d/prometheus-operator-admission-webhook/0.log"
Sep 16 14:56:01.788609 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:01.788532    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76c585499c-v4r57_3c4fc912-3823-4df0-81de-668f4708b204/telemeter-client/0.log"
Sep 16 14:56:01.838197 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:01.838172    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76c585499c-v4r57_3c4fc912-3823-4df0-81de-668f4708b204/reload/0.log"
Sep 16 14:56:01.878197 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:01.878171    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76c585499c-v4r57_3c4fc912-3823-4df0-81de-668f4708b204/kube-rbac-proxy/0.log"
Sep 16 14:56:03.685049 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:03.685010    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-csd2f_a5f306dc-d337-4d52-8d54-f115dc90fcc5/networking-console-plugin/0.log"
Sep 16 14:56:04.516015 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:04.515990    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/1.log"
Sep 16 14:56:04.523620 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:04.523594    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s4pkv_ce0aa50c-e70c-4d0a-86e2-ee90fa008ebe/console-operator/2.log"
Sep 16 14:56:04.823929 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:04.823841    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"]
Sep 16 14:56:04.824309 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:04.824295    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7436723-1660-4b8b-9809-7d37af51420e" containerName="maas-api"
Sep 16 14:56:04.824356 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:04.824310    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7436723-1660-4b8b-9809-7d37af51420e" containerName="maas-api"
Sep 16 14:56:04.824406 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:04.824397    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7436723-1660-4b8b-9809-7d37af51420e" containerName="maas-api"
Sep 16 14:56:04.827740 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:04.827719    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:04.830553 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:04.830535    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vbsd8\"/\"kube-root-ca.crt\""
Sep 16 14:56:04.830660 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:04.830610    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vbsd8\"/\"openshift-service-ca.crt\""
Sep 16 14:56:04.831561 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:04.831546    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vbsd8\"/\"default-dockercfg-gkpxd\""
Sep 16 14:56:04.846415 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:04.846391    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"]
Sep 16 14:56:04.988660 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:04.988628    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b586c38e-acbe-4811-bc5e-005ddffc983b-podres\") pod \"perf-node-gather-daemonset-xnlpv\" (UID: \"b586c38e-acbe-4811-bc5e-005ddffc983b\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:04.988660 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:04.988664    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42xzr\" (UniqueName: \"kubernetes.io/projected/b586c38e-acbe-4811-bc5e-005ddffc983b-kube-api-access-42xzr\") pod \"perf-node-gather-daemonset-xnlpv\" (UID: \"b586c38e-acbe-4811-bc5e-005ddffc983b\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:04.988937 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:04.988699    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b586c38e-acbe-4811-bc5e-005ddffc983b-sys\") pod \"perf-node-gather-daemonset-xnlpv\" (UID: \"b586c38e-acbe-4811-bc5e-005ddffc983b\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:04.988937 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:04.988827    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b586c38e-acbe-4811-bc5e-005ddffc983b-lib-modules\") pod \"perf-node-gather-daemonset-xnlpv\" (UID: \"b586c38e-acbe-4811-bc5e-005ddffc983b\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:04.988937 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:04.988875    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b586c38e-acbe-4811-bc5e-005ddffc983b-proc\") pod \"perf-node-gather-daemonset-xnlpv\" (UID: \"b586c38e-acbe-4811-bc5e-005ddffc983b\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:05.089796 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:05.089697    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b586c38e-acbe-4811-bc5e-005ddffc983b-sys\") pod \"perf-node-gather-daemonset-xnlpv\" (UID: \"b586c38e-acbe-4811-bc5e-005ddffc983b\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:05.089796 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:05.089784    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b586c38e-acbe-4811-bc5e-005ddffc983b-lib-modules\") pod \"perf-node-gather-daemonset-xnlpv\" (UID: \"b586c38e-acbe-4811-bc5e-005ddffc983b\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:05.090083 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:05.089823    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b586c38e-acbe-4811-bc5e-005ddffc983b-proc\") pod \"perf-node-gather-daemonset-xnlpv\" (UID: \"b586c38e-acbe-4811-bc5e-005ddffc983b\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:05.090083 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:05.089830    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b586c38e-acbe-4811-bc5e-005ddffc983b-sys\") pod \"perf-node-gather-daemonset-xnlpv\" (UID: \"b586c38e-acbe-4811-bc5e-005ddffc983b\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:05.090083 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:05.089926    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b586c38e-acbe-4811-bc5e-005ddffc983b-podres\") pod \"perf-node-gather-daemonset-xnlpv\" (UID: \"b586c38e-acbe-4811-bc5e-005ddffc983b\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:05.090083 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:05.089959    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42xzr\" (UniqueName: \"kubernetes.io/projected/b586c38e-acbe-4811-bc5e-005ddffc983b-kube-api-access-42xzr\") pod \"perf-node-gather-daemonset-xnlpv\" (UID: \"b586c38e-acbe-4811-bc5e-005ddffc983b\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:05.090083 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:05.089988    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b586c38e-acbe-4811-bc5e-005ddffc983b-lib-modules\") pod \"perf-node-gather-daemonset-xnlpv\" (UID: \"b586c38e-acbe-4811-bc5e-005ddffc983b\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:05.090083 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:05.089997    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b586c38e-acbe-4811-bc5e-005ddffc983b-podres\") pod \"perf-node-gather-daemonset-xnlpv\" (UID: \"b586c38e-acbe-4811-bc5e-005ddffc983b\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:05.090083 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:05.089928    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b586c38e-acbe-4811-bc5e-005ddffc983b-proc\") pod \"perf-node-gather-daemonset-xnlpv\" (UID: \"b586c38e-acbe-4811-bc5e-005ddffc983b\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:05.108240 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:05.108212    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42xzr\" (UniqueName: \"kubernetes.io/projected/b586c38e-acbe-4811-bc5e-005ddffc983b-kube-api-access-42xzr\") pod \"perf-node-gather-daemonset-xnlpv\" (UID: \"b586c38e-acbe-4811-bc5e-005ddffc983b\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:05.138050 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:05.138030    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:05.414711 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:05.414679    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"]
Sep 16 14:56:05.415870 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:05.415847    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-phq9c_4f3af190-2324-42b9-8ade-3ddd9189df6e/download-server/0.log"
Sep 16 14:56:05.416623 ip-10-0-128-50 kubenswrapper[2553]: W0916 14:56:05.416598    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb586c38e_acbe_4811_bc5e_005ddffc983b.slice/crio-2e97c32b92aadfec8edf0e82cb778df8e59becf08345c5939e0568e97b77a96f WatchSource:0}: Error finding container 2e97c32b92aadfec8edf0e82cb778df8e59becf08345c5939e0568e97b77a96f: Status 404 returned error can't find the container with id 2e97c32b92aadfec8edf0e82cb778df8e59becf08345c5939e0568e97b77a96f
Sep 16 14:56:05.463041 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:05.463014    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv" event={"ID":"b586c38e-acbe-4811-bc5e-005ddffc983b","Type":"ContainerStarted","Data":"2e97c32b92aadfec8edf0e82cb778df8e59becf08345c5939e0568e97b77a96f"}
Sep 16 14:56:06.190212 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:06.190182    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-25dg9_667e0c83-a29e-44a0-8cf7-936e046f6f05/volume-data-source-validator/0.log"
Sep 16 14:56:06.470041 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:06.470010    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv" event={"ID":"b586c38e-acbe-4811-bc5e-005ddffc983b","Type":"ContainerStarted","Data":"1e5681371444666cbdb7ecca090f29d6fcd0fd3e724ece2b68b72ec5db41629d"}
Sep 16 14:56:06.470210 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:06.470125    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:06.497425 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:06.497385    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv" podStartSLOduration=2.497371339 podStartE2EDuration="2.497371339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:56:06.496226357 +0000 UTC m=+2517.042968318" watchObservedRunningTime="2026-09-16 14:56:06.497371339 +0000 UTC m=+2517.044113306"
Sep 16 14:56:07.749649 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:07.749623    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9x4k6_68b65eb9-84ac-48c6-8ffc-49c9eee408b3/dns/0.log"
Sep 16 14:56:07.775164 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:07.775143    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9x4k6_68b65eb9-84ac-48c6-8ffc-49c9eee408b3/kube-rbac-proxy/0.log"
Sep 16 14:56:07.962654 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:07.962629    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s79nx_846a7905-fecc-4b70-b055-ee249dde8cb6/dns-node-resolver/0.log"
Sep 16 14:56:08.779246 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:08.779221    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-255dt_279e9549-530e-4e54-afbe-9c0954c59441/node-ca/0.log"
Sep 16 14:56:10.181193 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:10.181167    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5_63adf8da-3abd-458d-aae0-1a70b362c4f4/istio-proxy/0.log"
Sep 16 14:56:11.263687 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:11.263652    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-g9lgv_2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b/payload-processing/0.log"
Sep 16 14:56:12.448795 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:12.448760    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x7sq6_d906b113-d506-4232-8e95-23f47268744c/serve-healthcheck-canary/0.log"
Sep 16 14:56:12.485447 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:12.485417    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-xnlpv"
Sep 16 14:56:13.343672 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:13.343638    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mbq9r_3b2463f9-3054-4d2d-94c5-b07c406a9187/kube-rbac-proxy/0.log"
Sep 16 14:56:13.405584 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:13.405559    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mbq9r_3b2463f9-3054-4d2d-94c5-b07c406a9187/exporter/0.log"
Sep 16 14:56:13.485669 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:13.485642    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mbq9r_3b2463f9-3054-4d2d-94c5-b07c406a9187/extractor/0.log"
Sep 16 14:56:13.855903 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:13.855859    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5_63adf8da-3abd-458d-aae0-1a70b362c4f4/istio-proxy/0.log"
Sep 16 14:56:14.627986 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:14.627953    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-g9lgv_2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b/payload-processing/0.log"
Sep 16 14:56:16.509225 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:16.509198    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-5c2x5_63adf8da-3abd-458d-aae0-1a70b362c4f4/istio-proxy/0.log"
Sep 16 14:56:16.741411 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:16.741382    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-z9p4t_20018af2-d3a0-46a0-bafb-c88d059786d0/manager/0.log"
Sep 16 14:56:17.217099 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:17.217066    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mqkwd_8bba3386-b1f4-4638-b6e0-c052a04f207a/manager/0.log"
Sep 16 14:56:17.806209 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:17.806180    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-g9lgv_2b9d6dec-6bfa-483b-92a3-f1bc5e6c624b/payload-processing/0.log"
Sep 16 14:56:18.474839 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:18.474814    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b85944db-lvhcd_7c930415-76ce-4fc9-b21e-c3e7dee47a44/manager/0.log"
Sep 16 14:56:18.533921 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:18.533884    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b85944db-lvhcd_7c930415-76ce-4fc9-b21e-c3e7dee47a44/copy-manifests/0.log"
Sep 16 14:56:20.238499 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:20.238467    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5cbd5cd6f8-4kxhb_a683d72c-2060-4e6e-b3cb-050dd34e530a/manager/0.log"
Sep 16 14:56:20.345732 ip-10-0-128-50 kubenswrapper[2553]: I0916 14:56:20.345692    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-2wq7l_66c9f5c5-c461-46e1-86ac-b6429afe83ff/openshift-lws-operator/0.log"