Jul 28 18:01:04.560108 ip-10-0-139-99 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 18:01:04.980511 ip-10-0-139-99 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 18:01:04.980511 ip-10-0-139-99 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 18:01:04.980511 ip-10-0-139-99 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 18:01:04.980511 ip-10-0-139-99 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 18:01:04.980511 ip-10-0-139-99 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 18:01:04.981630 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.981516    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 18:01:04.985220 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985198    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 18:01:04.985220 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985215    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 18:01:04.985220 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985219    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 18:01:04.985220 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985223    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 18:01:04.985220 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985227    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985231    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985235    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985243    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985249    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985253    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985257    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985260    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985264    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985268    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985272    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985275    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985279    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985283    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985287    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985290    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985294    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985305    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985310    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 18:01:04.985503 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985316    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985320    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985324    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985328    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985332    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985336    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985340    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985344    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985348    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985352    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985356    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985361    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985365    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985370    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985376    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985383    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985387    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985392    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985398    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985403    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 18:01:04.986020 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985407    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985412    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985416    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985421    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985424    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985429    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985433    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985438    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985442    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985446    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985452    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985457    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985461    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985465    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985469    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985475    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985480    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985484    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985488    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985492    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 18:01:04.986708 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985497    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985501    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985505    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985510    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985514    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985518    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985522    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985526    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985531    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985535    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985539    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985544    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985549    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985553    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985558    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985562    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.985566    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986288    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986297    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986303    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 18:01:04.987553 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986308    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986316    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986321    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986325    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986330    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986335    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986339    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986343    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986348    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986353    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986357    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986361    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986366    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986371    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986378    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.986383    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987581    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987592    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987598    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987603    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 18:01:04.988556 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987608    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987613    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987618    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987622    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987627    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987632    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987637    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987641    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987646    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987650    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987654    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987658    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987663    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987667    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987672    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987677    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987681    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987686    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987690    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987694    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 18:01:04.989421 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987699    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987703    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987707    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987711    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987716    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987720    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987724    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987728    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987733    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987737    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987742    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987746    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987750    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987754    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987759    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987763    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987767    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987771    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987776    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987780    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 18:01:04.990316 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987786    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987792    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987796    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987801    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987822    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987827    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987832    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987836    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987840    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987844    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987849    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987853    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987858    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987863    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987867    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987871    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987875    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987878    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987883    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987887    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 18:01:04.991016 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987891    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987895    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987901    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987906    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987911    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987915    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987919    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987923    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987928    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987934    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987938    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987942    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987948    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987953    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987957    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987962    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987966    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987971    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 18:01:04.991551 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987976    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987980    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987984    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987988    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987993    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.987998    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988002    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988006    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988010    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988014    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988018    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988022    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988026    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988031    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988035    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988039    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988043    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988644    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988652    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988656    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 18:01:04.992055 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988661    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988666    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988670    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988674    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988679    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988683    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988687    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988691    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988695    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988699    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988704    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988708    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988712    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988717    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988721    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988726    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988730    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988734    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988739    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 18:01:04.992621 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988743    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988747    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988751    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988756    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988760    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988764    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988768    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988772    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988777    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988781    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988785    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988789    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988793    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988797    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988802    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988821    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988825    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988829    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988858    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988864    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 18:01:04.993128 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988869    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988874    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988882    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988890    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988895    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988900    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988905    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988909    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988914    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988918    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988922    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988927    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988931    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988935    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988939    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988946    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988951    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988955    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988960    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988964    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 18:01:04.993606 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988969    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988973    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988977    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988981    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988985    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988989    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988993    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.988997    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989002    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989005    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989009    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989013    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989017    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989021    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989026    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989033    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989037    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989042    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989046    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989050    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 18:01:04.994304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989054    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989058    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989062    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989068    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989074    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989079    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989083    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989087    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989091    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989095    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989099    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989103    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989110    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989114    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989118    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989122    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.989127    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989216    2564 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989227    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989236    2564 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989243    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 18:01:04.994910 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989250    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989272    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989280    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989287    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989293    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989297    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989305    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989311    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989316    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989321    2564 flags.go:64] FLAG: --cgroup-root=""
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989326    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989331    2564 flags.go:64] FLAG: --client-ca-file=""
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989336    2564 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989340    2564 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989348    2564 flags.go:64] FLAG: --cluster-domain=""
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989353    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989358    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989364    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989369    2564 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989375    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989380    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989384    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989390    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989396    2564 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 18:01:04.995409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989401    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989406    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989411    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989416    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989422    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989426    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989430    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989435    2564 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989439    2564 flags.go:64] FLAG: --enable-server="true"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989444    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989450    2564 flags.go:64] FLAG: --event-burst="100"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989456    2564 flags.go:64] FLAG: --event-qps="50"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989461    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989467    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989474    2564 flags.go:64] FLAG: --eviction-hard=""
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989480    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989485    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989489    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989494    2564 flags.go:64] FLAG: --eviction-soft=""
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989499    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989504    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989509    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989513    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989518    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989523    2564 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 18:01:04.996110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989528    2564 flags.go:64] FLAG: --feature-gates=""
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989534    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989539    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989544    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989549    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989554    2564 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989560    2564 flags.go:64] FLAG: --help="false"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989565    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-139-99.ec2.internal"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989570    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989575    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989580    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989585    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989591    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989596    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989600    2564 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989605    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989610    2564 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989615    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989620    2564 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989625    2564 flags.go:64] FLAG: --kube-reserved=""
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989629    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989636    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989641    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989646    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 18:01:04.996753 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989650    2564 flags.go:64] FLAG: --lock-file=""
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989654    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989659    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989664    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989674    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989678    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989683    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989688    2564 flags.go:64] FLAG: --logging-format="text"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989692    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989699    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989704    2564 flags.go:64] FLAG: --manifest-url=""
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989709    2564 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989715    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989720    2564 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989728    2564 flags.go:64] FLAG: --max-pods="110"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989733    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989739    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989743    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989748    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989753    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989758    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989762    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989773    2564 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989778    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989783    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 18:01:04.997346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989788    2564 flags.go:64] FLAG: --pod-cidr=""
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989793    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989801    2564 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989824    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989832    2564 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989837    2564 flags.go:64] FLAG: --port="10250"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989842    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989847    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0157b20b79b1472e4"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989852    2564 flags.go:64] FLAG: --qos-reserved=""
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989857    2564 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989862    2564 flags.go:64] FLAG: --register-node="true"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989866    2564 flags.go:64] FLAG: --register-with-taints=""
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989872    2564 flags.go:64] FLAG: --registry-burst="10"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989877    2564 flags.go:64] FLAG: --registry-qps="5"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989882    2564 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989886    2564 flags.go:64] FLAG: --reserved-memory=""
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989892    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989898    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989903    2564 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989908    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989913    2564 flags.go:64] FLAG: --runonce="false"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989918    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989924    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989930    2564 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989934    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 18:01:04.997983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989939    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989944    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989949    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989954    2564 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989959    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989963    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989968    2564 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989973    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989978    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989982    2564 flags.go:64] FLAG: --system-cgroups=""
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989987    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.989997    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.990003    2564 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.990008    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.990015    2564 flags.go:64] FLAG: --tls-min-version=""
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.990019    2564 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.990024    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.990029    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.990033    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.990038    2564 flags.go:64] FLAG: --v="2"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.990044    2564 flags.go:64] FLAG: --version="false"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.990050    2564 flags.go:64] FLAG: --vmodule=""
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.990056    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.990062    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990245    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 18:01:04.998656 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990253    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990258    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990263    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990267    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990273    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990278    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990282    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990287    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990293    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990299    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990304    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990308    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990313    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990318    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990322    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990326    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990330    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990334    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990341    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 18:01:04.999339 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990345    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990349    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990354    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990359    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990365    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990370    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990375    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990380    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990385    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990390    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990395    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990400    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990405    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990410    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990415    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990420    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990425    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990431    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990436    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990441    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 18:01:04.999988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990445    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990450    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990454    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990459    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990463    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990468    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990472    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990476    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990480    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990485    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990489    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990495    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990500    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990504    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990508    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990513    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990517    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990521    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990525    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990529    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 18:01:05.000637 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990534    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990540    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990547    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990551    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990555    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990559    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990564    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990570    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990574    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990580    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990584    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990589    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990593    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990597    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990601    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990605    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990610    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990614    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990618    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990622    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 18:01:05.001157 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990626    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990631    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990635    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990641    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990645    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990650    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990654    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990658    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990663    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990667    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990671    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990676    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990680    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990684    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990688    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990693    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990697    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990702    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 18:01:05.001644 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:04.990706    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 18:01:05.002158 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:04.991438    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 18:01:05.002158 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.001142    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 18:01:05.002158 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.001157    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 18:01:05.002158 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.002074    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 18:01:05.002158 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.002082    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 18:01:05.002158 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002138    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 18:01:05.002158 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002143    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 18:01:05.002158 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002148    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 18:01:05.002158 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002152    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 18:01:05.002158 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002155    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 18:01:05.002158 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002158    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 18:01:05.002158 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002161    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 18:01:05.002158 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002164    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002168    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002171    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002174    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002177    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002179    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002182    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002184    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002187    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002190    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002193    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002195    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002198    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002200    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002204    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002207    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002210    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002213    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002215    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002219    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 18:01:05.002489 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002223    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002226    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002229    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002231    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002234    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002236    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002239    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002241    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002244    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002246    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002250    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002254    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002257    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002260    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002264    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002267    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002270    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002272    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002275    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 18:01:05.002988 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002278    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002281    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002284    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002286    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002289    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002291    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002294    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002296    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002300    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002302    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002304    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002307    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002310    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002312    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002315    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002317    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002320    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002322    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002325    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002328    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002330    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 18:01:05.003445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002333    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002335    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002338    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002340    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002343    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002346    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002350    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002353    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002355    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002358    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002360    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002363    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002366    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002369    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002371    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002374    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002376    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002379    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002381    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002384    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 18:01:05.003963 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002386    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 18:01:05.004444 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002389    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 18:01:05.004444 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002392    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 18:01:05.004444 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002394    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 18:01:05.004444 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002397    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 18:01:05.004444 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002399    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 18:01:05.004444 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002402    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 18:01:05.004444 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002404    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 18:01:05.004444 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002407    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 18:01:05.004444 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002409    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 18:01:05.004444 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002412    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 18:01:05.004444 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002414    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 18:01:05.004444 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.002419    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 18:01:05.004444 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002536    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 18:01:05.004444 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002541    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002544    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002547    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002551    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002554    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002557    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002560    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002562    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002565    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002569    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002573    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002579    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002582    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002585    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002588    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002591    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002594    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002596    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002599    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 18:01:05.004798 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002602    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002605    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002608    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002611    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002613    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002616    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002618    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002621    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002623    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002626    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002629    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002632    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002635    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002637    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002640    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002643    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002645    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002648    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002651    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002654    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 18:01:05.005262 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002656    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002659    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002662    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002665    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002667    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002670    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002673    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002675    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002678    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002680    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002683    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002685    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002689    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002692    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002695    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002698    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002700    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002703    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002706    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002708    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 18:01:05.005744 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002711    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002713    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002716    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002719    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002721    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002723    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002726    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002728    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002731    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002734    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002737    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002739    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002743    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002746    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002749    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002752    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002754    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002757    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002760    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002763    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 18:01:05.006284 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002766    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002768    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002770    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002773    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002775    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002778    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002781    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002783    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002786    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002788    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002791    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002793    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002796    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002798    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002801    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002818    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002823    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002825    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 18:01:05.006761 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:05.002828    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 18:01:05.007222 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.002833    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 18:01:05.007222 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.002930    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 18:01:05.007222 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.005693    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 18:01:05.007222 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.007176    2564 server.go:1019] "Starting client certificate rotation"
Jul 28 18:01:05.007343 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.007271    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 18:01:05.007343 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.007311    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 18:01:05.010012 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.009999    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 18:01:05.015484 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.015468    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 18:01:05.028317 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.028302    2564 log.go:25] "Validated CRI v1 runtime API"
Jul 28 18:01:05.030592 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.030570    2564 log.go:25] "Validated CRI v1 image API"
Jul 28 18:01:05.031482 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.031467    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 18:01:05.034635 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.034610    2564 fs.go:135] Filesystem UUIDs: map[542d1a7d-4f6a-41ca-8361-d4a431d65ea7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d636f58d-c16e-42d3-aeba-80b82c5cda9e:/dev/nvme0n1p3]
Jul 28 18:01:05.034635 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.034629    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 18:01:05.039039 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.039016    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 18:01:05.041311 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.041204    2564 manager.go:216] Machine: {Timestamp:2026-07-28 18:01:05.038882965 +0000 UTC m=+0.354385088 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109447 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec216e52748bdcb3bf8755064c6f08c2 SystemUUID:ec216e52-748b-dcb3-bf87-55064c6f08c2 BootID:e10de1b1-1f1c-404c-9e71-621c5b2fcfcc Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:96:af:a4:35:25 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:96:af:a4:35:25 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:f6:1f:f4:e8:67 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 18:01:05.041311 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.041305    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 18:01:05.041422 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.041379    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 18:01:05.044604 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.044568    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 18:01:05.044735 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.044606    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 18:01:05.045930 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.045919    2564 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 18:01:05.045970 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.045933    2564 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 18:01:05.045970 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.045940    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 18:01:05.046606 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.046596    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 18:01:05.047755 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.047744    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 18:01:05.050036 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.050024    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 18:01:05.050141 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.050131    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 18:01:05.050185 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.050160    2564 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 18:01:05.050185 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.050173    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 18:01:05.050185 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.050185    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 18:01:05.050314 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.050194    2564 kubelet.go:398] "Adding apiserver pod source"
Jul 28 18:01:05.050314 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.050204    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 18:01:05.050570 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.050555    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 18:01:05.050628 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.050573    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 18:01:05.055416 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.055399    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 18:01:05.057372 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.057359    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 18:01:05.057436 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.057387    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 18:01:05.059106 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.059083    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 18:01:05.059184 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.059116    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 18:01:05.059184 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.059125    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 18:01:05.059184 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.059135    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 18:01:05.059184 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.059156    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 18:01:05.059184 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.059162    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 18:01:05.059184 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.059170    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 18:01:05.059184 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.059175    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 18:01:05.059184 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.059182    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 18:01:05.059184 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.059188    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 18:01:05.059409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.059197    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 18:01:05.059409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.059202    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 18:01:05.059896 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.059885    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 18:01:05.059896 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.059897    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 18:01:05.060135 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.060111    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 18:01:05.060228 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.060204    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 18:01:05.063162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.063148    2564 server.go:1268] "Started kubelet"
Jul 28 18:01:05.063245 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.063220    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 18:01:05.063309 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.063223    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 18:01:05.063309 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.063294    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 18:01:05.063503 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.063486    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 18:01:05.064025 ip-10-0-139-99 systemd[1]: Started Kubernetes Kubelet.
Jul 28 18:01:05.065144 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.064960    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 18:01:05.065362 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.065342    2564 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 18:01:05.069025 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.069002    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gmmmd"
Jul 28 18:01:05.070046 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.070014    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 18:01:05.072161 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.071184    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-99.ec2.internal.18c686069135b8d6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-99.ec2.internal,UID:ip-10-0-139-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-99.ec2.internal,},FirstTimestamp:2026-07-28 18:01:05.063123158 +0000 UTC m=+0.378625261,LastTimestamp:2026-07-28 18:01:05.063123158 +0000 UTC m=+0.378625261,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-99.ec2.internal,}"
Jul 28 18:01:05.072161 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.072158    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 18:01:05.072299 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.072164    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 18:01:05.072299 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.072253    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Jul 28 18:01:05.072781 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.072769    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 18:01:05.072898 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.072874    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 18:01:05.073015 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.072769    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 18:01:05.073015 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.072982    2564 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 18:01:05.073015 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.072986    2564 factory.go:55] Registering systemd factory
Jul 28 18:01:05.073141 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.073022    2564 factory.go:223] Registration of the systemd container factory successfully
Jul 28 18:01:05.073141 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.072991    2564 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 18:01:05.073414 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.073400    2564 factory.go:153] Registering CRI-O factory
Jul 28 18:01:05.073456 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.073419    2564 factory.go:223] Registration of the crio container factory successfully
Jul 28 18:01:05.073489 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.073472    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 18:01:05.073523 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.073494    2564 factory.go:103] Registering Raw factory
Jul 28 18:01:05.073523 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.073509    2564 manager.go:1196] Started watching for new ooms in manager
Jul 28 18:01:05.073990 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.073977    2564 manager.go:349] Starting recovery of all containers
Jul 28 18:01:05.079154 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.079125    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gmmmd"
Jul 28 18:01:05.080126 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.080098    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 18:01:05.080240 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.080216    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 18:01:05.086522 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.086504    2564 manager.go:354] Recovery completed
Jul 28 18:01:05.090911 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.090898    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 18:01:05.092890 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.092876    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 18:01:05.092986 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.092903    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 18:01:05.092986 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.092913    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientPID"
Jul 28 18:01:05.093471 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.093459    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 18:01:05.093471 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.093470    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 18:01:05.093552 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.093471    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 18:01:05.093552 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.093545    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 18:01:05.094147 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.094132    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 18:01:05.094204 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.094158    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 18:01:05.094204 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.094170    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientPID"
Jul 28 18:01:05.094831 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.094717    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-99.ec2.internal.18c6860692fbee93  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-99.ec2.internal,UID:ip-10-0-139-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-99.ec2.internal,},FirstTimestamp:2026-07-28 18:01:05.092890259 +0000 UTC m=+0.408392364,LastTimestamp:2026-07-28 18:01:05.092890259 +0000 UTC m=+0.408392364,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-99.ec2.internal,}"
Jul 28 18:01:05.095351 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.095340    2564 policy_none.go:49] "None policy: Start"
Jul 28 18:01:05.095387 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.095355    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 18:01:05.095387 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.095364    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 18:01:05.096497 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.096486    2564 policy_none.go:47] "Start"
Jul 28 18:01:05.136457 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.136327    2564 manager.go:337] "Starting Device Plugin manager"
Jul 28 18:01:05.136554 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.136481    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 18:01:05.136554 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.136494    2564 server.go:87] "Starting device plugin registration server"
Jul 28 18:01:05.136744 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.136731    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 18:01:05.136828 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.136746    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 18:01:05.136875 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.136851    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 18:01:05.136947 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.136933    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 18:01:05.136947 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.136946    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 18:01:05.137457 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.137426    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 18:01:05.137531 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.137480    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-99.ec2.internal\" not found"
Jul 28 18:01:05.209138 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.209110    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 18:01:05.210942 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.210228    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 18:01:05.210942 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.210274    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 18:01:05.210942 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.210300    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 18:01:05.210942 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.210340    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 18:01:05.213459 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.213441    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 18:01:05.237096 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.237027    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 18:01:05.238012 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.237993    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 18:01:05.238128 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.238029    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 18:01:05.238128 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.238046    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientPID"
Jul 28 18:01:05.238128 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.238074    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.245323 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.245306    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.245391 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.245333    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-99.ec2.internal\": node \"ip-10-0-139-99.ec2.internal\" not found"
Jul 28 18:01:05.256582 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.256564    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Jul 28 18:01:05.310710 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.310676    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"]
Jul 28 18:01:05.310882 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.310754    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 18:01:05.311842 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.311803    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 18:01:05.311941 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.311852    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 18:01:05.311941 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.311863    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientPID"
Jul 28 18:01:05.313441 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.313427    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 18:01:05.313560 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.313545    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.313601 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.313581    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 18:01:05.314243 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.314226    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 18:01:05.314350 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.314242    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 18:01:05.314350 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.314259    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 18:01:05.314350 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.314268    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 18:01:05.314350 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.314274    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientPID"
Jul 28 18:01:05.314350 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.314281    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientPID"
Jul 28 18:01:05.315855 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.315839    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.315923 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.315874    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 18:01:05.316593 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.316578    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 18:01:05.316666 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.316605    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 18:01:05.316666 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.316617    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeHasSufficientPID"
Jul 28 18:01:05.342677 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.342653    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-99.ec2.internal\" not found" node="ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.346332 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.346317    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-99.ec2.internal\" not found" node="ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.357567 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.357547    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Jul 28 18:01:05.375255 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.375227    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6226ec06bb69fa100018d0ee4c073bb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"c6226ec06bb69fa100018d0ee4c073bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.375255 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.375254    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6226ec06bb69fa100018d0ee4c073bb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"c6226ec06bb69fa100018d0ee4c073bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.375408 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.375276    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6da522a8a76aa2b396d458299e956b18-config\") pod \"kube-apiserver-proxy-ip-10-0-139-99.ec2.internal\" (UID: \"6da522a8a76aa2b396d458299e956b18\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.458192 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.458159    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Jul 28 18:01:05.475820 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.475779    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6226ec06bb69fa100018d0ee4c073bb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"c6226ec06bb69fa100018d0ee4c073bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.475906 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.475811    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6226ec06bb69fa100018d0ee4c073bb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"c6226ec06bb69fa100018d0ee4c073bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.475906 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.475859    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6da522a8a76aa2b396d458299e956b18-config\") pod \"kube-apiserver-proxy-ip-10-0-139-99.ec2.internal\" (UID: \"6da522a8a76aa2b396d458299e956b18\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.475906 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.475879    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6226ec06bb69fa100018d0ee4c073bb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"c6226ec06bb69fa100018d0ee4c073bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.476018 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.475885    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6226ec06bb69fa100018d0ee4c073bb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal\" (UID: \"c6226ec06bb69fa100018d0ee4c073bb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.476018 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.475891    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6da522a8a76aa2b396d458299e956b18-config\") pod \"kube-apiserver-proxy-ip-10-0-139-99.ec2.internal\" (UID: \"6da522a8a76aa2b396d458299e956b18\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.558829 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.558726    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Jul 28 18:01:05.651313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.651269    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.651454 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:05.651269    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Jul 28 18:01:05.659003 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.658981    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Jul 28 18:01:05.759790 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.759751    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Jul 28 18:01:05.860305 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.860229    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Jul 28 18:01:05.961632 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:05.961584    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Jul 28 18:01:06.007452 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:06.007425    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 18:01:06.007917 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:06.007612    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 18:01:06.062369 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:06.062330    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-99.ec2.internal\" not found"
Jul 28 18:01:06.084795 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:06.084755    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:56:05 +0000 UTC" deadline="2028-03-07 06:23:42.056098671 +0000 UTC"
Jul 28 18:01:06.084795 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:06.084792    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14100h22m35.971311384s"
Jul 28 18:01:06.128054 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:06.127985    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 18:01:06.172657 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:06.172630    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal"
Jul 28 18:01:06.185609 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:06.185589    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 18:01:06.187553 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:06.187539    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal"
Jul 28 18:01:06.198515 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:06.198493    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 18:01:06.262275 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:06.262243    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 18:01:06.397445 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:06.397403    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6226ec06bb69fa100018d0ee4c073bb.slice/crio-88f56661da00fa416bc13d4e11524842a8b4641f4fe5db1e13ada7662abfaad0 WatchSource:0}: Error finding container 88f56661da00fa416bc13d4e11524842a8b4641f4fe5db1e13ada7662abfaad0: Status 404 returned error can't find the container with id 88f56661da00fa416bc13d4e11524842a8b4641f4fe5db1e13ada7662abfaad0
Jul 28 18:01:06.402391 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:06.402374    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:01:06.517927 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:06.517897    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 18:01:07.052911 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.052874    2564 apiserver.go:52] "Watching apiserver"
Jul 28 18:01:07.062087 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.062054    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 18:01:07.064637 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.064604    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m","openshift-cluster-node-tuning-operator/tuned-c47lt","openshift-dns/node-resolver-4b5g6","openshift-image-registry/node-ca-xfhqj","openshift-multus/network-metrics-daemon-trhfb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal","openshift-multus/multus-additional-cni-plugins-6vpvk","openshift-multus/multus-zpkp7","openshift-network-diagnostics/network-check-target-tkx4l","openshift-network-operator/iptables-alerter-tnh5v","openshift-ovn-kubernetes/ovnkube-node-z2m7g","kube-system/global-pull-secret-syncer-pfrhs","kube-system/konnectivity-agent-pq64d"]
Jul 28 18:01:07.066350 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.066331    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:07.066472 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.066424    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:07.067797 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.067775    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:07.067923 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.067865    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:07.069203 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.069182    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.070900 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.070859    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4b5g6"
Jul 28 18:01:07.072213 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.072142    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 18:01:07.072213 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.072182    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zbxrc\""
Jul 28 18:01:07.072415 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.072369    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 18:01:07.072529 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.072510    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xfhqj"
Jul 28 18:01:07.073045 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.073028    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 18:01:07.073479 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.073428    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 18:01:07.073607 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.073555    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 18:01:07.073824 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.073724    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d68mp\""
Jul 28 18:01:07.075092 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.074704    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:07.075092 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.074801    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:07.075246 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.075106    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 18:01:07.075246 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.075218    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 18:01:07.075704 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.075460    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 18:01:07.076920 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.075850    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7psn4\""
Jul 28 18:01:07.078264 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.078244    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.079561 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.079539    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pq64d"
Jul 28 18:01:07.080933 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.080909    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 18:01:07.081213 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.081195    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 18:01:07.081770 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.081204    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 18:01:07.081770 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.081437    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 18:01:07.081770 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.081442    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.081770 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.081594    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gjvvz\""
Jul 28 18:01:07.082128 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.082110    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 18:01:07.082381 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.082366    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gxjfd\""
Jul 28 18:01:07.082438 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.082381    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 18:01:07.084054 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.084034    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.084342 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.084319    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 18:01:07.084454 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.084432    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 18:01:07.084539 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.084434    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sv8m7\""
Jul 28 18:01:07.084635 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.084566    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 18:01:07.085720 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.085702    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tnh5v"
Jul 28 18:01:07.086481 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086458    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret\") pod \"global-pull-secret-syncer-pfrhs\" (UID: \"e69399ee-0e2d-477a-857d-79c7c39543a0\") " pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:07.086580 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086484    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-run\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.086580 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086536    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-lib-modules\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.086580 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086568    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-tuned\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.086748 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086601    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mphq\" (UniqueName: \"kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq\") pod \"network-check-target-tkx4l\" (UID: \"7485a88f-1174-493b-bef2-159f0b4c9646\") " pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:07.086748 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086628    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-var-lib-kubelet\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.086748 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086657    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-modprobe-d\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.086748 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086681    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-sysctl-d\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.086748 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086706    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-host\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.086748 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086729    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e69399ee-0e2d-477a-857d-79c7c39543a0-dbus\") pod \"global-pull-secret-syncer-pfrhs\" (UID: \"e69399ee-0e2d-477a-857d-79c7c39543a0\") " pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:07.087162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086755    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-sysconfig\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.087162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086780    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c73d0621-b609-4cea-a1c9-bf66bd5c0156-hosts-file\") pod \"node-resolver-4b5g6\" (UID: \"c73d0621-b609-4cea-a1c9-bf66bd5c0156\") " pod="openshift-dns/node-resolver-4b5g6"
Jul 28 18:01:07.087162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086803    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fd60e32-0794-41d2-ad51-83789cb97b4c-host\") pod \"node-ca-xfhqj\" (UID: \"8fd60e32-0794-41d2-ad51-83789cb97b4c\") " pod="openshift-image-registry/node-ca-xfhqj"
Jul 28 18:01:07.087162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086827    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs\") pod \"network-metrics-daemon-trhfb\" (UID: \"185c4451-1106-4555-855c-4f48a9348827\") " pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:07.087162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086851    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9cbf\" (UniqueName: \"kubernetes.io/projected/185c4451-1106-4555-855c-4f48a9348827-kube-api-access-z9cbf\") pod \"network-metrics-daemon-trhfb\" (UID: \"185c4451-1106-4555-855c-4f48a9348827\") " pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:07.087162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086872    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8fd60e32-0794-41d2-ad51-83789cb97b4c-serviceca\") pod \"node-ca-xfhqj\" (UID: \"8fd60e32-0794-41d2-ad51-83789cb97b4c\") " pod="openshift-image-registry/node-ca-xfhqj"
Jul 28 18:01:07.087162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086913    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cskxq\""
Jul 28 18:01:07.087162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086923    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-sysctl-conf\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.087162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086946    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-sys\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.087162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.086987    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-systemd\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.087162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.087018    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25def06c-00d4-415c-b06c-6bab3cc1d690-tmp\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.087162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.087046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4btxl\" (UniqueName: \"kubernetes.io/projected/c73d0621-b609-4cea-a1c9-bf66bd5c0156-kube-api-access-4btxl\") pod \"node-resolver-4b5g6\" (UID: \"c73d0621-b609-4cea-a1c9-bf66bd5c0156\") " pod="openshift-dns/node-resolver-4b5g6"
Jul 28 18:01:07.087162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.087075    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjvnx\" (UniqueName: \"kubernetes.io/projected/8fd60e32-0794-41d2-ad51-83789cb97b4c-kube-api-access-sjvnx\") pod \"node-ca-xfhqj\" (UID: \"8fd60e32-0794-41d2-ad51-83789cb97b4c\") " pod="openshift-image-registry/node-ca-xfhqj"
Jul 28 18:01:07.087162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.087104    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-kubernetes\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.087162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.087134    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsr68\" (UniqueName: \"kubernetes.io/projected/25def06c-00d4-415c-b06c-6bab3cc1d690-kube-api-access-fsr68\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.087921 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.087184    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 18:01:07.087921 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.087208    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73d0621-b609-4cea-a1c9-bf66bd5c0156-tmp-dir\") pod \"node-resolver-4b5g6\" (UID: \"c73d0621-b609-4cea-a1c9-bf66bd5c0156\") " pod="openshift-dns/node-resolver-4b5g6"
Jul 28 18:01:07.087921 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.087211    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 18:01:07.087921 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.087248    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e69399ee-0e2d-477a-857d-79c7c39543a0-kubelet-config\") pod \"global-pull-secret-syncer-pfrhs\" (UID: \"e69399ee-0e2d-477a-857d-79c7c39543a0\") " pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:07.087921 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.087650    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.088333 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.088312    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5pbqg\""
Jul 28 18:01:07.088448 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.088424    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 18:01:07.088557 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.088537    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 18:01:07.088615 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.088545    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 18:01:07.088683 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.088667    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 18:01:07.090487 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.090300    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 18:01:07.090487 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.090434    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 18:01:07.090981 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.090964    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 18:01:07.091977 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.091957    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 18:01:07.091977 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.091969    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 18:01:07.091977 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.091976    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 18:01:07.092164 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.092009    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p76jd\""
Jul 28 18:01:07.111233 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.111211    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v8wc5"
Jul 28 18:01:07.136025 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.135999    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v8wc5"
Jul 28 18:01:07.173727 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.173701    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 18:01:07.187444 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187412    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8fd60e32-0794-41d2-ad51-83789cb97b4c-serviceca\") pod \"node-ca-xfhqj\" (UID: \"8fd60e32-0794-41d2-ad51-83789cb97b4c\") " pod="openshift-image-registry/node-ca-xfhqj"
Jul 28 18:01:07.187597 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187467    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7vlv\" (UniqueName: \"kubernetes.io/projected/d4012634-2038-408b-972e-c205bb8d4e74-kube-api-access-t7vlv\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.187597 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187500    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-kubelet\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.187597 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187528    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-var-lib-openvswitch\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.187597 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187572    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-run-openvswitch\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.187829 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187611    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.187829 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187633    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-cni-bin\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.187829 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187657    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-ovnkube-config\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.187829 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187685    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-multus-conf-dir\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.187829 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25b8eefe-670b-44d6-bf1b-18444642d5b7-host-slash\") pod \"iptables-alerter-tnh5v\" (UID: \"25b8eefe-670b-44d6-bf1b-18444642d5b7\") " pod="openshift-network-operator/iptables-alerter-tnh5v"
Jul 28 18:01:07.187829 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187744    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73d0621-b609-4cea-a1c9-bf66bd5c0156-tmp-dir\") pod \"node-resolver-4b5g6\" (UID: \"c73d0621-b609-4cea-a1c9-bf66bd5c0156\") " pod="openshift-dns/node-resolver-4b5g6"
Jul 28 18:01:07.187829 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187774    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f4b2f27-907c-4aa3-af81-d2b4a221e24d-konnectivity-ca\") pod \"konnectivity-agent-pq64d\" (UID: \"6f4b2f27-907c-4aa3-af81-d2b4a221e24d\") " pod="kube-system/konnectivity-agent-pq64d"
Jul 28 18:01:07.187829 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187800    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-log-socket\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.187829 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187825    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-cni-netd\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.188265 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187868    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-ovn-node-metrics-cert\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.188265 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187915    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret\") pod \"global-pull-secret-syncer-pfrhs\" (UID: \"e69399ee-0e2d-477a-857d-79c7c39543a0\") " pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:07.188265 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187967    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-cnibin\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.188265 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.187995    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-multus-socket-dir-parent\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.188265 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188029    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8fd60e32-0794-41d2-ad51-83789cb97b4c-serviceca\") pod \"node-ca-xfhqj\" (UID: \"8fd60e32-0794-41d2-ad51-83789cb97b4c\") " pod="openshift-image-registry/node-ca-xfhqj"
Jul 28 18:01:07.188265 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188031    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-sys-fs\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.188265 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188072    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a63ca3e-bad2-4221-8493-38c09b9ad532-cnibin\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.188265 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188085    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73d0621-b609-4cea-a1c9-bf66bd5c0156-tmp-dir\") pod \"node-resolver-4b5g6\" (UID: \"c73d0621-b609-4cea-a1c9-bf66bd5c0156\") " pod="openshift-dns/node-resolver-4b5g6"
Jul 28 18:01:07.188265 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188102    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-env-overrides\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.188265 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.188117    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 18:01:07.188265 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-host\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.188265 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188151    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-registration-dir\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.188265 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188181    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-run-k8s-cni-cncf-io\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.188265 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.188201    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret podName:e69399ee-0e2d-477a-857d-79c7c39543a0 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:07.688178525 +0000 UTC m=+3.003680638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret") pod "global-pull-secret-syncer-pfrhs" (UID: "e69399ee-0e2d-477a-857d-79c7c39543a0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 18:01:07.188265 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188206    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-host\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.188265 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188248    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkpgl\" (UniqueName: \"kubernetes.io/projected/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-kube-api-access-qkpgl\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188279    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a63ca3e-bad2-4221-8493-38c09b9ad532-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188307    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e69399ee-0e2d-477a-857d-79c7c39543a0-dbus\") pod \"global-pull-secret-syncer-pfrhs\" (UID: \"e69399ee-0e2d-477a-857d-79c7c39543a0\") " pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188334    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-sysconfig\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188358    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fd60e32-0794-41d2-ad51-83789cb97b4c-host\") pod \"node-ca-xfhqj\" (UID: \"8fd60e32-0794-41d2-ad51-83789cb97b4c\") " pod="openshift-image-registry/node-ca-xfhqj"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188385    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9cbf\" (UniqueName: \"kubernetes.io/projected/185c4451-1106-4555-855c-4f48a9348827-kube-api-access-z9cbf\") pod \"network-metrics-daemon-trhfb\" (UID: \"185c4451-1106-4555-855c-4f48a9348827\") " pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188413    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-device-dir\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188421    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-sysconfig\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188424    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fd60e32-0794-41d2-ad51-83789cb97b4c-host\") pod \"node-ca-xfhqj\" (UID: \"8fd60e32-0794-41d2-ad51-83789cb97b4c\") " pod="openshift-image-registry/node-ca-xfhqj"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-var-lib-cni-bin\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188467    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a63ca3e-bad2-4221-8493-38c09b9ad532-system-cni-dir\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188488    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e69399ee-0e2d-477a-857d-79c7c39543a0-dbus\") pod \"global-pull-secret-syncer-pfrhs\" (UID: \"e69399ee-0e2d-477a-857d-79c7c39543a0\") " pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188493    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p75x2\" (UniqueName: \"kubernetes.io/projected/7a63ca3e-bad2-4221-8493-38c09b9ad532-kube-api-access-p75x2\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188538    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-cni-binary-copy\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188562    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-run-netns\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188586    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-var-lib-kubelet\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188630    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25b8eefe-670b-44d6-bf1b-18444642d5b7-iptables-alerter-script\") pod \"iptables-alerter-tnh5v\" (UID: \"25b8eefe-670b-44d6-bf1b-18444642d5b7\") " pod="openshift-network-operator/iptables-alerter-tnh5v"
Jul 28 18:01:07.189063 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188663    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-node-log\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188697    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-run-ovn-kubernetes\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188748    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-sysctl-conf\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188830    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-sys\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188866    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-hostroot\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188895    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7a63ca3e-bad2-4221-8493-38c09b9ad532-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188915    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-sysctl-conf\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188906    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-sys\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188928    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl8qg\" (UniqueName: \"kubernetes.io/projected/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-kube-api-access-jl8qg\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.188964    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-systemd\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189006    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25def06c-00d4-415c-b06c-6bab3cc1d690-tmp\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189039    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-systemd\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189088    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4btxl\" (UniqueName: \"kubernetes.io/projected/c73d0621-b609-4cea-a1c9-bf66bd5c0156-kube-api-access-4btxl\") pod \"node-resolver-4b5g6\" (UID: \"c73d0621-b609-4cea-a1c9-bf66bd5c0156\") " pod="openshift-dns/node-resolver-4b5g6"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjvnx\" (UniqueName: \"kubernetes.io/projected/8fd60e32-0794-41d2-ad51-83789cb97b4c-kube-api-access-sjvnx\") pod \"node-ca-xfhqj\" (UID: \"8fd60e32-0794-41d2-ad51-83789cb97b4c\") " pod="openshift-image-registry/node-ca-xfhqj"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189207    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-run-netns\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189235    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189281    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-kubernetes\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.189999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsr68\" (UniqueName: \"kubernetes.io/projected/25def06c-00d4-415c-b06c-6bab3cc1d690-kube-api-access-fsr68\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189358    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189378    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-socket-dir\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189404    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-etc-selinux\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-os-release\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189465    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-run-multus-certs\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189487    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-kubernetes\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189490    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f4b2f27-907c-4aa3-af81-d2b4a221e24d-agent-certs\") pod \"konnectivity-agent-pq64d\" (UID: \"6f4b2f27-907c-4aa3-af81-d2b4a221e24d\") " pod="kube-system/konnectivity-agent-pq64d"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189526    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e69399ee-0e2d-477a-857d-79c7c39543a0-kubelet-config\") pod \"global-pull-secret-syncer-pfrhs\" (UID: \"e69399ee-0e2d-477a-857d-79c7c39543a0\") " pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189549    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-run\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189573    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-lib-modules\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189598    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-tuned\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189623    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7a63ca3e-bad2-4221-8493-38c09b9ad532-os-release\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189650    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-run-systemd\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189652    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-run\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189674    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-etc-openvswitch\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189702    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mphq\" (UniqueName: \"kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq\") pod \"network-check-target-tkx4l\" (UID: \"7485a88f-1174-493b-bef2-159f0b4c9646\") " pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:07.190759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189714    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e69399ee-0e2d-477a-857d-79c7c39543a0-kubelet-config\") pod \"global-pull-secret-syncer-pfrhs\" (UID: \"e69399ee-0e2d-477a-857d-79c7c39543a0\") " pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189729    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-var-lib-kubelet\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189755    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-var-lib-cni-multus\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189773    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-lib-modules\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189784    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-etc-kubernetes\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189808    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a63ca3e-bad2-4221-8493-38c09b9ad532-cni-binary-copy\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189833    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7a63ca3e-bad2-4221-8493-38c09b9ad532-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189845    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-var-lib-kubelet\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189895    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-slash\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189923    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-ovnkube-script-lib\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189948    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-modprobe-d\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189970    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-sysctl-d\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.189991    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-system-cni-dir\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.190024    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-multus-cni-dir\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.190049    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-multus-daemon-config\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.190071    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-systemd-units\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.190076    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-modprobe-d\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.191567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.190092    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-run-ovn\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.192403 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.190124    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c73d0621-b609-4cea-a1c9-bf66bd5c0156-hosts-file\") pod \"node-resolver-4b5g6\" (UID: \"c73d0621-b609-4cea-a1c9-bf66bd5c0156\") " pod="openshift-dns/node-resolver-4b5g6"
Jul 28 18:01:07.192403 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.190150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs\") pod \"network-metrics-daemon-trhfb\" (UID: \"185c4451-1106-4555-855c-4f48a9348827\") " pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:07.192403 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.190164    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-sysctl-d\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.192403 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.190180    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqf4p\" (UniqueName: \"kubernetes.io/projected/25b8eefe-670b-44d6-bf1b-18444642d5b7-kube-api-access-xqf4p\") pod \"iptables-alerter-tnh5v\" (UID: \"25b8eefe-670b-44d6-bf1b-18444642d5b7\") " pod="openshift-network-operator/iptables-alerter-tnh5v"
Jul 28 18:01:07.192403 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.190221    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c73d0621-b609-4cea-a1c9-bf66bd5c0156-hosts-file\") pod \"node-resolver-4b5g6\" (UID: \"c73d0621-b609-4cea-a1c9-bf66bd5c0156\") " pod="openshift-dns/node-resolver-4b5g6"
Jul 28 18:01:07.192403 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.190277    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 18:01:07.192403 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.190350    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs podName:185c4451-1106-4555-855c-4f48a9348827 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:07.690332767 +0000 UTC m=+3.005834874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs") pod "network-metrics-daemon-trhfb" (UID: "185c4451-1106-4555-855c-4f48a9348827") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 18:01:07.193056 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.193037    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/25def06c-00d4-415c-b06c-6bab3cc1d690-etc-tuned\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.193164 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.193080    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25def06c-00d4-415c-b06c-6bab3cc1d690-tmp\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.196197 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.196173    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 18:01:07.196294 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.196203    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 18:01:07.196294 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.196220    2564 projected.go:196] Error preparing data for projected volume kube-api-access-5mphq for pod openshift-network-diagnostics/network-check-target-tkx4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 18:01:07.196294 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.196288    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq podName:7485a88f-1174-493b-bef2-159f0b4c9646 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:07.696271166 +0000 UTC m=+3.011773275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5mphq" (UniqueName: "kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq") pod "network-check-target-tkx4l" (UID: "7485a88f-1174-493b-bef2-159f0b4c9646") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 18:01:07.198372 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.198346    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9cbf\" (UniqueName: \"kubernetes.io/projected/185c4451-1106-4555-855c-4f48a9348827-kube-api-access-z9cbf\") pod \"network-metrics-daemon-trhfb\" (UID: \"185c4451-1106-4555-855c-4f48a9348827\") " pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:07.198764 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.198739    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjvnx\" (UniqueName: \"kubernetes.io/projected/8fd60e32-0794-41d2-ad51-83789cb97b4c-kube-api-access-sjvnx\") pod \"node-ca-xfhqj\" (UID: \"8fd60e32-0794-41d2-ad51-83789cb97b4c\") " pod="openshift-image-registry/node-ca-xfhqj"
Jul 28 18:01:07.198878 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.198857    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4btxl\" (UniqueName: \"kubernetes.io/projected/c73d0621-b609-4cea-a1c9-bf66bd5c0156-kube-api-access-4btxl\") pod \"node-resolver-4b5g6\" (UID: \"c73d0621-b609-4cea-a1c9-bf66bd5c0156\") " pod="openshift-dns/node-resolver-4b5g6"
Jul 28 18:01:07.199087 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.199073    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsr68\" (UniqueName: \"kubernetes.io/projected/25def06c-00d4-415c-b06c-6bab3cc1d690-kube-api-access-fsr68\") pod \"tuned-c47lt\" (UID: \"25def06c-00d4-415c-b06c-6bab3cc1d690\") " pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.216159 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.216114    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal" event={"ID":"c6226ec06bb69fa100018d0ee4c073bb","Type":"ContainerStarted","Data":"88f56661da00fa416bc13d4e11524842a8b4641f4fe5db1e13ada7662abfaad0"}
Jul 28 18:01:07.238977 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.238948    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 18:01:07.290597 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.290566    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-multus-conf-dir\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.290768 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.290650    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-multus-conf-dir\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.290768 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.290701    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25b8eefe-670b-44d6-bf1b-18444642d5b7-host-slash\") pod \"iptables-alerter-tnh5v\" (UID: \"25b8eefe-670b-44d6-bf1b-18444642d5b7\") " pod="openshift-network-operator/iptables-alerter-tnh5v"
Jul 28 18:01:07.290768 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.290732    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f4b2f27-907c-4aa3-af81-d2b4a221e24d-konnectivity-ca\") pod \"konnectivity-agent-pq64d\" (UID: \"6f4b2f27-907c-4aa3-af81-d2b4a221e24d\") " pod="kube-system/konnectivity-agent-pq64d"
Jul 28 18:01:07.290940 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.290840    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25b8eefe-670b-44d6-bf1b-18444642d5b7-host-slash\") pod \"iptables-alerter-tnh5v\" (UID: \"25b8eefe-670b-44d6-bf1b-18444642d5b7\") " pod="openshift-network-operator/iptables-alerter-tnh5v"
Jul 28 18:01:07.291023 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.290981    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-log-socket\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.291243 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-log-socket\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.291369 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-cni-netd\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.291369 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291283    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-ovn-node-metrics-cert\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.291369 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291320    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-cnibin\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.291369 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291329    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f4b2f27-907c-4aa3-af81-d2b4a221e24d-konnectivity-ca\") pod \"konnectivity-agent-pq64d\" (UID: \"6f4b2f27-907c-4aa3-af81-d2b4a221e24d\") " pod="kube-system/konnectivity-agent-pq64d"
Jul 28 18:01:07.291369 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291346    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-multus-socket-dir-parent\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.291369 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291352    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-cni-netd\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.291687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291391    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-sys-fs\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.291687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291406    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-multus-socket-dir-parent\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.291687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291419    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a63ca3e-bad2-4221-8493-38c09b9ad532-cnibin\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.291687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291465    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-cnibin\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.291687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291465    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-env-overrides\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.291687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291505    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-registration-dir\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.291687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291533    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-run-k8s-cni-cncf-io\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.291687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291557    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkpgl\" (UniqueName: \"kubernetes.io/projected/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-kube-api-access-qkpgl\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.291687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291583    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a63ca3e-bad2-4221-8493-38c09b9ad532-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.291687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291595    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-sys-fs\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.291687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291613    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-device-dir\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.291687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291617    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-registration-dir\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.291687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291630    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a63ca3e-bad2-4221-8493-38c09b9ad532-cnibin\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.291687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291652    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-var-lib-cni-bin\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.291687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291659    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-run-k8s-cni-cncf-io\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.291687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291681    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a63ca3e-bad2-4221-8493-38c09b9ad532-system-cni-dir\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p75x2\" (UniqueName: \"kubernetes.io/projected/7a63ca3e-bad2-4221-8493-38c09b9ad532-kube-api-access-p75x2\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291733    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-cni-binary-copy\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291719    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-var-lib-cni-bin\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291757    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-run-netns\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291781    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a63ca3e-bad2-4221-8493-38c09b9ad532-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291803    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-run-netns\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291844    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-device-dir\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291902    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-var-lib-kubelet\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291931    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25b8eefe-670b-44d6-bf1b-18444642d5b7-iptables-alerter-script\") pod \"iptables-alerter-tnh5v\" (UID: \"25b8eefe-670b-44d6-bf1b-18444642d5b7\") " pod="openshift-network-operator/iptables-alerter-tnh5v"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291945    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-var-lib-kubelet\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291760    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a63ca3e-bad2-4221-8493-38c09b9ad532-system-cni-dir\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291994    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-node-log\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.291951    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-node-log\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292022    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-env-overrides\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292037    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-run-ovn-kubernetes\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292062    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-hostroot\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292105    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-hostroot\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.292558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292117    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-run-ovn-kubernetes\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7a63ca3e-bad2-4221-8493-38c09b9ad532-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292184    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl8qg\" (UniqueName: \"kubernetes.io/projected/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-kube-api-access-jl8qg\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292215    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-run-netns\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292242    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292270    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-socket-dir\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292294    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-etc-selinux\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292320    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-os-release\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292343    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-run-multus-certs\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292366    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f4b2f27-907c-4aa3-af81-d2b4a221e24d-agent-certs\") pod \"konnectivity-agent-pq64d\" (UID: \"6f4b2f27-907c-4aa3-af81-d2b4a221e24d\") " pod="kube-system/konnectivity-agent-pq64d"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292417    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7a63ca3e-bad2-4221-8493-38c09b9ad532-os-release\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-run-systemd\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292445    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-cni-binary-copy\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292467    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-etc-openvswitch\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292508    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-var-lib-cni-multus\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292513    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25b8eefe-670b-44d6-bf1b-18444642d5b7-iptables-alerter-script\") pod \"iptables-alerter-tnh5v\" (UID: \"25b8eefe-670b-44d6-bf1b-18444642d5b7\") " pod="openshift-network-operator/iptables-alerter-tnh5v"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292528    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-os-release\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.293486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292507    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-run-netns\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.294314 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.292545    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.294399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.294364    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-ovn-node-metrics-cert\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.294478 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.294423    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-etc-kubernetes\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.294538 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.294483    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a63ca3e-bad2-4221-8493-38c09b9ad532-cni-binary-copy\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.294538 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.294524    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7a63ca3e-bad2-4221-8493-38c09b9ad532-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.294644 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.294564    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-slash\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.294969 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.294944    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-etc-selinux\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.295078 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.295010    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-ovnkube-script-lib\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.295078 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.295050    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-system-cni-dir\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.295194 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.295082    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-multus-cni-dir\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.295194 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.295114    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-multus-daemon-config\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.295194 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.295142    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-systemd-units\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.295194 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.295174    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-run-ovn\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.295397 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.295210    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-socket-dir\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.295459 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.295419    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-run-ovn\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.295515 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.295485    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7a63ca3e-bad2-4221-8493-38c09b9ad532-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.295515 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.295489    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-run-multus-certs\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.295653 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.295630    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-system-cni-dir\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.295723 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.295685    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7a63ca3e-bad2-4221-8493-38c09b9ad532-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.295784 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.295723    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-multus-cni-dir\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.296064 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.295875    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-systemd-units\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.296064 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.295937    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-run-systemd\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.296064 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296005    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7a63ca3e-bad2-4221-8493-38c09b9ad532-os-release\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.296064 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296049    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-host-var-lib-cni-multus\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.296310 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296067    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-ovnkube-script-lib\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.296310 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296094    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-etc-openvswitch\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.296310 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296141    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-slash\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.296310 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296225    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqf4p\" (UniqueName: \"kubernetes.io/projected/25b8eefe-670b-44d6-bf1b-18444642d5b7-kube-api-access-xqf4p\") pod \"iptables-alerter-tnh5v\" (UID: \"25b8eefe-670b-44d6-bf1b-18444642d5b7\") " pod="openshift-network-operator/iptables-alerter-tnh5v"
Jul 28 18:01:07.296310 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296253    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-multus-daemon-config\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.296310 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296258    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7vlv\" (UniqueName: \"kubernetes.io/projected/d4012634-2038-408b-972e-c205bb8d4e74-kube-api-access-t7vlv\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.296655 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296314    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-kubelet\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.296655 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296350    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-var-lib-openvswitch\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.296655 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296389    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-run-openvswitch\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.296655 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296423    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.296655 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296453    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-cni-bin\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.296655 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-ovnkube-config\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.296655 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296571    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-etc-kubernetes\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.297037 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296763    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-kubelet\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.297037 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296825    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-var-lib-openvswitch\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.297037 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296877    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-run-openvswitch\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.297037 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.296964    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d4012634-2038-408b-972e-c205bb8d4e74-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.297037 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.297013    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-host-cni-bin\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.297286 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.297089    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f4b2f27-907c-4aa3-af81-d2b4a221e24d-agent-certs\") pod \"konnectivity-agent-pq64d\" (UID: \"6f4b2f27-907c-4aa3-af81-d2b4a221e24d\") " pod="kube-system/konnectivity-agent-pq64d"
Jul 28 18:01:07.297286 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.297110    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a63ca3e-bad2-4221-8493-38c09b9ad532-cni-binary-copy\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.297286 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.297131    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-ovnkube-config\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.302311 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.302286    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl8qg\" (UniqueName: \"kubernetes.io/projected/cc9e0734-f0ec-4eab-93b9-feaf9eccffbd-kube-api-access-jl8qg\") pod \"ovnkube-node-z2m7g\" (UID: \"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.302411 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.302400    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p75x2\" (UniqueName: \"kubernetes.io/projected/7a63ca3e-bad2-4221-8493-38c09b9ad532-kube-api-access-p75x2\") pod \"multus-additional-cni-plugins-6vpvk\" (UID: \"7a63ca3e-bad2-4221-8493-38c09b9ad532\") " pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.302737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.302714    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkpgl\" (UniqueName: \"kubernetes.io/projected/7bb978d6-b37d-40bf-86ca-937cd0ee1f03-kube-api-access-qkpgl\") pod \"multus-zpkp7\" (UID: \"7bb978d6-b37d-40bf-86ca-937cd0ee1f03\") " pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.305775 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.305728    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7vlv\" (UniqueName: \"kubernetes.io/projected/d4012634-2038-408b-972e-c205bb8d4e74-kube-api-access-t7vlv\") pod \"aws-ebs-csi-driver-node-gst8m\" (UID: \"d4012634-2038-408b-972e-c205bb8d4e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.306078 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.306061    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqf4p\" (UniqueName: \"kubernetes.io/projected/25b8eefe-670b-44d6-bf1b-18444642d5b7-kube-api-access-xqf4p\") pod \"iptables-alerter-tnh5v\" (UID: \"25b8eefe-670b-44d6-bf1b-18444642d5b7\") " pod="openshift-network-operator/iptables-alerter-tnh5v"
Jul 28 18:01:07.384004 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.383961    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c47lt"
Jul 28 18:01:07.394102 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.394070    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4b5g6"
Jul 28 18:01:07.402162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.402140    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xfhqj"
Jul 28 18:01:07.407253 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.407226    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zpkp7"
Jul 28 18:01:07.414052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.414032    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pq64d"
Jul 28 18:01:07.420898 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.420867    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m"
Jul 28 18:01:07.430874 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.430854    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6vpvk"
Jul 28 18:01:07.438704 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.438681    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tnh5v"
Jul 28 18:01:07.444823 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.444804    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:07.673752 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:07.673699    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb978d6_b37d_40bf_86ca_937cd0ee1f03.slice/crio-da16d295348d2e4aa7698279d76d2b61a53b3c65bd3b2acc086db499b57b8acb WatchSource:0}: Error finding container da16d295348d2e4aa7698279d76d2b61a53b3c65bd3b2acc086db499b57b8acb: Status 404 returned error can't find the container with id da16d295348d2e4aa7698279d76d2b61a53b3c65bd3b2acc086db499b57b8acb
Jul 28 18:01:07.685833 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:07.685808    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9e0734_f0ec_4eab_93b9_feaf9eccffbd.slice/crio-e73840abf4195a0e9f51c85b3dd17e04de356364e1cdba55094ed9f11a423d0b WatchSource:0}: Error finding container e73840abf4195a0e9f51c85b3dd17e04de356364e1cdba55094ed9f11a423d0b: Status 404 returned error can't find the container with id e73840abf4195a0e9f51c85b3dd17e04de356364e1cdba55094ed9f11a423d0b
Jul 28 18:01:07.688432 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:07.688407    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73d0621_b609_4cea_a1c9_bf66bd5c0156.slice/crio-35b775828fec4d6b5f95af2fb5d1570654bad4c53bdfff8dce2c4b9e0cf5c501 WatchSource:0}: Error finding container 35b775828fec4d6b5f95af2fb5d1570654bad4c53bdfff8dce2c4b9e0cf5c501: Status 404 returned error can't find the container with id 35b775828fec4d6b5f95af2fb5d1570654bad4c53bdfff8dce2c4b9e0cf5c501
Jul 28 18:01:07.689260 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:07.689225    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd60e32_0794_41d2_ad51_83789cb97b4c.slice/crio-ee96647bb4fd6fd736d3056874aa9bf85d94c1040641425b77a9dc3aa192e60e WatchSource:0}: Error finding container ee96647bb4fd6fd736d3056874aa9bf85d94c1040641425b77a9dc3aa192e60e: Status 404 returned error can't find the container with id ee96647bb4fd6fd736d3056874aa9bf85d94c1040641425b77a9dc3aa192e60e
Jul 28 18:01:07.690363 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:07.690322    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a63ca3e_bad2_4221_8493_38c09b9ad532.slice/crio-da84de2742bbe7d80f89920d0eb63168e1a4a5fe7df1deaf616f0fdd88936b33 WatchSource:0}: Error finding container da84de2742bbe7d80f89920d0eb63168e1a4a5fe7df1deaf616f0fdd88936b33: Status 404 returned error can't find the container with id da84de2742bbe7d80f89920d0eb63168e1a4a5fe7df1deaf616f0fdd88936b33
Jul 28 18:01:07.690971 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:07.690930    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4b2f27_907c_4aa3_af81_d2b4a221e24d.slice/crio-2cf015751ed7cdab6b97a6a9f021fbe8f33a63f5b457277f23639a291764c546 WatchSource:0}: Error finding container 2cf015751ed7cdab6b97a6a9f021fbe8f33a63f5b457277f23639a291764c546: Status 404 returned error can't find the container with id 2cf015751ed7cdab6b97a6a9f021fbe8f33a63f5b457277f23639a291764c546
Jul 28 18:01:07.692512 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:07.692491    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b8eefe_670b_44d6_bf1b_18444642d5b7.slice/crio-f6bbd4ea6054a8bb1d152c8d39fb9a8d83b34c30d6c1d14df8684464f6f4004a WatchSource:0}: Error finding container f6bbd4ea6054a8bb1d152c8d39fb9a8d83b34c30d6c1d14df8684464f6f4004a: Status 404 returned error can't find the container with id f6bbd4ea6054a8bb1d152c8d39fb9a8d83b34c30d6c1d14df8684464f6f4004a
Jul 28 18:01:07.693246 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:07.693168    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25def06c_00d4_415c_b06c_6bab3cc1d690.slice/crio-6a5d061a868b6cd0f715e11bfc6c1728feb42f39529b7d5eacfda7f78ac37e60 WatchSource:0}: Error finding container 6a5d061a868b6cd0f715e11bfc6c1728feb42f39529b7d5eacfda7f78ac37e60: Status 404 returned error can't find the container with id 6a5d061a868b6cd0f715e11bfc6c1728feb42f39529b7d5eacfda7f78ac37e60
Jul 28 18:01:07.694246 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:07.694199    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4012634_2038_408b_972e_c205bb8d4e74.slice/crio-04a3a8f6b13d0f43e5830dd922f7f9145fc284c21ae92ab6fdbab21a9fbe51f8 WatchSource:0}: Error finding container 04a3a8f6b13d0f43e5830dd922f7f9145fc284c21ae92ab6fdbab21a9fbe51f8: Status 404 returned error can't find the container with id 04a3a8f6b13d0f43e5830dd922f7f9145fc284c21ae92ab6fdbab21a9fbe51f8
Jul 28 18:01:07.700307 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.700149    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mphq\" (UniqueName: \"kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq\") pod \"network-check-target-tkx4l\" (UID: \"7485a88f-1174-493b-bef2-159f0b4c9646\") " pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:07.700307 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.700204    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs\") pod \"network-metrics-daemon-trhfb\" (UID: \"185c4451-1106-4555-855c-4f48a9348827\") " pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:07.700307 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:07.700245    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret\") pod \"global-pull-secret-syncer-pfrhs\" (UID: \"e69399ee-0e2d-477a-857d-79c7c39543a0\") " pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:07.700526 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.700376    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 18:01:07.700526 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.700434    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 18:01:07.700526 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.700378    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 18:01:07.700526 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.700439    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs podName:185c4451-1106-4555-855c-4f48a9348827 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:08.700417795 +0000 UTC m=+4.015919905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs") pod "network-metrics-daemon-trhfb" (UID: "185c4451-1106-4555-855c-4f48a9348827") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 18:01:07.700526 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.700500    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret podName:e69399ee-0e2d-477a-857d-79c7c39543a0 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:08.700482966 +0000 UTC m=+4.015985061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret") pod "global-pull-secret-syncer-pfrhs" (UID: "e69399ee-0e2d-477a-857d-79c7c39543a0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 18:01:07.700526 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.700474    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 18:01:07.700526 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.700519    2564 projected.go:196] Error preparing data for projected volume kube-api-access-5mphq for pod openshift-network-diagnostics/network-check-target-tkx4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 18:01:07.701032 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:07.700556    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq podName:7485a88f-1174-493b-bef2-159f0b4c9646 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:08.700544243 +0000 UTC m=+4.016046336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mphq" (UniqueName: "kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq") pod "network-check-target-tkx4l" (UID: "7485a88f-1174-493b-bef2-159f0b4c9646") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 18:01:07.702674 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:07.702653    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da522a8a76aa2b396d458299e956b18.slice/crio-71026456a16e4488c617a662c26fdb30aa5751f522b66f33eec3775fb756a04c WatchSource:0}: Error finding container 71026456a16e4488c617a662c26fdb30aa5751f522b66f33eec3775fb756a04c: Status 404 returned error can't find the container with id 71026456a16e4488c617a662c26fdb30aa5751f522b66f33eec3775fb756a04c
Jul 28 18:01:08.138979 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.138721    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:56:07 +0000 UTC" deadline="2028-02-24 23:41:33.100901545 +0000 UTC"
Jul 28 18:01:08.138979 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.138980    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13829h40m24.961930525s"
Jul 28 18:01:08.211106 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.211029    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:08.211267 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:08.211166    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:08.221412 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.221349    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zpkp7" event={"ID":"7bb978d6-b37d-40bf-86ca-937cd0ee1f03","Type":"ContainerStarted","Data":"da16d295348d2e4aa7698279d76d2b61a53b3c65bd3b2acc086db499b57b8acb"}
Jul 28 18:01:08.223427 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.223267    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m" event={"ID":"d4012634-2038-408b-972e-c205bb8d4e74","Type":"ContainerStarted","Data":"04a3a8f6b13d0f43e5830dd922f7f9145fc284c21ae92ab6fdbab21a9fbe51f8"}
Jul 28 18:01:08.224999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.224971    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c47lt" event={"ID":"25def06c-00d4-415c-b06c-6bab3cc1d690","Type":"ContainerStarted","Data":"6a5d061a868b6cd0f715e11bfc6c1728feb42f39529b7d5eacfda7f78ac37e60"}
Jul 28 18:01:08.228152 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.228070    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tnh5v" event={"ID":"25b8eefe-670b-44d6-bf1b-18444642d5b7","Type":"ContainerStarted","Data":"f6bbd4ea6054a8bb1d152c8d39fb9a8d83b34c30d6c1d14df8684464f6f4004a"}
Jul 28 18:01:08.235431 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.234308    2564 generic.go:356] "Generic (PLEG): container finished" podID="c6226ec06bb69fa100018d0ee4c073bb" containerID="9d736c8545514b5e339f81dbd080355227364baa614b07e747a4fd32b6881575" exitCode=0
Jul 28 18:01:08.235431 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.234444    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal" event={"ID":"c6226ec06bb69fa100018d0ee4c073bb","Type":"ContainerDied","Data":"9d736c8545514b5e339f81dbd080355227364baa614b07e747a4fd32b6881575"}
Jul 28 18:01:08.239523 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.239345    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal" event={"ID":"6da522a8a76aa2b396d458299e956b18","Type":"ContainerStarted","Data":"71026456a16e4488c617a662c26fdb30aa5751f522b66f33eec3775fb756a04c"}
Jul 28 18:01:08.242608 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.242535    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pq64d" event={"ID":"6f4b2f27-907c-4aa3-af81-d2b4a221e24d","Type":"ContainerStarted","Data":"2cf015751ed7cdab6b97a6a9f021fbe8f33a63f5b457277f23639a291764c546"}
Jul 28 18:01:08.249455 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.249373    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vpvk" event={"ID":"7a63ca3e-bad2-4221-8493-38c09b9ad532","Type":"ContainerStarted","Data":"da84de2742bbe7d80f89920d0eb63168e1a4a5fe7df1deaf616f0fdd88936b33"}
Jul 28 18:01:08.255577 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.255541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xfhqj" event={"ID":"8fd60e32-0794-41d2-ad51-83789cb97b4c","Type":"ContainerStarted","Data":"ee96647bb4fd6fd736d3056874aa9bf85d94c1040641425b77a9dc3aa192e60e"}
Jul 28 18:01:08.263851 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.263768    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4b5g6" event={"ID":"c73d0621-b609-4cea-a1c9-bf66bd5c0156","Type":"ContainerStarted","Data":"35b775828fec4d6b5f95af2fb5d1570654bad4c53bdfff8dce2c4b9e0cf5c501"}
Jul 28 18:01:08.266126 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.266034    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g" event={"ID":"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd","Type":"ContainerStarted","Data":"e73840abf4195a0e9f51c85b3dd17e04de356364e1cdba55094ed9f11a423d0b"}
Jul 28 18:01:08.709977 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.708908    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mphq\" (UniqueName: \"kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq\") pod \"network-check-target-tkx4l\" (UID: \"7485a88f-1174-493b-bef2-159f0b4c9646\") " pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:08.709977 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.708972    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs\") pod \"network-metrics-daemon-trhfb\" (UID: \"185c4451-1106-4555-855c-4f48a9348827\") " pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:08.709977 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:08.709011    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret\") pod \"global-pull-secret-syncer-pfrhs\" (UID: \"e69399ee-0e2d-477a-857d-79c7c39543a0\") " pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:08.709977 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:08.709144    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 18:01:08.709977 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:08.709226    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret podName:e69399ee-0e2d-477a-857d-79c7c39543a0 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:10.709191093 +0000 UTC m=+6.024693187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret") pod "global-pull-secret-syncer-pfrhs" (UID: "e69399ee-0e2d-477a-857d-79c7c39543a0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 18:01:08.709977 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:08.709664    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 18:01:08.709977 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:08.709684    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 18:01:08.709977 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:08.709698    2564 projected.go:196] Error preparing data for projected volume kube-api-access-5mphq for pod openshift-network-diagnostics/network-check-target-tkx4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 18:01:08.709977 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:08.709745    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq podName:7485a88f-1174-493b-bef2-159f0b4c9646 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:10.709728043 +0000 UTC m=+6.025230144 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mphq" (UniqueName: "kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq") pod "network-check-target-tkx4l" (UID: "7485a88f-1174-493b-bef2-159f0b4c9646") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 18:01:08.709977 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:08.709807    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 18:01:08.709977 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:08.709841    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs podName:185c4451-1106-4555-855c-4f48a9348827 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:10.709830361 +0000 UTC m=+6.025332463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs") pod "network-metrics-daemon-trhfb" (UID: "185c4451-1106-4555-855c-4f48a9348827") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 18:01:09.139436 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:09.139320    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:56:07 +0000 UTC" deadline="2028-03-25 09:32:50.809538923 +0000 UTC"
Jul 28 18:01:09.139436 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:09.139387    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14535h31m41.670156088s"
Jul 28 18:01:09.226006 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:09.225366    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:09.226006 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:09.225399    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:09.226006 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:09.225518    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:09.226006 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:09.225627    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:09.278947 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:09.278914    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal" event={"ID":"c6226ec06bb69fa100018d0ee4c073bb","Type":"ContainerStarted","Data":"977175077f6b67c112c1f2dfe82cc69493a8cd89b09fd941f8e629c1c8223e39"}
Jul 28 18:01:09.299150 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:09.298104    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-99.ec2.internal" podStartSLOduration=3.298086445 podStartE2EDuration="3.298086445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:09.295845349 +0000 UTC m=+4.611347478" watchObservedRunningTime="2026-07-28 18:01:09.298086445 +0000 UTC m=+4.613588559"
Jul 28 18:01:10.211930 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:10.211332    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:10.211930 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:10.211505    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:10.725446 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:10.725404    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mphq\" (UniqueName: \"kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq\") pod \"network-check-target-tkx4l\" (UID: \"7485a88f-1174-493b-bef2-159f0b4c9646\") " pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:10.725625 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:10.725462    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs\") pod \"network-metrics-daemon-trhfb\" (UID: \"185c4451-1106-4555-855c-4f48a9348827\") " pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:10.725625 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:10.725497    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret\") pod \"global-pull-secret-syncer-pfrhs\" (UID: \"e69399ee-0e2d-477a-857d-79c7c39543a0\") " pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:10.725745 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:10.725635    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 18:01:10.725745 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:10.725697    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret podName:e69399ee-0e2d-477a-857d-79c7c39543a0 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:14.725679291 +0000 UTC m=+10.041181389 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret") pod "global-pull-secret-syncer-pfrhs" (UID: "e69399ee-0e2d-477a-857d-79c7c39543a0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 18:01:10.726139 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:10.726120    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 18:01:10.726226 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:10.726144    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 18:01:10.726226 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:10.726157    2564 projected.go:196] Error preparing data for projected volume kube-api-access-5mphq for pod openshift-network-diagnostics/network-check-target-tkx4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 18:01:10.726226 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:10.726201    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq podName:7485a88f-1174-493b-bef2-159f0b4c9646 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:14.726186775 +0000 UTC m=+10.041688868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mphq" (UniqueName: "kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq") pod "network-check-target-tkx4l" (UID: "7485a88f-1174-493b-bef2-159f0b4c9646") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 18:01:10.726384 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:10.726257    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 18:01:10.726384 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:10.726288    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs podName:185c4451-1106-4555-855c-4f48a9348827 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:14.726277431 +0000 UTC m=+10.041779529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs") pod "network-metrics-daemon-trhfb" (UID: "185c4451-1106-4555-855c-4f48a9348827") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 18:01:11.210592 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:11.210509    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:11.210762 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:11.210660    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:11.211069 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:11.211049    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:11.211173 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:11.211154    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:12.210688 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:12.210651    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:12.211154 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:12.210794    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:13.211487 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:13.211434    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:13.211937 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:13.211623    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:13.212256 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:13.212121    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:13.212256 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:13.212221    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:14.211203 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:14.211171    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:14.211371 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:14.211311    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:14.762352 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:14.762319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret\") pod \"global-pull-secret-syncer-pfrhs\" (UID: \"e69399ee-0e2d-477a-857d-79c7c39543a0\") " pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:14.762798 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:14.762377    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mphq\" (UniqueName: \"kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq\") pod \"network-check-target-tkx4l\" (UID: \"7485a88f-1174-493b-bef2-159f0b4c9646\") " pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:14.762798 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:14.762410    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs\") pod \"network-metrics-daemon-trhfb\" (UID: \"185c4451-1106-4555-855c-4f48a9348827\") " pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:14.762798 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:14.762518    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 18:01:14.762798 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:14.762544    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 18:01:14.762798 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:14.762565    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 18:01:14.762798 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:14.762575    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs podName:185c4451-1106-4555-855c-4f48a9348827 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:22.762556935 +0000 UTC m=+18.078059026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs") pod "network-metrics-daemon-trhfb" (UID: "185c4451-1106-4555-855c-4f48a9348827") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 18:01:14.762798 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:14.762578    2564 projected.go:196] Error preparing data for projected volume kube-api-access-5mphq for pod openshift-network-diagnostics/network-check-target-tkx4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 18:01:14.762798 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:14.762518    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 18:01:14.762798 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:14.762633    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq podName:7485a88f-1174-493b-bef2-159f0b4c9646 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:22.762615433 +0000 UTC m=+18.078117537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mphq" (UniqueName: "kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq") pod "network-check-target-tkx4l" (UID: "7485a88f-1174-493b-bef2-159f0b4c9646") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 18:01:14.762798 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:14.762665    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret podName:e69399ee-0e2d-477a-857d-79c7c39543a0 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:22.762642426 +0000 UTC m=+18.078144521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret") pod "global-pull-secret-syncer-pfrhs" (UID: "e69399ee-0e2d-477a-857d-79c7c39543a0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 18:01:15.215045 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:15.214969    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:15.215205 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:15.215109    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:15.215501 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:15.215476    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:15.215612 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:15.215590    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:16.210998 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:16.210961    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:16.211447 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:16.211095    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:17.211433 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:17.211362    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:17.211866 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:17.211490    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:17.212056 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:17.211941    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:17.212056 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:17.212022    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:18.211454 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:18.211407    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:18.211859 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:18.211538    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:19.210857 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:19.210820    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:19.211006 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:19.210837    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:19.211006 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:19.210960    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:19.211105 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:19.211011    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:20.210711 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:20.210675    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:20.211167 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:20.210798    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:21.210565 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:21.210470    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:21.210736 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:21.210604    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:21.210736 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:21.210660    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:21.211114 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:21.210760    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:22.211510 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:22.211473    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:22.212015 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:22.211596    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:22.819565 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:22.819536    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret\") pod \"global-pull-secret-syncer-pfrhs\" (UID: \"e69399ee-0e2d-477a-857d-79c7c39543a0\") " pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:22.819719 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:22.819590    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mphq\" (UniqueName: \"kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq\") pod \"network-check-target-tkx4l\" (UID: \"7485a88f-1174-493b-bef2-159f0b4c9646\") " pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:22.819719 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:22.819613    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs\") pod \"network-metrics-daemon-trhfb\" (UID: \"185c4451-1106-4555-855c-4f48a9348827\") " pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:22.819719 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:22.819669    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 18:01:22.819719 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:22.819690    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 18:01:22.819934 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:22.819728    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs podName:185c4451-1106-4555-855c-4f48a9348827 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:38.819714748 +0000 UTC m=+34.135216839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs") pod "network-metrics-daemon-trhfb" (UID: "185c4451-1106-4555-855c-4f48a9348827") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 18:01:22.819934 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:22.819744    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret podName:e69399ee-0e2d-477a-857d-79c7c39543a0 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:38.819735306 +0000 UTC m=+34.135237400 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret") pod "global-pull-secret-syncer-pfrhs" (UID: "e69399ee-0e2d-477a-857d-79c7c39543a0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 18:01:22.819934 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:22.819766    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 18:01:22.819934 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:22.819787    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 18:01:22.819934 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:22.819800    2564 projected.go:196] Error preparing data for projected volume kube-api-access-5mphq for pod openshift-network-diagnostics/network-check-target-tkx4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 18:01:22.819934 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:22.819870    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq podName:7485a88f-1174-493b-bef2-159f0b4c9646 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:38.819853502 +0000 UTC m=+34.135355607 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mphq" (UniqueName: "kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq") pod "network-check-target-tkx4l" (UID: "7485a88f-1174-493b-bef2-159f0b4c9646") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 18:01:23.210975 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:23.210893    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:23.211163 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:23.210893    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:23.211163 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:23.211028    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:23.211163 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:23.211123    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:24.211459 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:24.211425    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:24.211931 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:24.211555    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:25.211621 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:25.211582    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:25.212067 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:25.211753    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:25.212067 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:25.211818    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:25.212067 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:25.211950    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:26.211501 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:26.211466    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:26.211674 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:26.211604    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:27.211142 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.210727    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:27.211229 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.210772    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:27.211291 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:27.211269    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:27.211333 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:27.211294    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:27.323166 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.323058    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xfhqj" event={"ID":"8fd60e32-0794-41d2-ad51-83789cb97b4c","Type":"ContainerStarted","Data":"52ebb6496d445ccebd2acc2a2b827fbd557feb9655a9ccce4facc11b739182c5"}
Jul 28 18:01:27.324510 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.324489    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4b5g6" event={"ID":"c73d0621-b609-4cea-a1c9-bf66bd5c0156","Type":"ContainerStarted","Data":"8e563735fb1cf9d903112126948d1676bb80311f3d8e3f4731890795952c2dca"}
Jul 28 18:01:27.327044 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.327019    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g" event={"ID":"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd","Type":"ContainerStarted","Data":"1b9341a5d082d17d75356b1030d1182b488663d4d45cc790615178ea273e1edf"}
Jul 28 18:01:27.327142 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.327086    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g" event={"ID":"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd","Type":"ContainerStarted","Data":"00443aabcd69b0431527d9979925894680847239d0efebaa8e5982fcf4973545"}
Jul 28 18:01:27.327142 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.327130    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g" event={"ID":"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd","Type":"ContainerStarted","Data":"57bfabea36a2a7886ad3d1e1df219a737c182fb46d922c481a308f09ff4d66f4"}
Jul 28 18:01:27.328572 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.328548    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zpkp7" event={"ID":"7bb978d6-b37d-40bf-86ca-937cd0ee1f03","Type":"ContainerStarted","Data":"94d5c21b96e500d5bcb9ef2528bd7172d4675978e458f523d8b7aa2485632910"}
Jul 28 18:01:27.330143 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.330119    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m" event={"ID":"d4012634-2038-408b-972e-c205bb8d4e74","Type":"ContainerStarted","Data":"bea52e313ecf8403feaf790587fc843621acc9262c2345520d944656e259ac45"}
Jul 28 18:01:27.331433 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.331413    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c47lt" event={"ID":"25def06c-00d4-415c-b06c-6bab3cc1d690","Type":"ContainerStarted","Data":"ee3a395e9288db34a0bf6903b440dea373b3171d45a520dc9e519f95d9ec5771"}
Jul 28 18:01:27.332727 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.332698    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal" event={"ID":"6da522a8a76aa2b396d458299e956b18","Type":"ContainerStarted","Data":"d5eae56ce98c275dcc702d0ab0b333077ce55ff1f84c797ea7c531fcc33d1fa6"}
Jul 28 18:01:27.334046 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.334026    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pq64d" event={"ID":"6f4b2f27-907c-4aa3-af81-d2b4a221e24d","Type":"ContainerStarted","Data":"f8f334804c8fc08113b1bf04626c9c1ef40c495dca3be4efd75f25a4f1378ecc"}
Jul 28 18:01:27.335634 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.335592    2564 generic.go:356] "Generic (PLEG): container finished" podID="7a63ca3e-bad2-4221-8493-38c09b9ad532" containerID="c91c1c0f89be6f622462332c23c4c18f133d688d1a816e114cd7884ca19af5da" exitCode=0
Jul 28 18:01:27.335634 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.335627    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vpvk" event={"ID":"7a63ca3e-bad2-4221-8493-38c09b9ad532","Type":"ContainerDied","Data":"c91c1c0f89be6f622462332c23c4c18f133d688d1a816e114cd7884ca19af5da"}
Jul 28 18:01:27.341309 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.341273    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xfhqj" podStartSLOduration=3.23709595 podStartE2EDuration="22.341261158s" totalImagesPullingTime="19.104165208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:27.341064274 +0000 UTC m=+22.656566386" watchObservedRunningTime="2026-07-28 18:01:27.341261158 +0000 UTC m=+22.656763278"
Jul 28 18:01:27.358024 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.357975    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-99.ec2.internal" podStartSLOduration=21.357962834 podStartE2EDuration="21.357962834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:27.357214908 +0000 UTC m=+22.672717039" watchObservedRunningTime="2026-07-28 18:01:27.357962834 +0000 UTC m=+22.673464948"
Jul 28 18:01:27.411464 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.411418    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c47lt" podStartSLOduration=3.294339836 podStartE2EDuration="22.411398421s" totalImagesPullingTime="19.117058585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:27.405084043 +0000 UTC m=+22.720586155" watchObservedRunningTime="2026-07-28 18:01:27.411398421 +0000 UTC m=+22.726900535"
Jul 28 18:01:27.453649 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.453599    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4b5g6" podStartSLOduration=3.344067278 podStartE2EDuration="22.453579712s" totalImagesPullingTime="19.109512434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:27.452675757 +0000 UTC m=+22.768177904" watchObservedRunningTime="2026-07-28 18:01:27.453579712 +0000 UTC m=+22.769081827"
Jul 28 18:01:27.473502 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:27.473462    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zpkp7" podStartSLOduration=3.296868924 podStartE2EDuration="22.47344728s" totalImagesPullingTime="19.176578356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:27.473053242 +0000 UTC m=+22.788555352" watchObservedRunningTime="2026-07-28 18:01:27.47344728 +0000 UTC m=+22.788949393"
Jul 28 18:01:28.210843 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:28.210801    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:28.210981 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:28.210936    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:28.267406 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:28.267377    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 18:01:28.341718 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:28.341677    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g" event={"ID":"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd","Type":"ContainerStarted","Data":"73341e4ea91b59ea5c5633f3a29e9ada71158c5e85264cd63fc6555aa56e040c"}
Jul 28 18:01:28.342234 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:28.341776    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g" event={"ID":"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd","Type":"ContainerStarted","Data":"86b0278ed48e63dfef7947456360e3c0e61bb28db922bd3da96c3726e311ba77"}
Jul 28 18:01:28.342234 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:28.341847    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g" event={"ID":"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd","Type":"ContainerStarted","Data":"15087c2107e53b8302a1cc39113a485df150c99d214df55ea0a9c6570a42d8e3"}
Jul 28 18:01:28.344157 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:28.344125    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m" event={"ID":"d4012634-2038-408b-972e-c205bb8d4e74","Type":"ContainerStarted","Data":"d8a1f24063bd5bb9b7b90478a432a56d1ea7ff087d55aa440d8674a863e403e2"}
Jul 28 18:01:28.346283 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:28.346257    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tnh5v" event={"ID":"25b8eefe-670b-44d6-bf1b-18444642d5b7","Type":"ContainerStarted","Data":"95e6712785cede62f154dfa0747b0b2a776f7afeb8083e8877cf2873e2c40cef"}
Jul 28 18:01:28.364641 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:28.364565    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pq64d" podStartSLOduration=4.263125915 podStartE2EDuration="23.364553751s" totalImagesPullingTime="19.101427836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:27.491184325 +0000 UTC m=+22.806686441" watchObservedRunningTime="2026-07-28 18:01:28.364553751 +0000 UTC m=+23.680055864"
Jul 28 18:01:28.365127 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:28.365102    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tnh5v" podStartSLOduration=4.267090678 podStartE2EDuration="23.365094372s" totalImagesPullingTime="19.098003694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:28.364683066 +0000 UTC m=+23.680185161" watchObservedRunningTime="2026-07-28 18:01:28.365094372 +0000 UTC m=+23.680596517"
Jul 28 18:01:29.150245 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:29.150123    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T18:01:28.267393836Z","UUID":"7f0a4552-39f6-4ee9-971a-17f0d32c1937","Handler":null,"Name":"","Endpoint":""}
Jul 28 18:01:29.152068 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:29.152044    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 18:01:29.152233 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:29.152100    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 18:01:29.210546 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:29.210512    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:29.210737 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:29.210628    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:29.210737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:29.210699    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:29.210886 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:29.210826    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:29.352608 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:29.352581    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pq64d"
Jul 28 18:01:29.974894 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:29.974611    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pq64d"
Jul 28 18:01:29.975334 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:29.975317    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pq64d"
Jul 28 18:01:30.211430 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:30.211403    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:30.211598 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:30.211511    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:30.354454 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:30.354413    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g" event={"ID":"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd","Type":"ContainerStarted","Data":"82deaa9c3c775ce7fd8e29323606eeb3e1c576b9a7df74d5ead00ee084ef27c3"}
Jul 28 18:01:30.356876 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:30.356845    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m" event={"ID":"d4012634-2038-408b-972e-c205bb8d4e74","Type":"ContainerStarted","Data":"b47f32d640208c8aa1b23b4228e684d3777e5da24ad9cbe6edfd86ac2a37407e"}
Jul 28 18:01:30.357504 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:30.357481    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pq64d"
Jul 28 18:01:30.376402 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:30.376351    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gst8m" podStartSLOduration=3.901110781 podStartE2EDuration="25.376335008s" totalImagesPullingTime="21.475224227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:01:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:30.375889817 +0000 UTC m=+25.691391953" watchObservedRunningTime="2026-07-28 18:01:30.376335008 +0000 UTC m=+25.691837121"
Jul 28 18:01:31.211237 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:31.211204    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:31.211474 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:31.211204    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:31.211474 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:31.211358    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:31.211474 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:31.211428    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:32.211512 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:32.211256    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:32.212180 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:32.211517    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:32.362640 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:32.362610    2564 generic.go:356] "Generic (PLEG): container finished" podID="7a63ca3e-bad2-4221-8493-38c09b9ad532" containerID="7aae27f80034995687f9d49d001a0177fb6669f8d9527b37b53073950d9e97c8" exitCode=0
Jul 28 18:01:32.362834 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:32.362691    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vpvk" event={"ID":"7a63ca3e-bad2-4221-8493-38c09b9ad532","Type":"ContainerDied","Data":"7aae27f80034995687f9d49d001a0177fb6669f8d9527b37b53073950d9e97c8"}
Jul 28 18:01:32.367293 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:32.367266    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g" event={"ID":"cc9e0734-f0ec-4eab-93b9-feaf9eccffbd","Type":"ContainerStarted","Data":"cf3f1ba4570c6b35d63134ae85501d8ecb07b669a35e10b0165a7b022f4b5289"}
Jul 28 18:01:32.367525 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:32.367509    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:32.367621 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:32.367611    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:32.369792 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:32.368007    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:32.383534 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:32.383512    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:32.383634 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:32.383607    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:01:32.416955 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:32.416913    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g" podStartSLOduration=8.240689995 podStartE2EDuration="27.41689881s" totalImagesPullingTime="19.176208815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:32.416401724 +0000 UTC m=+27.731903836" watchObservedRunningTime="2026-07-28 18:01:32.41689881 +0000 UTC m=+27.732400923"
Jul 28 18:01:33.210916 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:33.210882    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:33.211105 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:33.210882    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:33.211105 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:33.210992    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:33.211214 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:33.211145    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:33.849662 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:33.849453    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tkx4l"]
Jul 28 18:01:33.850176 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:33.849742    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:33.850176 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:33.849859    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:33.852664 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:33.852644    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pfrhs"]
Jul 28 18:01:33.852773 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:33.852721    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:33.852855 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:33.852830    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:33.853506 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:33.853482    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-trhfb"]
Jul 28 18:01:33.853622 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:33.853575    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:33.853691 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:33.853675    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:34.373507 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:34.373477    2564 generic.go:356] "Generic (PLEG): container finished" podID="7a63ca3e-bad2-4221-8493-38c09b9ad532" containerID="1a1c0d9a7d76918416871baf971349603d4eb7dd89e6b476a701e927f7868c07" exitCode=0
Jul 28 18:01:34.373750 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:34.373534    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vpvk" event={"ID":"7a63ca3e-bad2-4221-8493-38c09b9ad532","Type":"ContainerDied","Data":"1a1c0d9a7d76918416871baf971349603d4eb7dd89e6b476a701e927f7868c07"}
Jul 28 18:01:35.212403 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:35.212371    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:35.212891 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:35.212479    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:35.378186 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:35.378152    2564 generic.go:356] "Generic (PLEG): container finished" podID="7a63ca3e-bad2-4221-8493-38c09b9ad532" containerID="dd9c61f55a0053556590b8f4e86f69774a61f45064f257bdf08e2ba8aa7c7aeb" exitCode=0
Jul 28 18:01:35.378326 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:35.378217    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vpvk" event={"ID":"7a63ca3e-bad2-4221-8493-38c09b9ad532","Type":"ContainerDied","Data":"dd9c61f55a0053556590b8f4e86f69774a61f45064f257bdf08e2ba8aa7c7aeb"}
Jul 28 18:01:36.211311 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:36.211231    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:36.211311 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:36.211268    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:36.211550 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:36.211418    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:36.211550 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:36.211516    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:37.210775 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:37.210737    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:37.211519 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:37.210938    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:38.035797 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:38.035767    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4b5g6_c73d0621-b609-4cea-a1c9-bf66bd5c0156/dns-node-resolver/0.log"
Jul 28 18:01:38.210894 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:38.210855    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:38.211328 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:38.210855    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:38.211328 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:38.210991    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:38.211328 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:38.211047    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:38.839557 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:38.839520    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mphq\" (UniqueName: \"kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq\") pod \"network-check-target-tkx4l\" (UID: \"7485a88f-1174-493b-bef2-159f0b4c9646\") " pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:38.839757 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:38.839571    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs\") pod \"network-metrics-daemon-trhfb\" (UID: \"185c4451-1106-4555-855c-4f48a9348827\") " pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:38.839757 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:38.839603    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret\") pod \"global-pull-secret-syncer-pfrhs\" (UID: \"e69399ee-0e2d-477a-857d-79c7c39543a0\") " pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:38.840114 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:38.840021    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 18:01:38.840114 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:38.840054    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 18:01:38.840114 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:38.840029    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 18:01:38.840330 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:38.840119    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret podName:e69399ee-0e2d-477a-857d-79c7c39543a0 nodeName:}" failed. No retries permitted until 2026-07-28 18:02:10.840086155 +0000 UTC m=+66.155588265 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret") pod "global-pull-secret-syncer-pfrhs" (UID: "e69399ee-0e2d-477a-857d-79c7c39543a0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 18:01:38.840330 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:38.840166    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs podName:185c4451-1106-4555-855c-4f48a9348827 nodeName:}" failed. No retries permitted until 2026-07-28 18:02:10.840128763 +0000 UTC m=+66.155630919 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs") pod "network-metrics-daemon-trhfb" (UID: "185c4451-1106-4555-855c-4f48a9348827") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 18:01:38.840330 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:38.840085    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 18:01:38.840330 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:38.840190    2564 projected.go:196] Error preparing data for projected volume kube-api-access-5mphq for pod openshift-network-diagnostics/network-check-target-tkx4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 18:01:38.840330 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:38.840233    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq podName:7485a88f-1174-493b-bef2-159f0b4c9646 nodeName:}" failed. No retries permitted until 2026-07-28 18:02:10.840215007 +0000 UTC m=+66.155717112 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mphq" (UniqueName: "kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq") pod "network-check-target-tkx4l" (UID: "7485a88f-1174-493b-bef2-159f0b4c9646") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 18:01:39.211333 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:39.211253    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:39.211940 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:39.211392    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trhfb" podUID="185c4451-1106-4555-855c-4f48a9348827"
Jul 28 18:01:39.418242 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:39.418213    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xfhqj_8fd60e32-0794-41d2-ad51-83789cb97b4c/node-ca/0.log"
Jul 28 18:01:40.210701 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.210672    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:40.210876 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.210672    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:40.210876 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:40.210777    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfrhs" podUID="e69399ee-0e2d-477a-857d-79c7c39543a0"
Jul 28 18:01:40.210876 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:40.210845    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkx4l" podUID="7485a88f-1174-493b-bef2-159f0b4c9646"
Jul 28 18:01:40.393363 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.393330    2564 generic.go:356] "Generic (PLEG): container finished" podID="7a63ca3e-bad2-4221-8493-38c09b9ad532" containerID="1fd46d228177af64ba0fc846d770db1cbc1270e09631c034788fac38d6d87cab" exitCode=0
Jul 28 18:01:40.393832 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.393375    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vpvk" event={"ID":"7a63ca3e-bad2-4221-8493-38c09b9ad532","Type":"ContainerDied","Data":"1fd46d228177af64ba0fc846d770db1cbc1270e09631c034788fac38d6d87cab"}
Jul 28 18:01:40.466626 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.466606    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-99.ec2.internal" event="NodeReady"
Jul 28 18:01:40.466734 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.466724    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 18:01:40.513528 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.513344    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rz8gm"]
Jul 28 18:01:40.534830 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.534791    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kr5wn"]
Jul 28 18:01:40.534958 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.534939    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:40.539399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.538881    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g9rrw\""
Jul 28 18:01:40.539399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.538938    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 18:01:40.539399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.539230    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 18:01:40.557178 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.557156    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rz8gm"]
Jul 28 18:01:40.557277 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.557189    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kr5wn"]
Jul 28 18:01:40.557277 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.557211    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xvdpr"]
Jul 28 18:01:40.557277 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.557242    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kr5wn"
Jul 28 18:01:40.560361 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.560337    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 18:01:40.560460 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.560363    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 18:01:40.560591 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.560576    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-f89dl\""
Jul 28 18:01:40.560844 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.560828    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 18:01:40.579919 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.579900    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xvdpr"]
Jul 28 18:01:40.580018 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.579992    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:40.583072 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.583053    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 18:01:40.583180 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.583084    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 18:01:40.583307 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.583292    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 18:01:40.583538 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.583525    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4jjmm\""
Jul 28 18:01:40.583597 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.583582    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 18:01:40.652516 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.652493    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/615e5794-659e-4726-b961-8e5be52f2272-config-volume\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:40.652516 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.652518    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:40.652659 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.652539    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsbcj\" (UniqueName: \"kubernetes.io/projected/726cdb79-f52a-4194-923b-bb594c60490d-kube-api-access-zsbcj\") pod \"ingress-canary-kr5wn\" (UID: \"726cdb79-f52a-4194-923b-bb594c60490d\") " pod="openshift-ingress-canary/ingress-canary-kr5wn"
Jul 28 18:01:40.652659 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.652559    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615e5794-659e-4726-b961-8e5be52f2272-tmp-dir\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:40.652659 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.652637    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wqpg\" (UniqueName: \"kubernetes.io/projected/615e5794-659e-4726-b961-8e5be52f2272-kube-api-access-5wqpg\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:40.652766 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.652688    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/726cdb79-f52a-4194-923b-bb594c60490d-cert\") pod \"ingress-canary-kr5wn\" (UID: \"726cdb79-f52a-4194-923b-bb594c60490d\") " pod="openshift-ingress-canary/ingress-canary-kr5wn"
Jul 28 18:01:40.753152 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.753121    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/615e5794-659e-4726-b961-8e5be52f2272-config-volume\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:40.753152 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.753153    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:40.753322 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.753176    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c765ba9a-acaf-44e5-890a-42e943220b81-crio-socket\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:40.753322 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.753269    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsbcj\" (UniqueName: \"kubernetes.io/projected/726cdb79-f52a-4194-923b-bb594c60490d-kube-api-access-zsbcj\") pod \"ingress-canary-kr5wn\" (UID: \"726cdb79-f52a-4194-923b-bb594c60490d\") " pod="openshift-ingress-canary/ingress-canary-kr5wn"
Jul 28 18:01:40.753322 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:40.753273    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 18:01:40.753322 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.753289    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2cnd\" (UniqueName: \"kubernetes.io/projected/c765ba9a-acaf-44e5-890a-42e943220b81-kube-api-access-x2cnd\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:40.753322 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.753314    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615e5794-659e-4726-b961-8e5be52f2272-tmp-dir\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:40.753525 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:40.753342    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls podName:615e5794-659e-4726-b961-8e5be52f2272 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:41.253325061 +0000 UTC m=+36.568827152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls") pod "dns-default-rz8gm" (UID: "615e5794-659e-4726-b961-8e5be52f2272") : secret "dns-default-metrics-tls" not found
Jul 28 18:01:40.753525 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.753378    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wqpg\" (UniqueName: \"kubernetes.io/projected/615e5794-659e-4726-b961-8e5be52f2272-kube-api-access-5wqpg\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:40.753525 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.753414    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/726cdb79-f52a-4194-923b-bb594c60490d-cert\") pod \"ingress-canary-kr5wn\" (UID: \"726cdb79-f52a-4194-923b-bb594c60490d\") " pod="openshift-ingress-canary/ingress-canary-kr5wn"
Jul 28 18:01:40.753525 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.753447    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:40.753525 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.753473    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c765ba9a-acaf-44e5-890a-42e943220b81-data-volume\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:40.753525 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.753495    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c765ba9a-acaf-44e5-890a-42e943220b81-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:40.753756 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.753555    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615e5794-659e-4726-b961-8e5be52f2272-tmp-dir\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:40.753756 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.753746    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/615e5794-659e-4726-b961-8e5be52f2272-config-volume\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:40.757782 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.757761    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/726cdb79-f52a-4194-923b-bb594c60490d-cert\") pod \"ingress-canary-kr5wn\" (UID: \"726cdb79-f52a-4194-923b-bb594c60490d\") " pod="openshift-ingress-canary/ingress-canary-kr5wn"
Jul 28 18:01:40.764079 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.764015    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wqpg\" (UniqueName: \"kubernetes.io/projected/615e5794-659e-4726-b961-8e5be52f2272-kube-api-access-5wqpg\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:40.764198 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.764181    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsbcj\" (UniqueName: \"kubernetes.io/projected/726cdb79-f52a-4194-923b-bb594c60490d-kube-api-access-zsbcj\") pod \"ingress-canary-kr5wn\" (UID: \"726cdb79-f52a-4194-923b-bb594c60490d\") " pod="openshift-ingress-canary/ingress-canary-kr5wn"
Jul 28 18:01:40.854441 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.854413    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c765ba9a-acaf-44e5-890a-42e943220b81-crio-socket\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:40.854559 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.854449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2cnd\" (UniqueName: \"kubernetes.io/projected/c765ba9a-acaf-44e5-890a-42e943220b81-kube-api-access-x2cnd\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:40.854630 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.854608    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:40.854670 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.854631    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c765ba9a-acaf-44e5-890a-42e943220b81-crio-socket\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:40.854670 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.854648    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c765ba9a-acaf-44e5-890a-42e943220b81-data-volume\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:40.854728 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.854673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c765ba9a-acaf-44e5-890a-42e943220b81-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:40.854769 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:40.854725    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 18:01:40.854801 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:40.854785    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls podName:c765ba9a-acaf-44e5-890a-42e943220b81 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:41.354771071 +0000 UTC m=+36.670273166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvdpr" (UID: "c765ba9a-acaf-44e5-890a-42e943220b81") : secret "insights-runtime-extractor-tls" not found
Jul 28 18:01:40.854982 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.854963    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c765ba9a-acaf-44e5-890a-42e943220b81-data-volume\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:40.855274 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.855258    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c765ba9a-acaf-44e5-890a-42e943220b81-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:40.864733 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.864716    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kr5wn"
Jul 28 18:01:40.865537 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:40.865521    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2cnd\" (UniqueName: \"kubernetes.io/projected/c765ba9a-acaf-44e5-890a-42e943220b81-kube-api-access-x2cnd\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:41.022181 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:41.022115    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kr5wn"]
Jul 28 18:01:41.026084 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:41.026056    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726cdb79_f52a_4194_923b_bb594c60490d.slice/crio-4f7b2e10cd9f0147efcf7c2c17c6ba5b6ec65a324fe1d0065419036db780fb7b WatchSource:0}: Error finding container 4f7b2e10cd9f0147efcf7c2c17c6ba5b6ec65a324fe1d0065419036db780fb7b: Status 404 returned error can't find the container with id 4f7b2e10cd9f0147efcf7c2c17c6ba5b6ec65a324fe1d0065419036db780fb7b
Jul 28 18:01:41.211369 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:41.211335    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:01:41.213927 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:41.213903    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 18:01:41.214035 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:41.213903    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x879t\""
Jul 28 18:01:41.258212 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:41.258184    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:41.258323 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:41.258306    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 18:01:41.258384 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:41.258375    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls podName:615e5794-659e-4726-b961-8e5be52f2272 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:42.258363217 +0000 UTC m=+37.573865308 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls") pod "dns-default-rz8gm" (UID: "615e5794-659e-4726-b961-8e5be52f2272") : secret "dns-default-metrics-tls" not found
Jul 28 18:01:41.358746 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:41.358666    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:41.358886 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:41.358780    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 18:01:41.358886 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:41.358859    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls podName:c765ba9a-acaf-44e5-890a-42e943220b81 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:42.358843834 +0000 UTC m=+37.674345925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvdpr" (UID: "c765ba9a-acaf-44e5-890a-42e943220b81") : secret "insights-runtime-extractor-tls" not found
Jul 28 18:01:41.397148 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:41.397116    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kr5wn" event={"ID":"726cdb79-f52a-4194-923b-bb594c60490d","Type":"ContainerStarted","Data":"4f7b2e10cd9f0147efcf7c2c17c6ba5b6ec65a324fe1d0065419036db780fb7b"}
Jul 28 18:01:41.400045 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:41.400021    2564 generic.go:356] "Generic (PLEG): container finished" podID="7a63ca3e-bad2-4221-8493-38c09b9ad532" containerID="410d1390b5c46bd2462f7c1459034d01a729a0eaccbc30940cbf2a14a8b01bee" exitCode=0
Jul 28 18:01:41.400178 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:41.400069    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vpvk" event={"ID":"7a63ca3e-bad2-4221-8493-38c09b9ad532","Type":"ContainerDied","Data":"410d1390b5c46bd2462f7c1459034d01a729a0eaccbc30940cbf2a14a8b01bee"}
Jul 28 18:01:42.211108 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.210873    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:01:42.211284 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.210925    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:01:42.215339 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.215319    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 18:01:42.216394 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.216371    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9p4qw\""
Jul 28 18:01:42.216518 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.216401    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 18:01:42.216518 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.216450    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 18:01:42.267692 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.267666    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:42.267838 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:42.267775    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 18:01:42.267882 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:42.267854    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls podName:615e5794-659e-4726-b961-8e5be52f2272 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:44.267837118 +0000 UTC m=+39.583339229 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls") pod "dns-default-rz8gm" (UID: "615e5794-659e-4726-b961-8e5be52f2272") : secret "dns-default-metrics-tls" not found
Jul 28 18:01:42.361974 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.361946    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9wgst"]
Jul 28 18:01:42.363908 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.363885    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9wgst"
Jul 28 18:01:42.368161 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.368080    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:42.368285 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:42.368182    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 18:01:42.368285 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:42.368254    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls podName:c765ba9a-acaf-44e5-890a-42e943220b81 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:44.368236018 +0000 UTC m=+39.683738111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvdpr" (UID: "c765ba9a-acaf-44e5-890a-42e943220b81") : secret "insights-runtime-extractor-tls" not found
Jul 28 18:01:42.368787 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.368659    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 18:01:42.368787 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.368670    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-q6g2v\""
Jul 28 18:01:42.369637 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.369620    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 18:01:42.392905 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.392882    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9wgst"]
Jul 28 18:01:42.405895 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.405870    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6vpvk" event={"ID":"7a63ca3e-bad2-4221-8493-38c09b9ad532","Type":"ContainerStarted","Data":"a6da38143554cfe1ee6cdd98e77fcd2d7feaab47f0de2d1257c54f5fcc9024fe"}
Jul 28 18:01:42.468689 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.468605    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpk7q\" (UniqueName: \"kubernetes.io/projected/02c57bb5-3113-47ed-8052-15154fb5b1a1-kube-api-access-tpk7q\") pod \"migrator-6865c9967-9wgst\" (UID: \"02c57bb5-3113-47ed-8052-15154fb5b1a1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9wgst"
Jul 28 18:01:42.478200 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.478145    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6vpvk" podStartSLOduration=7.6083984749999996 podStartE2EDuration="37.478123142s" totalImagesPullingTime="29.869724667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:01:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:42.476295618 +0000 UTC m=+37.791797767" watchObservedRunningTime="2026-07-28 18:01:42.478123142 +0000 UTC m=+37.793625250"
Jul 28 18:01:42.569922 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.569780    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpk7q\" (UniqueName: \"kubernetes.io/projected/02c57bb5-3113-47ed-8052-15154fb5b1a1-kube-api-access-tpk7q\") pod \"migrator-6865c9967-9wgst\" (UID: \"02c57bb5-3113-47ed-8052-15154fb5b1a1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9wgst"
Jul 28 18:01:42.581180 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.581144    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpk7q\" (UniqueName: \"kubernetes.io/projected/02c57bb5-3113-47ed-8052-15154fb5b1a1-kube-api-access-tpk7q\") pod \"migrator-6865c9967-9wgst\" (UID: \"02c57bb5-3113-47ed-8052-15154fb5b1a1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9wgst"
Jul 28 18:01:42.675011 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:42.674976    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9wgst"
Jul 28 18:01:43.437518 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:43.437495    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9wgst"]
Jul 28 18:01:43.441006 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:43.440858    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c57bb5_3113_47ed_8052_15154fb5b1a1.slice/crio-346a746f06650e2f3c4149a61b5624be73d69730ae2b0a6e34c8341b62b52228 WatchSource:0}: Error finding container 346a746f06650e2f3c4149a61b5624be73d69730ae2b0a6e34c8341b62b52228: Status 404 returned error can't find the container with id 346a746f06650e2f3c4149a61b5624be73d69730ae2b0a6e34c8341b62b52228
Jul 28 18:01:44.283792 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:44.283746    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:44.284021 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:44.283916    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 18:01:44.284021 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:44.283976    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls podName:615e5794-659e-4726-b961-8e5be52f2272 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:48.283961371 +0000 UTC m=+43.599463462 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls") pod "dns-default-rz8gm" (UID: "615e5794-659e-4726-b961-8e5be52f2272") : secret "dns-default-metrics-tls" not found
Jul 28 18:01:44.384213 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:44.384180    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:44.384357 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:44.384289    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 18:01:44.384357 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:44.384347    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls podName:c765ba9a-acaf-44e5-890a-42e943220b81 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:48.38433022 +0000 UTC m=+43.699832325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvdpr" (UID: "c765ba9a-acaf-44e5-890a-42e943220b81") : secret "insights-runtime-extractor-tls" not found
Jul 28 18:01:44.412511 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:44.412481    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kr5wn" event={"ID":"726cdb79-f52a-4194-923b-bb594c60490d","Type":"ContainerStarted","Data":"ae340d1ac7433228549682204571987cb6dc0760817c178c8e52fec232a42326"}
Jul 28 18:01:44.413626 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:44.413606    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9wgst" event={"ID":"02c57bb5-3113-47ed-8052-15154fb5b1a1","Type":"ContainerStarted","Data":"346a746f06650e2f3c4149a61b5624be73d69730ae2b0a6e34c8341b62b52228"}
Jul 28 18:01:44.431504 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:44.431446    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kr5wn" podStartSLOduration=2.107444231 podStartE2EDuration="4.431426213s" totalImagesPullingTime="2.323981982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:44.429314197 +0000 UTC m=+39.744816312" watchObservedRunningTime="2026-07-28 18:01:44.431426213 +0000 UTC m=+39.746928328"
Jul 28 18:01:46.420747 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:46.420711    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9wgst" event={"ID":"02c57bb5-3113-47ed-8052-15154fb5b1a1","Type":"ContainerStarted","Data":"9308ceec12e3061c784062c7a57fb9f82ae136897ebbbdb6972de1e6c6bb22c1"}
Jul 28 18:01:46.421132 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:46.420764    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9wgst" event={"ID":"02c57bb5-3113-47ed-8052-15154fb5b1a1","Type":"ContainerStarted","Data":"16b88c13ace64f7560034f09697af619b0ee522283e6da15e0bd23cb4055e937"}
Jul 28 18:01:46.441787 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:46.441743    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9wgst" podStartSLOduration=2.329060892 podStartE2EDuration="4.441727414s" totalImagesPullingTime="2.112666522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:46.441205757 +0000 UTC m=+41.756707870" watchObservedRunningTime="2026-07-28 18:01:46.441727414 +0000 UTC m=+41.757229530"
Jul 28 18:01:48.313914 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:48.313875    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:48.314295 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:48.314030    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 18:01:48.314295 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:48.314099    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls podName:615e5794-659e-4726-b961-8e5be52f2272 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:56.314083687 +0000 UTC m=+51.629585778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls") pod "dns-default-rz8gm" (UID: "615e5794-659e-4726-b961-8e5be52f2272") : secret "dns-default-metrics-tls" not found
Jul 28 18:01:48.414680 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:48.414637    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:48.414863 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:48.414764    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 18:01:48.414863 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:01:48.414850    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls podName:c765ba9a-acaf-44e5-890a-42e943220b81 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:56.414831024 +0000 UTC m=+51.730333129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvdpr" (UID: "c765ba9a-acaf-44e5-890a-42e943220b81") : secret "insights-runtime-extractor-tls" not found
Jul 28 18:01:56.370492 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:56.370453    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:56.373475 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:56.373450    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/615e5794-659e-4726-b961-8e5be52f2272-metrics-tls\") pod \"dns-default-rz8gm\" (UID: \"615e5794-659e-4726-b961-8e5be52f2272\") " pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:56.447389 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:56.447360    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:56.471670 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:56.471639    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:56.474061 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:56.474030    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c765ba9a-acaf-44e5-890a-42e943220b81-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvdpr\" (UID: \"c765ba9a-acaf-44e5-890a-42e943220b81\") " pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:56.538363 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:56.538333    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xvdpr"
Jul 28 18:01:56.601526 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:56.601490    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rz8gm"]
Jul 28 18:01:56.605201 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:56.605132    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615e5794_659e_4726_b961_8e5be52f2272.slice/crio-3263677a5c79775aaf3c81ad2be452dbc24b3d50c2e93732729048654d48be04 WatchSource:0}: Error finding container 3263677a5c79775aaf3c81ad2be452dbc24b3d50c2e93732729048654d48be04: Status 404 returned error can't find the container with id 3263677a5c79775aaf3c81ad2be452dbc24b3d50c2e93732729048654d48be04
Jul 28 18:01:56.687605 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:56.687575    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xvdpr"]
Jul 28 18:01:56.692373 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:01:56.692347    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc765ba9a_acaf_44e5_890a_42e943220b81.slice/crio-55f89a79662ef1d5437a0d83e709defdfeaa13dffef70df6dd0c61639ba9301b WatchSource:0}: Error finding container 55f89a79662ef1d5437a0d83e709defdfeaa13dffef70df6dd0c61639ba9301b: Status 404 returned error can't find the container with id 55f89a79662ef1d5437a0d83e709defdfeaa13dffef70df6dd0c61639ba9301b
Jul 28 18:01:57.446302 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:57.446247    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvdpr" event={"ID":"c765ba9a-acaf-44e5-890a-42e943220b81","Type":"ContainerStarted","Data":"55f89a79662ef1d5437a0d83e709defdfeaa13dffef70df6dd0c61639ba9301b"}
Jul 28 18:01:57.447549 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:57.447525    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rz8gm" event={"ID":"615e5794-659e-4726-b961-8e5be52f2272","Type":"ContainerStarted","Data":"3263677a5c79775aaf3c81ad2be452dbc24b3d50c2e93732729048654d48be04"}
Jul 28 18:01:59.454817 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:59.454759    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rz8gm" event={"ID":"615e5794-659e-4726-b961-8e5be52f2272","Type":"ContainerStarted","Data":"41f2a0e418046b73c7204f2e773a012a3f7dc2e1e7bd099abcb4748b5cf570f2"}
Jul 28 18:01:59.455327 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:59.454838    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rz8gm" event={"ID":"615e5794-659e-4726-b961-8e5be52f2272","Type":"ContainerStarted","Data":"e8fc5bfe6276f255fab8dfaf1565d8e10aabf005ca7c6ceb0c850b1b6cfc373d"}
Jul 28 18:01:59.455327 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:59.454957    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:01:59.475399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:01:59.475314    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rz8gm" podStartSLOduration=17.435174312 podStartE2EDuration="19.475293508s" totalImagesPullingTime="2.040119196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:59.473146601 +0000 UTC m=+54.788648748" watchObservedRunningTime="2026-07-28 18:01:59.475293508 +0000 UTC m=+54.790795624"
Jul 28 18:02:04.387921 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:04.387746    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z2m7g"
Jul 28 18:02:04.475328 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:04.475280    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvdpr" event={"ID":"c765ba9a-acaf-44e5-890a-42e943220b81","Type":"ContainerStarted","Data":"1bfd2e98b53b75783084352624a8863cca1637841a27732193b9b850e3290c52"}
Jul 28 18:02:07.487710 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:07.487626    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvdpr" event={"ID":"c765ba9a-acaf-44e5-890a-42e943220b81","Type":"ContainerStarted","Data":"e75655549d2ff5525b81bd6d0018ddb960cc9a0d4a55e89f0ede651d16886cf7"}
Jul 28 18:02:09.459723 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:09.459690    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rz8gm"
Jul 28 18:02:10.174697 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.174481    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6"]
Jul 28 18:02:10.192622 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.192592    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"]
Jul 28 18:02:10.192818 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.192783    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6"
Jul 28 18:02:10.195773 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.195739    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 18:02:10.195941 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.195914    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 18:02:10.195941 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.195922    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 18:02:10.196066 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.196013    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 18:02:10.212613 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.212586    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6"]
Jul 28 18:02:10.212890 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.212646    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"]
Jul 28 18:02:10.212890 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.212727    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.215719 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.215696    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 18:02:10.215909 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.215885    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 18:02:10.216009 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.215957    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 18:02:10.216009 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.215998    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 18:02:10.273743 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.273711    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2mgg\" (UniqueName: \"kubernetes.io/projected/19ddcc6c-d438-4070-a3da-ea76dddd829e-kube-api-access-p2mgg\") pod \"klusterlet-addon-workmgr-5584f4b9f5-vd9k6\" (UID: \"19ddcc6c-d438-4070-a3da-ea76dddd829e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6"
Jul 28 18:02:10.273743 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.273741    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e71372eb-da75-4f5f-a3cf-38da1ab6b513-ca\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.273963 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.273758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh9zv\" (UniqueName: \"kubernetes.io/projected/e71372eb-da75-4f5f-a3cf-38da1ab6b513-kube-api-access-vh9zv\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.273963 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.273775    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ddcc6c-d438-4070-a3da-ea76dddd829e-tmp\") pod \"klusterlet-addon-workmgr-5584f4b9f5-vd9k6\" (UID: \"19ddcc6c-d438-4070-a3da-ea76dddd829e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6"
Jul 28 18:02:10.273963 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.273789    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e71372eb-da75-4f5f-a3cf-38da1ab6b513-hub\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.273963 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.273827    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/19ddcc6c-d438-4070-a3da-ea76dddd829e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5584f4b9f5-vd9k6\" (UID: \"19ddcc6c-d438-4070-a3da-ea76dddd829e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6"
Jul 28 18:02:10.273963 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.273846    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e71372eb-da75-4f5f-a3cf-38da1ab6b513-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.273963 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.273860    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e71372eb-da75-4f5f-a3cf-38da1ab6b513-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.273963 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.273894    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71372eb-da75-4f5f-a3cf-38da1ab6b513-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.374863 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.374832    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e71372eb-da75-4f5f-a3cf-38da1ab6b513-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.375050 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.374881    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71372eb-da75-4f5f-a3cf-38da1ab6b513-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.375050 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.374954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2mgg\" (UniqueName: \"kubernetes.io/projected/19ddcc6c-d438-4070-a3da-ea76dddd829e-kube-api-access-p2mgg\") pod \"klusterlet-addon-workmgr-5584f4b9f5-vd9k6\" (UID: \"19ddcc6c-d438-4070-a3da-ea76dddd829e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6"
Jul 28 18:02:10.375050 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.374971    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e71372eb-da75-4f5f-a3cf-38da1ab6b513-ca\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.375050 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.374996    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh9zv\" (UniqueName: \"kubernetes.io/projected/e71372eb-da75-4f5f-a3cf-38da1ab6b513-kube-api-access-vh9zv\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.375050 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.375015    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ddcc6c-d438-4070-a3da-ea76dddd829e-tmp\") pod \"klusterlet-addon-workmgr-5584f4b9f5-vd9k6\" (UID: \"19ddcc6c-d438-4070-a3da-ea76dddd829e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6"
Jul 28 18:02:10.375050 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.375031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e71372eb-da75-4f5f-a3cf-38da1ab6b513-hub\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.375387 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.375058    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/19ddcc6c-d438-4070-a3da-ea76dddd829e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5584f4b9f5-vd9k6\" (UID: \"19ddcc6c-d438-4070-a3da-ea76dddd829e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6"
Jul 28 18:02:10.375387 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.375085    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e71372eb-da75-4f5f-a3cf-38da1ab6b513-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.375387 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.375376    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ddcc6c-d438-4070-a3da-ea76dddd829e-tmp\") pod \"klusterlet-addon-workmgr-5584f4b9f5-vd9k6\" (UID: \"19ddcc6c-d438-4070-a3da-ea76dddd829e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6"
Jul 28 18:02:10.375800 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.375771    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71372eb-da75-4f5f-a3cf-38da1ab6b513-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.378221 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.378067    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e71372eb-da75-4f5f-a3cf-38da1ab6b513-hub\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.378468 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.378443    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e71372eb-da75-4f5f-a3cf-38da1ab6b513-ca\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.378545 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.378503    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e71372eb-da75-4f5f-a3cf-38da1ab6b513-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.378604 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.378565    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/19ddcc6c-d438-4070-a3da-ea76dddd829e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5584f4b9f5-vd9k6\" (UID: \"19ddcc6c-d438-4070-a3da-ea76dddd829e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6"
Jul 28 18:02:10.378714 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.378669    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e71372eb-da75-4f5f-a3cf-38da1ab6b513-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.389002 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.388942    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2mgg\" (UniqueName: \"kubernetes.io/projected/19ddcc6c-d438-4070-a3da-ea76dddd829e-kube-api-access-p2mgg\") pod \"klusterlet-addon-workmgr-5584f4b9f5-vd9k6\" (UID: \"19ddcc6c-d438-4070-a3da-ea76dddd829e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6"
Jul 28 18:02:10.389167 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.389104    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh9zv\" (UniqueName: \"kubernetes.io/projected/e71372eb-da75-4f5f-a3cf-38da1ab6b513-kube-api-access-vh9zv\") pod \"cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh\" (UID: \"e71372eb-da75-4f5f-a3cf-38da1ab6b513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.504521 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.504443    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6"
Jul 28 18:02:10.530801 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.530767    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"
Jul 28 18:02:10.878675 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.878631    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mphq\" (UniqueName: \"kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq\") pod \"network-check-target-tkx4l\" (UID: \"7485a88f-1174-493b-bef2-159f0b4c9646\") " pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:02:10.878888 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.878688    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs\") pod \"network-metrics-daemon-trhfb\" (UID: \"185c4451-1106-4555-855c-4f48a9348827\") " pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:02:10.878888 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.878730    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret\") pod \"global-pull-secret-syncer-pfrhs\" (UID: \"e69399ee-0e2d-477a-857d-79c7c39543a0\") " pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:02:10.881191 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.881168    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 18:02:10.881476 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.881452    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 18:02:10.881590 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.881504    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 18:02:10.892052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.892024    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 18:02:10.892194 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.892123    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e69399ee-0e2d-477a-857d-79c7c39543a0-original-pull-secret\") pod \"global-pull-secret-syncer-pfrhs\" (UID: \"e69399ee-0e2d-477a-857d-79c7c39543a0\") " pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:02:10.892194 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.892174    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185c4451-1106-4555-855c-4f48a9348827-metrics-certs\") pod \"network-metrics-daemon-trhfb\" (UID: \"185c4451-1106-4555-855c-4f48a9348827\") " pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:02:10.901408 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.901385    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mphq\" (UniqueName: \"kubernetes.io/projected/7485a88f-1174-493b-bef2-159f0b4c9646-kube-api-access-5mphq\") pod \"network-check-target-tkx4l\" (UID: \"7485a88f-1174-493b-bef2-159f0b4c9646\") " pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:02:10.922851 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.922802    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x879t\""
Jul 28 18:02:10.931108 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:10.931072    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trhfb"
Jul 28 18:02:11.025388 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:11.025349    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9p4qw\""
Jul 28 18:02:11.030563 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:11.030539    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfrhs"
Jul 28 18:02:11.033421 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:11.033399    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:02:12.157195 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:12.157160    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6"]
Jul 28 18:02:12.168379 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:02:12.168353    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ddcc6c_d438_4070_a3da_ea76dddd829e.slice/crio-77782fb7a79a1c1f4e0f2d18defcbefcbee08fed9c44595fe84ef00821aae738 WatchSource:0}: Error finding container 77782fb7a79a1c1f4e0f2d18defcbefcbee08fed9c44595fe84ef00821aae738: Status 404 returned error can't find the container with id 77782fb7a79a1c1f4e0f2d18defcbefcbee08fed9c44595fe84ef00821aae738
Jul 28 18:02:12.178700 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:12.178668    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh"]
Jul 28 18:02:12.183127 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:02:12.183104    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode71372eb_da75_4f5f_a3cf_38da1ab6b513.slice/crio-da4450b1e2aada185d2d9edec84102620bbda1ddb448c4127eef8b6431c380e9 WatchSource:0}: Error finding container da4450b1e2aada185d2d9edec84102620bbda1ddb448c4127eef8b6431c380e9: Status 404 returned error can't find the container with id da4450b1e2aada185d2d9edec84102620bbda1ddb448c4127eef8b6431c380e9
Jul 28 18:02:12.187999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:12.187977    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-trhfb"]
Jul 28 18:02:12.190713 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:02:12.190692    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185c4451_1106_4555_855c_4f48a9348827.slice/crio-abbece49748c245c46b3785ef618a86cac9e6a9ec443d88cb81dc89d1dc4dc8e WatchSource:0}: Error finding container abbece49748c245c46b3785ef618a86cac9e6a9ec443d88cb81dc89d1dc4dc8e: Status 404 returned error can't find the container with id abbece49748c245c46b3785ef618a86cac9e6a9ec443d88cb81dc89d1dc4dc8e
Jul 28 18:02:12.406170 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:12.406138    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pfrhs"]
Jul 28 18:02:12.409464 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:12.409435    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tkx4l"]
Jul 28 18:02:12.409735 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:02:12.409707    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69399ee_0e2d_477a_857d_79c7c39543a0.slice/crio-7f4886896ee2285c4c4ebb8e007dcfd3f6d8eac924125346d1f0f0778ad71c3f WatchSource:0}: Error finding container 7f4886896ee2285c4c4ebb8e007dcfd3f6d8eac924125346d1f0f0778ad71c3f: Status 404 returned error can't find the container with id 7f4886896ee2285c4c4ebb8e007dcfd3f6d8eac924125346d1f0f0778ad71c3f
Jul 28 18:02:12.413118 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:02:12.413097    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7485a88f_1174_493b_bef2_159f0b4c9646.slice/crio-4494d112bce85ffc1c0d3306ea5c0d01b95c00a56dd91b387e4fb98c6b21125a WatchSource:0}: Error finding container 4494d112bce85ffc1c0d3306ea5c0d01b95c00a56dd91b387e4fb98c6b21125a: Status 404 returned error can't find the container with id 4494d112bce85ffc1c0d3306ea5c0d01b95c00a56dd91b387e4fb98c6b21125a
Jul 28 18:02:12.509084 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:12.509047    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pfrhs" event={"ID":"e69399ee-0e2d-477a-857d-79c7c39543a0","Type":"ContainerStarted","Data":"7f4886896ee2285c4c4ebb8e007dcfd3f6d8eac924125346d1f0f0778ad71c3f"}
Jul 28 18:02:12.510089 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:12.510064    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6" event={"ID":"19ddcc6c-d438-4070-a3da-ea76dddd829e","Type":"ContainerStarted","Data":"77782fb7a79a1c1f4e0f2d18defcbefcbee08fed9c44595fe84ef00821aae738"}
Jul 28 18:02:12.511838 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:12.511803    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvdpr" event={"ID":"c765ba9a-acaf-44e5-890a-42e943220b81","Type":"ContainerStarted","Data":"f342ffa2dbe3209637e9ad5333ec571f81e05e64208f7025bb42b5737380c411"}
Jul 28 18:02:12.512907 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:12.512877    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tkx4l" event={"ID":"7485a88f-1174-493b-bef2-159f0b4c9646","Type":"ContainerStarted","Data":"4494d112bce85ffc1c0d3306ea5c0d01b95c00a56dd91b387e4fb98c6b21125a"}
Jul 28 18:02:12.513969 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:12.513947    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-trhfb" event={"ID":"185c4451-1106-4555-855c-4f48a9348827","Type":"ContainerStarted","Data":"abbece49748c245c46b3785ef618a86cac9e6a9ec443d88cb81dc89d1dc4dc8e"}
Jul 28 18:02:12.514956 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:12.514934    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh" event={"ID":"e71372eb-da75-4f5f-a3cf-38da1ab6b513","Type":"ContainerStarted","Data":"da4450b1e2aada185d2d9edec84102620bbda1ddb448c4127eef8b6431c380e9"}
Jul 28 18:02:12.530725 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:12.530676    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xvdpr" podStartSLOduration=17.316909237 podStartE2EDuration="32.530662034s" totalImagesPullingTime="15.213752797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:01:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:12.529746384 +0000 UTC m=+67.845248514" watchObservedRunningTime="2026-07-28 18:02:12.530662034 +0000 UTC m=+67.846164147"
Jul 28 18:02:13.524062 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:13.524023    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-trhfb" event={"ID":"185c4451-1106-4555-855c-4f48a9348827","Type":"ContainerStarted","Data":"a4f1422e23412ba423d52246b5dc1cf25ad7dd37082f3854e2b1b1a419503b27"}
Jul 28 18:02:14.535472 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:14.534696    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-trhfb" event={"ID":"185c4451-1106-4555-855c-4f48a9348827","Type":"ContainerStarted","Data":"b0cad4a57545af8d6cd410fe0b105cfe15c478574750c3d91b5291ba27d066f3"}
Jul 28 18:02:14.553796 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:14.553036    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-trhfb" podStartSLOduration=68.432076808 podStartE2EDuration="1m9.55301526s" totalImagesPullingTime="1.120938452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:14.55201265 +0000 UTC m=+69.867514790" watchObservedRunningTime="2026-07-28 18:02:14.55301526 +0000 UTC m=+69.868517374"
Jul 28 18:02:20.559766 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:20.559725    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tkx4l" event={"ID":"7485a88f-1174-493b-bef2-159f0b4c9646","Type":"ContainerStarted","Data":"04b6d781cb84b8d3d3b201a7dde5bf480e1541428972abe66c212eeba9032c98"}
Jul 28 18:02:20.560255 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:20.559879    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:02:20.565520 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:20.565492    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh" event={"ID":"e71372eb-da75-4f5f-a3cf-38da1ab6b513","Type":"ContainerStarted","Data":"528906a11d5286dd12fe6e2bf10f019ee1694d38bb08b57b3be8128873d9485f"}
Jul 28 18:02:20.565663 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:20.565535    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh" event={"ID":"e71372eb-da75-4f5f-a3cf-38da1ab6b513","Type":"ContainerStarted","Data":"d9490200bf9298f519caf12ad520a6bda61d6e205577db86c7b385c1f5c0fe20"}
Jul 28 18:02:20.565663 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:20.565553    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh" event={"ID":"e71372eb-da75-4f5f-a3cf-38da1ab6b513","Type":"ContainerStarted","Data":"370995f98d23cf665bcecab26158e2ab5685d6723b0e215b1a14583dbfab1fa9"}
Jul 28 18:02:20.566905 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:20.566876    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pfrhs" event={"ID":"e69399ee-0e2d-477a-857d-79c7c39543a0","Type":"ContainerStarted","Data":"d8eb58891d363a15738576b40cd221a845e4e529359191639975660c06490733"}
Jul 28 18:02:20.568301 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:20.568269    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6" event={"ID":"19ddcc6c-d438-4070-a3da-ea76dddd829e","Type":"ContainerStarted","Data":"2c1ed3309a7348e272ccb01bf5af89800f7ccc912a98ad1973e96e4ba1310512"}
Jul 28 18:02:20.568441 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:20.568425    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6"
Jul 28 18:02:20.570045 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:20.570027    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6"
Jul 28 18:02:20.577376 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:20.577334    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tkx4l" podStartSLOduration=67.890069233 podStartE2EDuration="1m15.577322384s" totalImagesPullingTime="7.687253151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:20.5770192 +0000 UTC m=+75.892521337" watchObservedRunningTime="2026-07-28 18:02:20.577322384 +0000 UTC m=+75.892824536"
Jul 28 18:02:20.592884 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:20.592845    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pfrhs" podStartSLOduration=67.896359249 podStartE2EDuration="1m15.59283328s" totalImagesPullingTime="7.696474031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:20.592551268 +0000 UTC m=+75.908053384" watchObservedRunningTime="2026-07-28 18:02:20.59283328 +0000 UTC m=+75.908335387"
Jul 28 18:02:20.613377 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:20.613338    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5584f4b9f5-vd9k6" podStartSLOduration=2.691186303 podStartE2EDuration="10.613324235s" totalImagesPullingTime="7.922137932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:20.612387306 +0000 UTC m=+75.927889421" watchObservedRunningTime="2026-07-28 18:02:20.613324235 +0000 UTC m=+75.928826347"
Jul 28 18:02:20.634653 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:20.634613    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8ddcbd64-rl6rh" podStartSLOduration=2.711525489 podStartE2EDuration="10.634597709s" totalImagesPullingTime="7.92307222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:20.634132654 +0000 UTC m=+75.949634768" watchObservedRunningTime="2026-07-28 18:02:20.634597709 +0000 UTC m=+75.950099822"
Jul 28 18:02:21.039037 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.039006    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7khcj"]
Jul 28 18:02:21.041365 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.041345    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.045893 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.045867    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 18:02:21.045992 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.045872    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 18:02:21.045992 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.045983    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-r5f9p\""
Jul 28 18:02:21.046983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.046958    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 18:02:21.047068 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.046961    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 18:02:21.047068 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.046994    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 18:02:21.047068 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.046991    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 18:02:21.157080 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.157049    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.157257 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.157091    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-sys\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.157257 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.157109    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-textfile\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.157257 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.157135    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-tls\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.157257 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.157155    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-accelerators-collector-config\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.157257 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.157180    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-root\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.157257 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.157196    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-metrics-client-ca\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.157257 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.157212    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-wtmp\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.157484 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.157285    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znqwp\" (UniqueName: \"kubernetes.io/projected/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-kube-api-access-znqwp\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.258208 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.258174    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-tls\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.258385 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.258218    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-accelerators-collector-config\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.258385 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.258246    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-root\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.258385 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.258271    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-metrics-client-ca\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.258385 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.258293    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-wtmp\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.258385 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:02:21.258321    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 18:02:21.258635 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.258394    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-root\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.258635 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:02:21.258405    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-tls podName:61a722d1-355b-4055-adc5-7aa9ee3fcd5b nodeName:}" failed. No retries permitted until 2026-07-28 18:02:21.758382452 +0000 UTC m=+77.073884542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-tls") pod "node-exporter-7khcj" (UID: "61a722d1-355b-4055-adc5-7aa9ee3fcd5b") : secret "node-exporter-tls" not found
Jul 28 18:02:21.258635 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.258327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znqwp\" (UniqueName: \"kubernetes.io/projected/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-kube-api-access-znqwp\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.258635 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.258463    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.258635 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.258469    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-wtmp\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.258635 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.258519    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-sys\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.258635 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.258542    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-textfile\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.258635 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.258566    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-sys\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.259009 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.258782    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-accelerators-collector-config\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.259009 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.258862    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-textfile\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.259009 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.258866    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-metrics-client-ca\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.260714 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.260693    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.271162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.271141    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znqwp\" (UniqueName: \"kubernetes.io/projected/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-kube-api-access-znqwp\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.762426 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.762387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-tls\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.764697 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.764672    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61a722d1-355b-4055-adc5-7aa9ee3fcd5b-node-exporter-tls\") pod \"node-exporter-7khcj\" (UID: \"61a722d1-355b-4055-adc5-7aa9ee3fcd5b\") " pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.950189 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:21.950152    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7khcj"
Jul 28 18:02:21.960298 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:02:21.960265    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a722d1_355b_4055_adc5_7aa9ee3fcd5b.slice/crio-54bdc9efb5c63bddc3e31179de74d8ee3b2420076860e4d9ed9a6c29559d382f WatchSource:0}: Error finding container 54bdc9efb5c63bddc3e31179de74d8ee3b2420076860e4d9ed9a6c29559d382f: Status 404 returned error can't find the container with id 54bdc9efb5c63bddc3e31179de74d8ee3b2420076860e4d9ed9a6c29559d382f
Jul 28 18:02:22.576883 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:22.576844    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7khcj" event={"ID":"61a722d1-355b-4055-adc5-7aa9ee3fcd5b","Type":"ContainerStarted","Data":"54bdc9efb5c63bddc3e31179de74d8ee3b2420076860e4d9ed9a6c29559d382f"}
Jul 28 18:02:23.581772 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:23.581729    2564 generic.go:356] "Generic (PLEG): container finished" podID="61a722d1-355b-4055-adc5-7aa9ee3fcd5b" containerID="b58af4f32677962bea56dbada36bf71b0627f620ae661cdb0619cbe52dfcb80c" exitCode=0
Jul 28 18:02:23.582161 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:23.581829    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7khcj" event={"ID":"61a722d1-355b-4055-adc5-7aa9ee3fcd5b","Type":"ContainerDied","Data":"b58af4f32677962bea56dbada36bf71b0627f620ae661cdb0619cbe52dfcb80c"}
Jul 28 18:02:24.587775 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:24.587744    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7khcj" event={"ID":"61a722d1-355b-4055-adc5-7aa9ee3fcd5b","Type":"ContainerStarted","Data":"cb1d814650f3bf393d30a04b211f58e3f6307ac6fc2dbbe70357d47e27750b4a"}
Jul 28 18:02:24.588159 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:24.587794    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7khcj" event={"ID":"61a722d1-355b-4055-adc5-7aa9ee3fcd5b","Type":"ContainerStarted","Data":"adefb314a053ee14c911ffc73fdca7231803b7c6add8fe57eaa1480acce2a2e6"}
Jul 28 18:02:24.608546 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:24.608501    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7khcj" podStartSLOduration=2.5392987959999997 podStartE2EDuration="3.608487163s" totalImagesPullingTime="1.006406883s" totalInitContainerRuntime="62.781484ms" isStatefulPod=true podCreationTimestamp="2026-07-28 18:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:24.607245486 +0000 UTC m=+79.922747635" watchObservedRunningTime="2026-07-28 18:02:24.608487163 +0000 UTC m=+79.923989275"
Jul 28 18:02:38.400025 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.399897    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5db8fb95fc-k5wzt"]
Jul 28 18:02:38.444578 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.444544    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db8fb95fc-k5wzt"]
Jul 28 18:02:38.444726 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.444641    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.449104 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.447692    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 18:02:38.449104 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.447961    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 18:02:38.449591 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.449408    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 18:02:38.449591 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.449429    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 18:02:38.449591 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.449413    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9j2gt\""
Jul 28 18:02:38.449591 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.449451    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 18:02:38.449591 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.449502    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 18:02:38.449934 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.449676    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 18:02:38.569667 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.569636    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-service-ca\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.569843 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.569674    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-oauth-serving-cert\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.569843 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.569714    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-config\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.569843 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.569770    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-serving-cert\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.569843 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.569793    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmsgh\" (UniqueName: \"kubernetes.io/projected/2b8fef3a-f091-4603-b8b8-505b18a9205c-kube-api-access-jmsgh\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.569843 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.569836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-oauth-config\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.671132 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.671061    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-oauth-config\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.671132 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.671098    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-service-ca\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.671132 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.671115    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-oauth-serving-cert\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.671335 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.671150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-config\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.671335 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.671193    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-serving-cert\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.671335 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.671217    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmsgh\" (UniqueName: \"kubernetes.io/projected/2b8fef3a-f091-4603-b8b8-505b18a9205c-kube-api-access-jmsgh\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.671871 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.671845    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-oauth-serving-cert\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.671986 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.671946    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-config\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.672046 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.671988    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-service-ca\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.673679 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.673651    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-serving-cert\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.673771 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.673737    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-oauth-config\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.680248 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.680227    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmsgh\" (UniqueName: \"kubernetes.io/projected/2b8fef3a-f091-4603-b8b8-505b18a9205c-kube-api-access-jmsgh\") pod \"console-5db8fb95fc-k5wzt\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") " pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.755759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.755738    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:38.880683 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:38.880661    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db8fb95fc-k5wzt"]
Jul 28 18:02:38.883039 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:02:38.883008    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b8fef3a_f091_4603_b8b8_505b18a9205c.slice/crio-8bc861954813a4d5bc8fb87f98a6d246b77e2fda9450a3d1e1006aee4e15ea9b WatchSource:0}: Error finding container 8bc861954813a4d5bc8fb87f98a6d246b77e2fda9450a3d1e1006aee4e15ea9b: Status 404 returned error can't find the container with id 8bc861954813a4d5bc8fb87f98a6d246b77e2fda9450a3d1e1006aee4e15ea9b
Jul 28 18:02:39.638165 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:39.638123    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db8fb95fc-k5wzt" event={"ID":"2b8fef3a-f091-4603-b8b8-505b18a9205c","Type":"ContainerStarted","Data":"8bc861954813a4d5bc8fb87f98a6d246b77e2fda9450a3d1e1006aee4e15ea9b"}
Jul 28 18:02:42.649054 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:42.649019    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db8fb95fc-k5wzt" event={"ID":"2b8fef3a-f091-4603-b8b8-505b18a9205c","Type":"ContainerStarted","Data":"efeab6504cc242b4a6a840c983e80282e8de043b1a3a5692e3205393af64265c"}
Jul 28 18:02:42.667588 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:42.667544    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5db8fb95fc-k5wzt" podStartSLOduration=1.936886823 podStartE2EDuration="4.667530394s" totalImagesPullingTime="2.730643571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:42.667029652 +0000 UTC m=+97.982531785" watchObservedRunningTime="2026-07-28 18:02:42.667530394 +0000 UTC m=+97.983032522"
Jul 28 18:02:47.792746 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:47.792713    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7db67d646-q65zz"]
Jul 28 18:02:47.935390 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:47.935345    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7db67d646-q65zz"]
Jul 28 18:02:47.935390 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:47.935361    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:47.944247 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:47.944224    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 18:02:48.036741 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.036712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f63baaa-e74f-4493-b4ac-690431798e9e-console-serving-cert\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.036914 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.036751    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-console-config\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.036914 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.036774    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k67sp\" (UniqueName: \"kubernetes.io/projected/5f63baaa-e74f-4493-b4ac-690431798e9e-kube-api-access-k67sp\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.036914 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.036825    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-service-ca\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.036914 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.036858    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-oauth-serving-cert\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.036914 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.036877    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-trusted-ca-bundle\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.037084 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.036927    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f63baaa-e74f-4493-b4ac-690431798e9e-console-oauth-config\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.137831 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.137712    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-console-config\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.137831 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.137762    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k67sp\" (UniqueName: \"kubernetes.io/projected/5f63baaa-e74f-4493-b4ac-690431798e9e-kube-api-access-k67sp\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.137831 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.137789    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-service-ca\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.138105 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.137841    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-oauth-serving-cert\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.138105 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.137871    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-trusted-ca-bundle\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.138105 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.137918    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f63baaa-e74f-4493-b4ac-690431798e9e-console-oauth-config\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.138254 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.138103    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f63baaa-e74f-4493-b4ac-690431798e9e-console-serving-cert\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.138522 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.138493    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-service-ca\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.138613 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.138527    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-console-config\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.138613 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.138571    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-oauth-serving-cert\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.138772 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.138752    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-trusted-ca-bundle\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.140473 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.140449    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f63baaa-e74f-4493-b4ac-690431798e9e-console-oauth-config\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.140556 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.140457    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f63baaa-e74f-4493-b4ac-690431798e9e-console-serving-cert\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.146517 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.146499    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k67sp\" (UniqueName: \"kubernetes.io/projected/5f63baaa-e74f-4493-b4ac-690431798e9e-kube-api-access-k67sp\") pod \"console-7db67d646-q65zz\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") " pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.244732 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.244698    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:48.400929 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.400863    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7db67d646-q65zz"]
Jul 28 18:02:48.403969 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:02:48.403948    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f63baaa_e74f_4493_b4ac_690431798e9e.slice/crio-0c773a7ffabf1e5e57e0b10a7635914ea691942dc58cec064c208937c6aa134c WatchSource:0}: Error finding container 0c773a7ffabf1e5e57e0b10a7635914ea691942dc58cec064c208937c6aa134c: Status 404 returned error can't find the container with id 0c773a7ffabf1e5e57e0b10a7635914ea691942dc58cec064c208937c6aa134c
Jul 28 18:02:48.672501 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.672417    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7db67d646-q65zz" event={"ID":"5f63baaa-e74f-4493-b4ac-690431798e9e","Type":"ContainerStarted","Data":"ade9ae6ef4140981311024c92ac98cfb5f115ef0d0481fbe21824fc48ffaf699"}
Jul 28 18:02:48.672501 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.672466    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7db67d646-q65zz" event={"ID":"5f63baaa-e74f-4493-b4ac-690431798e9e","Type":"ContainerStarted","Data":"0c773a7ffabf1e5e57e0b10a7635914ea691942dc58cec064c208937c6aa134c"}
Jul 28 18:02:48.691313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.691267    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7db67d646-q65zz" podStartSLOduration=1.691250071 podStartE2EDuration="1.691250071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:48.690556141 +0000 UTC m=+104.006058275" watchObservedRunningTime="2026-07-28 18:02:48.691250071 +0000 UTC m=+104.006752186"
Jul 28 18:02:48.756790 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.756757    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:48.756956 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.756842    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:48.761549 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:48.761527    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:49.679616 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:49.679589    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:02:51.574222 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:51.574188    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tkx4l"
Jul 28 18:02:58.245346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:58.245285    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:58.245754 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:58.245375    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:58.250125 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:58.250104    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:58.711296 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:58.711269    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:02:58.781245 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:02:58.781215    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5db8fb95fc-k5wzt"]
Jul 28 18:03:23.805514 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:23.805457    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5db8fb95fc-k5wzt" podUID="2b8fef3a-f091-4603-b8b8-505b18a9205c" containerName="console" containerID="cri-o://efeab6504cc242b4a6a840c983e80282e8de043b1a3a5692e3205393af64265c" gracePeriod=15
Jul 28 18:03:24.037465 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.037443    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:03:24.197421 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.197332    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmsgh\" (UniqueName: \"kubernetes.io/projected/2b8fef3a-f091-4603-b8b8-505b18a9205c-kube-api-access-jmsgh\") pod \"2b8fef3a-f091-4603-b8b8-505b18a9205c\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") "
Jul 28 18:03:24.197421 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.197375    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-service-ca\") pod \"2b8fef3a-f091-4603-b8b8-505b18a9205c\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") "
Jul 28 18:03:24.197421 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.197404    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-config\") pod \"2b8fef3a-f091-4603-b8b8-505b18a9205c\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") "
Jul 28 18:03:24.197421 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.197427    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-serving-cert\") pod \"2b8fef3a-f091-4603-b8b8-505b18a9205c\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") "
Jul 28 18:03:24.197742 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.197453    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-oauth-config\") pod \"2b8fef3a-f091-4603-b8b8-505b18a9205c\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") "
Jul 28 18:03:24.197742 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.197601    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-oauth-serving-cert\") pod \"2b8fef3a-f091-4603-b8b8-505b18a9205c\" (UID: \"2b8fef3a-f091-4603-b8b8-505b18a9205c\") "
Jul 28 18:03:24.197889 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.197855    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-config" (OuterVolumeSpecName: "console-config") pod "2b8fef3a-f091-4603-b8b8-505b18a9205c" (UID: "2b8fef3a-f091-4603-b8b8-505b18a9205c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:03:24.197938 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.197885    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-service-ca" (OuterVolumeSpecName: "service-ca") pod "2b8fef3a-f091-4603-b8b8-505b18a9205c" (UID: "2b8fef3a-f091-4603-b8b8-505b18a9205c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:03:24.197971 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.197937    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2b8fef3a-f091-4603-b8b8-505b18a9205c" (UID: "2b8fef3a-f091-4603-b8b8-505b18a9205c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:03:24.199765 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.199743    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8fef3a-f091-4603-b8b8-505b18a9205c-kube-api-access-jmsgh" (OuterVolumeSpecName: "kube-api-access-jmsgh") pod "2b8fef3a-f091-4603-b8b8-505b18a9205c" (UID: "2b8fef3a-f091-4603-b8b8-505b18a9205c"). InnerVolumeSpecName "kube-api-access-jmsgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:03:24.199867 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.199775    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2b8fef3a-f091-4603-b8b8-505b18a9205c" (UID: "2b8fef3a-f091-4603-b8b8-505b18a9205c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:03:24.199867 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.199845    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2b8fef3a-f091-4603-b8b8-505b18a9205c" (UID: "2b8fef3a-f091-4603-b8b8-505b18a9205c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:03:24.298219 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.298187    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmsgh\" (UniqueName: \"kubernetes.io/projected/2b8fef3a-f091-4603-b8b8-505b18a9205c-kube-api-access-jmsgh\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:03:24.298219 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.298214    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:03:24.298382 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.298226    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:03:24.298382 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.298238    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-serving-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:03:24.298382 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.298250    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b8fef3a-f091-4603-b8b8-505b18a9205c-console-oauth-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:03:24.298382 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.298261    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b8fef3a-f091-4603-b8b8-505b18a9205c-oauth-serving-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:03:24.794538 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.794499    2564 generic.go:356] "Generic (PLEG): container finished" podID="2b8fef3a-f091-4603-b8b8-505b18a9205c" containerID="efeab6504cc242b4a6a840c983e80282e8de043b1a3a5692e3205393af64265c" exitCode=0
Jul 28 18:03:24.794705 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.794547    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db8fb95fc-k5wzt"
Jul 28 18:03:24.794705 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.794546    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db8fb95fc-k5wzt" event={"ID":"2b8fef3a-f091-4603-b8b8-505b18a9205c","Type":"ContainerDied","Data":"efeab6504cc242b4a6a840c983e80282e8de043b1a3a5692e3205393af64265c"}
Jul 28 18:03:24.794705 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.794658    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db8fb95fc-k5wzt" event={"ID":"2b8fef3a-f091-4603-b8b8-505b18a9205c","Type":"ContainerDied","Data":"8bc861954813a4d5bc8fb87f98a6d246b77e2fda9450a3d1e1006aee4e15ea9b"}
Jul 28 18:03:24.794705 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.794683    2564 scope.go:117] "RemoveContainer" containerID="efeab6504cc242b4a6a840c983e80282e8de043b1a3a5692e3205393af64265c"
Jul 28 18:03:24.803030 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.803013    2564 scope.go:117] "RemoveContainer" containerID="efeab6504cc242b4a6a840c983e80282e8de043b1a3a5692e3205393af64265c"
Jul 28 18:03:24.803317 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:03:24.803296    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efeab6504cc242b4a6a840c983e80282e8de043b1a3a5692e3205393af64265c\": container with ID starting with efeab6504cc242b4a6a840c983e80282e8de043b1a3a5692e3205393af64265c not found: ID does not exist" containerID="efeab6504cc242b4a6a840c983e80282e8de043b1a3a5692e3205393af64265c"
Jul 28 18:03:24.803411 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.803326    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efeab6504cc242b4a6a840c983e80282e8de043b1a3a5692e3205393af64265c"} err="failed to get container status \"efeab6504cc242b4a6a840c983e80282e8de043b1a3a5692e3205393af64265c\": rpc error: code = NotFound desc = could not find container \"efeab6504cc242b4a6a840c983e80282e8de043b1a3a5692e3205393af64265c\": container with ID starting with efeab6504cc242b4a6a840c983e80282e8de043b1a3a5692e3205393af64265c not found: ID does not exist"
Jul 28 18:03:24.814790 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.814764    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5db8fb95fc-k5wzt"]
Jul 28 18:03:24.818296 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:24.818276    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5db8fb95fc-k5wzt"]
Jul 28 18:03:25.214975 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:25.214895    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b8fef3a-f091-4603-b8b8-505b18a9205c" path="/var/lib/kubelet/pods/2b8fef3a-f091-4603-b8b8-505b18a9205c/volumes"
Jul 28 18:03:50.773291 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.773258    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f69cbc78f-fm2xv"]
Jul 28 18:03:50.773652 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.773498    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b8fef3a-f091-4603-b8b8-505b18a9205c" containerName="console"
Jul 28 18:03:50.773652 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.773509    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b8fef3a-f091-4603-b8b8-505b18a9205c" containerName="console"
Jul 28 18:03:50.773652 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.773555    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b8fef3a-f091-4603-b8b8-505b18a9205c" containerName="console"
Jul 28 18:03:50.776347 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.776332    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.788539 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.788514    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f69cbc78f-fm2xv"]
Jul 28 18:03:50.867406 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.867376    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-serving-cert\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.867406 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.867408    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-oauth-serving-cert\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.867660 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.867435    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-trusted-ca-bundle\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.867660 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.867454    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-config\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.867660 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.867524    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mq9l\" (UniqueName: \"kubernetes.io/projected/4d002201-8428-4e79-94f0-cd5fde7e44aa-kube-api-access-5mq9l\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.867660 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.867571    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-oauth-config\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.867660 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.867598    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-service-ca\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.968236 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.968205    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-trusted-ca-bundle\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.968236 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.968242    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-config\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.968492 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.968266    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mq9l\" (UniqueName: \"kubernetes.io/projected/4d002201-8428-4e79-94f0-cd5fde7e44aa-kube-api-access-5mq9l\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.968492 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.968290    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-oauth-config\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.968492 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.968308    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-service-ca\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.968492 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.968336    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-serving-cert\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.968492 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.968357    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-oauth-serving-cert\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.969110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.969083    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-service-ca\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.969215 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.969154    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-config\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.969215 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.969202    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-oauth-serving-cert\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.969289 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.969256    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-trusted-ca-bundle\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.970937 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.970907    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-oauth-config\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.971004 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.970937    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-serving-cert\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:50.977726 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:50.977707    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mq9l\" (UniqueName: \"kubernetes.io/projected/4d002201-8428-4e79-94f0-cd5fde7e44aa-kube-api-access-5mq9l\") pod \"console-5f69cbc78f-fm2xv\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") " pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:51.085226 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:51.085136    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:03:51.214871 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:03:51.214842    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d002201_8428_4e79_94f0_cd5fde7e44aa.slice/crio-9cc4801d359a4c45a18d73e4f6adc561fa7fee5208739e8e3817990ead35cb59 WatchSource:0}: Error finding container 9cc4801d359a4c45a18d73e4f6adc561fa7fee5208739e8e3817990ead35cb59: Status 404 returned error can't find the container with id 9cc4801d359a4c45a18d73e4f6adc561fa7fee5208739e8e3817990ead35cb59
Jul 28 18:03:51.215042 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:51.215024    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f69cbc78f-fm2xv"]
Jul 28 18:03:51.893521 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:51.893484    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f69cbc78f-fm2xv" event={"ID":"4d002201-8428-4e79-94f0-cd5fde7e44aa","Type":"ContainerStarted","Data":"d4d40a7ef79071a31e13200c64538b92ec5c41ad678d3a381b1666b1bed99cda"}
Jul 28 18:03:51.893967 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:51.893556    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f69cbc78f-fm2xv" event={"ID":"4d002201-8428-4e79-94f0-cd5fde7e44aa","Type":"ContainerStarted","Data":"9cc4801d359a4c45a18d73e4f6adc561fa7fee5208739e8e3817990ead35cb59"}
Jul 28 18:03:51.923049 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:03:51.922995    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f69cbc78f-fm2xv" podStartSLOduration=1.922979195 podStartE2EDuration="1.922979195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:03:51.922489691 +0000 UTC m=+167.237991805" watchObservedRunningTime="2026-07-28 18:03:51.922979195 +0000 UTC m=+167.238481380"
Jul 28 18:04:01.086178 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:01.086069    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:04:01.086178 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:01.086136    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:04:01.090784 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:01.090764    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:04:01.932341 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:01.932309    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:04:01.979166 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:01.979139    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7db67d646-q65zz"]
Jul 28 18:04:27.000602 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.000542    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7db67d646-q65zz" podUID="5f63baaa-e74f-4493-b4ac-690431798e9e" containerName="console" containerID="cri-o://ade9ae6ef4140981311024c92ac98cfb5f115ef0d0481fbe21824fc48ffaf699" gracePeriod=15
Jul 28 18:04:27.234749 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.234727    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:04:27.313471 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.313439    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-service-ca\") pod \"5f63baaa-e74f-4493-b4ac-690431798e9e\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") "
Jul 28 18:04:27.313471 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.313478    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k67sp\" (UniqueName: \"kubernetes.io/projected/5f63baaa-e74f-4493-b4ac-690431798e9e-kube-api-access-k67sp\") pod \"5f63baaa-e74f-4493-b4ac-690431798e9e\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") "
Jul 28 18:04:27.313692 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.313516    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-console-config\") pod \"5f63baaa-e74f-4493-b4ac-690431798e9e\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") "
Jul 28 18:04:27.313692 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.313551    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f63baaa-e74f-4493-b4ac-690431798e9e-console-oauth-config\") pod \"5f63baaa-e74f-4493-b4ac-690431798e9e\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") "
Jul 28 18:04:27.313692 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.313578    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f63baaa-e74f-4493-b4ac-690431798e9e-console-serving-cert\") pod \"5f63baaa-e74f-4493-b4ac-690431798e9e\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") "
Jul 28 18:04:27.313692 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.313626    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-trusted-ca-bundle\") pod \"5f63baaa-e74f-4493-b4ac-690431798e9e\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") "
Jul 28 18:04:27.313692 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.313654    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-oauth-serving-cert\") pod \"5f63baaa-e74f-4493-b4ac-690431798e9e\" (UID: \"5f63baaa-e74f-4493-b4ac-690431798e9e\") "
Jul 28 18:04:27.313980 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.313952    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-service-ca" (OuterVolumeSpecName: "service-ca") pod "5f63baaa-e74f-4493-b4ac-690431798e9e" (UID: "5f63baaa-e74f-4493-b4ac-690431798e9e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:04:27.314039 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.313977    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-console-config" (OuterVolumeSpecName: "console-config") pod "5f63baaa-e74f-4493-b4ac-690431798e9e" (UID: "5f63baaa-e74f-4493-b4ac-690431798e9e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:04:27.314039 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.314008    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5f63baaa-e74f-4493-b4ac-690431798e9e" (UID: "5f63baaa-e74f-4493-b4ac-690431798e9e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:04:27.314282 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.314253    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5f63baaa-e74f-4493-b4ac-690431798e9e" (UID: "5f63baaa-e74f-4493-b4ac-690431798e9e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:04:27.315800 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.315773    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f63baaa-e74f-4493-b4ac-690431798e9e-kube-api-access-k67sp" (OuterVolumeSpecName: "kube-api-access-k67sp") pod "5f63baaa-e74f-4493-b4ac-690431798e9e" (UID: "5f63baaa-e74f-4493-b4ac-690431798e9e"). InnerVolumeSpecName "kube-api-access-k67sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:04:27.315800 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.315789    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f63baaa-e74f-4493-b4ac-690431798e9e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5f63baaa-e74f-4493-b4ac-690431798e9e" (UID: "5f63baaa-e74f-4493-b4ac-690431798e9e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:04:27.316042 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.316023    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f63baaa-e74f-4493-b4ac-690431798e9e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5f63baaa-e74f-4493-b4ac-690431798e9e" (UID: "5f63baaa-e74f-4493-b4ac-690431798e9e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:04:27.414648 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.414611    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-console-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:04:27.414648 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.414642    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f63baaa-e74f-4493-b4ac-690431798e9e-console-oauth-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:04:27.414648 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.414652    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f63baaa-e74f-4493-b4ac-690431798e9e-console-serving-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:04:27.414900 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.414660    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-trusted-ca-bundle\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:04:27.414900 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.414670    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-oauth-serving-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:04:27.414900 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.414679    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f63baaa-e74f-4493-b4ac-690431798e9e-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:04:27.414900 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:27.414688    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k67sp\" (UniqueName: \"kubernetes.io/projected/5f63baaa-e74f-4493-b4ac-690431798e9e-kube-api-access-k67sp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:04:28.017997 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:28.017960    2564 generic.go:356] "Generic (PLEG): container finished" podID="5f63baaa-e74f-4493-b4ac-690431798e9e" containerID="ade9ae6ef4140981311024c92ac98cfb5f115ef0d0481fbe21824fc48ffaf699" exitCode=0
Jul 28 18:04:28.017997 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:28.018003    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7db67d646-q65zz"
Jul 28 18:04:28.018423 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:28.018049    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7db67d646-q65zz" event={"ID":"5f63baaa-e74f-4493-b4ac-690431798e9e","Type":"ContainerDied","Data":"ade9ae6ef4140981311024c92ac98cfb5f115ef0d0481fbe21824fc48ffaf699"}
Jul 28 18:04:28.018423 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:28.018092    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7db67d646-q65zz" event={"ID":"5f63baaa-e74f-4493-b4ac-690431798e9e","Type":"ContainerDied","Data":"0c773a7ffabf1e5e57e0b10a7635914ea691942dc58cec064c208937c6aa134c"}
Jul 28 18:04:28.018423 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:28.018113    2564 scope.go:117] "RemoveContainer" containerID="ade9ae6ef4140981311024c92ac98cfb5f115ef0d0481fbe21824fc48ffaf699"
Jul 28 18:04:28.027766 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:28.027746    2564 scope.go:117] "RemoveContainer" containerID="ade9ae6ef4140981311024c92ac98cfb5f115ef0d0481fbe21824fc48ffaf699"
Jul 28 18:04:28.028145 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:04:28.028118    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade9ae6ef4140981311024c92ac98cfb5f115ef0d0481fbe21824fc48ffaf699\": container with ID starting with ade9ae6ef4140981311024c92ac98cfb5f115ef0d0481fbe21824fc48ffaf699 not found: ID does not exist" containerID="ade9ae6ef4140981311024c92ac98cfb5f115ef0d0481fbe21824fc48ffaf699"
Jul 28 18:04:28.028253 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:28.028150    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade9ae6ef4140981311024c92ac98cfb5f115ef0d0481fbe21824fc48ffaf699"} err="failed to get container status \"ade9ae6ef4140981311024c92ac98cfb5f115ef0d0481fbe21824fc48ffaf699\": rpc error: code = NotFound desc = could not find container \"ade9ae6ef4140981311024c92ac98cfb5f115ef0d0481fbe21824fc48ffaf699\": container with ID starting with ade9ae6ef4140981311024c92ac98cfb5f115ef0d0481fbe21824fc48ffaf699 not found: ID does not exist"
Jul 28 18:04:28.044052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:28.044020    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7db67d646-q65zz"]
Jul 28 18:04:28.050514 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:28.050490    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7db67d646-q65zz"]
Jul 28 18:04:29.215213 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:04:29.215179    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f63baaa-e74f-4493-b4ac-690431798e9e" path="/var/lib/kubelet/pods/5f63baaa-e74f-4493-b4ac-690431798e9e/volumes"
Jul 28 18:05:04.270419 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.270382    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7474f96574-nfflq"]
Jul 28 18:05:04.270864 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.270668    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f63baaa-e74f-4493-b4ac-690431798e9e" containerName="console"
Jul 28 18:05:04.270864 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.270681    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f63baaa-e74f-4493-b4ac-690431798e9e" containerName="console"
Jul 28 18:05:04.270864 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.270747    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f63baaa-e74f-4493-b4ac-690431798e9e" containerName="console"
Jul 28 18:05:04.274577 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.274528    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.288773 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.288746    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7474f96574-nfflq"]
Jul 28 18:05:04.348838 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.348789    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-serving-cert\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.349046 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.348862    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-service-ca\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.349046 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.348899    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-oauth-config\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.349046 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.348955    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-trusted-ca-bundle\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.349046 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.349000    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-oauth-serving-cert\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.349046 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.349033    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sslvn\" (UniqueName: \"kubernetes.io/projected/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-kube-api-access-sslvn\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.349263 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.349095    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-config\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.449627 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.449594    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-serving-cert\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.449627 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.449634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-service-ca\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.449903 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.449651    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-oauth-config\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.449903 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.449681    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-trusted-ca-bundle\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.449903 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.449860    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-oauth-serving-cert\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.450058 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.449913    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sslvn\" (UniqueName: \"kubernetes.io/projected/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-kube-api-access-sslvn\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.450058 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.449949    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-config\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.450519 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.450491    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-service-ca\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.450639 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.450587    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-config\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.450742 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.450715    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-oauth-serving-cert\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.450827 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.450741    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-trusted-ca-bundle\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.452149 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.452131    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-oauth-config\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.452251 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.452229    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-serving-cert\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.460183 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.460159    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sslvn\" (UniqueName: \"kubernetes.io/projected/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-kube-api-access-sslvn\") pod \"console-7474f96574-nfflq\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") " pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.585171 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.585079    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:04.709761 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:04.709728    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7474f96574-nfflq"]
Jul 28 18:05:04.712678 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:05:04.712653    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2cfd53_d01f_4f60_b1f1_f0c2ded4b63a.slice/crio-e0b340948d1fc653e218a0c6625bcff46797963b3134daef60d5be033c04fb70 WatchSource:0}: Error finding container e0b340948d1fc653e218a0c6625bcff46797963b3134daef60d5be033c04fb70: Status 404 returned error can't find the container with id e0b340948d1fc653e218a0c6625bcff46797963b3134daef60d5be033c04fb70
Jul 28 18:05:05.144966 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:05.144927    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7474f96574-nfflq" event={"ID":"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a","Type":"ContainerStarted","Data":"f0b252a860d541b7d5e9650a8570820468b4de8bcccdf03d3a117eb829661afb"}
Jul 28 18:05:05.145114 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:05.144979    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7474f96574-nfflq" event={"ID":"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a","Type":"ContainerStarted","Data":"e0b340948d1fc653e218a0c6625bcff46797963b3134daef60d5be033c04fb70"}
Jul 28 18:05:05.170489 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:05.170433    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7474f96574-nfflq" podStartSLOduration=1.170413157 podStartE2EDuration="1.170413157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:05:05.169750129 +0000 UTC m=+240.485252241" watchObservedRunningTime="2026-07-28 18:05:05.170413157 +0000 UTC m=+240.485915272"
Jul 28 18:05:14.585660 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:14.585622    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:14.586165 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:14.585677    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:14.590296 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:14.590273    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:15.181972 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:15.181947    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:05:15.247873 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:15.247798    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f69cbc78f-fm2xv"]
Jul 28 18:05:40.265980 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.265918    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f69cbc78f-fm2xv" podUID="4d002201-8428-4e79-94f0-cd5fde7e44aa" containerName="console" containerID="cri-o://d4d40a7ef79071a31e13200c64538b92ec5c41ad678d3a381b1666b1bed99cda" gracePeriod=15
Jul 28 18:05:40.502257 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.502230    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:05:40.603321 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.603255    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-serving-cert\") pod \"4d002201-8428-4e79-94f0-cd5fde7e44aa\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") "
Jul 28 18:05:40.603321 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.603287    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-config\") pod \"4d002201-8428-4e79-94f0-cd5fde7e44aa\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") "
Jul 28 18:05:40.603321 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.603310    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mq9l\" (UniqueName: \"kubernetes.io/projected/4d002201-8428-4e79-94f0-cd5fde7e44aa-kube-api-access-5mq9l\") pod \"4d002201-8428-4e79-94f0-cd5fde7e44aa\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") "
Jul 28 18:05:40.603551 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.603358    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-oauth-serving-cert\") pod \"4d002201-8428-4e79-94f0-cd5fde7e44aa\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") "
Jul 28 18:05:40.603551 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.603398    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-service-ca\") pod \"4d002201-8428-4e79-94f0-cd5fde7e44aa\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") "
Jul 28 18:05:40.603551 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.603441    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-trusted-ca-bundle\") pod \"4d002201-8428-4e79-94f0-cd5fde7e44aa\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") "
Jul 28 18:05:40.603705 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.603572    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-oauth-config\") pod \"4d002201-8428-4e79-94f0-cd5fde7e44aa\" (UID: \"4d002201-8428-4e79-94f0-cd5fde7e44aa\") "
Jul 28 18:05:40.603760 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.603721    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-config" (OuterVolumeSpecName: "console-config") pod "4d002201-8428-4e79-94f0-cd5fde7e44aa" (UID: "4d002201-8428-4e79-94f0-cd5fde7e44aa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:05:40.603835 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.603758    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-service-ca" (OuterVolumeSpecName: "service-ca") pod "4d002201-8428-4e79-94f0-cd5fde7e44aa" (UID: "4d002201-8428-4e79-94f0-cd5fde7e44aa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:05:40.603835 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.603825    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:05:40.603919 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.603833    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4d002201-8428-4e79-94f0-cd5fde7e44aa" (UID: "4d002201-8428-4e79-94f0-cd5fde7e44aa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:05:40.603919 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.603868    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4d002201-8428-4e79-94f0-cd5fde7e44aa" (UID: "4d002201-8428-4e79-94f0-cd5fde7e44aa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:05:40.605397 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.605365    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4d002201-8428-4e79-94f0-cd5fde7e44aa" (UID: "4d002201-8428-4e79-94f0-cd5fde7e44aa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:05:40.605572 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.605552    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4d002201-8428-4e79-94f0-cd5fde7e44aa" (UID: "4d002201-8428-4e79-94f0-cd5fde7e44aa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:05:40.605626 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.605569    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d002201-8428-4e79-94f0-cd5fde7e44aa-kube-api-access-5mq9l" (OuterVolumeSpecName: "kube-api-access-5mq9l") pod "4d002201-8428-4e79-94f0-cd5fde7e44aa" (UID: "4d002201-8428-4e79-94f0-cd5fde7e44aa"). InnerVolumeSpecName "kube-api-access-5mq9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:05:40.705068 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.705032    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-oauth-serving-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:05:40.705068 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.705064    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:05:40.705068 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.705074    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d002201-8428-4e79-94f0-cd5fde7e44aa-trusted-ca-bundle\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:05:40.705231 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.705083    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-oauth-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:05:40.705231 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.705092    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d002201-8428-4e79-94f0-cd5fde7e44aa-console-serving-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:05:40.705231 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:40.705100    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mq9l\" (UniqueName: \"kubernetes.io/projected/4d002201-8428-4e79-94f0-cd5fde7e44aa-kube-api-access-5mq9l\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:05:41.264503 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:41.264472    2564 generic.go:356] "Generic (PLEG): container finished" podID="4d002201-8428-4e79-94f0-cd5fde7e44aa" containerID="d4d40a7ef79071a31e13200c64538b92ec5c41ad678d3a381b1666b1bed99cda" exitCode=0
Jul 28 18:05:41.264664 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:41.264520    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f69cbc78f-fm2xv"
Jul 28 18:05:41.264664 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:41.264532    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f69cbc78f-fm2xv" event={"ID":"4d002201-8428-4e79-94f0-cd5fde7e44aa","Type":"ContainerDied","Data":"d4d40a7ef79071a31e13200c64538b92ec5c41ad678d3a381b1666b1bed99cda"}
Jul 28 18:05:41.264664 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:41.264572    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f69cbc78f-fm2xv" event={"ID":"4d002201-8428-4e79-94f0-cd5fde7e44aa","Type":"ContainerDied","Data":"9cc4801d359a4c45a18d73e4f6adc561fa7fee5208739e8e3817990ead35cb59"}
Jul 28 18:05:41.264664 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:41.264593    2564 scope.go:117] "RemoveContainer" containerID="d4d40a7ef79071a31e13200c64538b92ec5c41ad678d3a381b1666b1bed99cda"
Jul 28 18:05:41.272400 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:41.272265    2564 scope.go:117] "RemoveContainer" containerID="d4d40a7ef79071a31e13200c64538b92ec5c41ad678d3a381b1666b1bed99cda"
Jul 28 18:05:41.272600 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:05:41.272505    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d40a7ef79071a31e13200c64538b92ec5c41ad678d3a381b1666b1bed99cda\": container with ID starting with d4d40a7ef79071a31e13200c64538b92ec5c41ad678d3a381b1666b1bed99cda not found: ID does not exist" containerID="d4d40a7ef79071a31e13200c64538b92ec5c41ad678d3a381b1666b1bed99cda"
Jul 28 18:05:41.272600 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:41.272529    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d40a7ef79071a31e13200c64538b92ec5c41ad678d3a381b1666b1bed99cda"} err="failed to get container status \"d4d40a7ef79071a31e13200c64538b92ec5c41ad678d3a381b1666b1bed99cda\": rpc error: code = NotFound desc = could not find container \"d4d40a7ef79071a31e13200c64538b92ec5c41ad678d3a381b1666b1bed99cda\": container with ID starting with d4d40a7ef79071a31e13200c64538b92ec5c41ad678d3a381b1666b1bed99cda not found: ID does not exist"
Jul 28 18:05:41.285185 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:41.285163    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f69cbc78f-fm2xv"]
Jul 28 18:05:41.289342 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:41.289320    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f69cbc78f-fm2xv"]
Jul 28 18:05:43.219141 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:05:43.219109    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d002201-8428-4e79-94f0-cd5fde7e44aa" path="/var/lib/kubelet/pods/4d002201-8428-4e79-94f0-cd5fde7e44aa/volumes"
Jul 28 18:06:05.088622 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:05.088592    2564 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 18:06:20.514777 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.514743    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s"]
Jul 28 18:06:20.517088 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.514988    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d002201-8428-4e79-94f0-cd5fde7e44aa" containerName="console"
Jul 28 18:06:20.517088 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.514999    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d002201-8428-4e79-94f0-cd5fde7e44aa" containerName="console"
Jul 28 18:06:20.517088 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.515045    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d002201-8428-4e79-94f0-cd5fde7e44aa" containerName="console"
Jul 28 18:06:20.517934 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.517918    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s"
Jul 28 18:06:20.520863 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.520840    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-87w5r\""
Jul 28 18:06:20.521044 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.520867    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 18:06:20.521044 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.520867    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 18:06:20.530042 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.530019    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s"]
Jul 28 18:06:20.573657 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.573620    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56spl\" (UniqueName: \"kubernetes.io/projected/053756b7-2040-430d-b260-c25ab5407609-kube-api-access-56spl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s\" (UID: \"053756b7-2040-430d-b260-c25ab5407609\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s"
Jul 28 18:06:20.573849 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.573690    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/053756b7-2040-430d-b260-c25ab5407609-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s\" (UID: \"053756b7-2040-430d-b260-c25ab5407609\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s"
Jul 28 18:06:20.573849 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.573734    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/053756b7-2040-430d-b260-c25ab5407609-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s\" (UID: \"053756b7-2040-430d-b260-c25ab5407609\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s"
Jul 28 18:06:20.674932 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.674899    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56spl\" (UniqueName: \"kubernetes.io/projected/053756b7-2040-430d-b260-c25ab5407609-kube-api-access-56spl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s\" (UID: \"053756b7-2040-430d-b260-c25ab5407609\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s"
Jul 28 18:06:20.674932 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.674937    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/053756b7-2040-430d-b260-c25ab5407609-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s\" (UID: \"053756b7-2040-430d-b260-c25ab5407609\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s"
Jul 28 18:06:20.675093 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.674964    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/053756b7-2040-430d-b260-c25ab5407609-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s\" (UID: \"053756b7-2040-430d-b260-c25ab5407609\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s"
Jul 28 18:06:20.675296 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.675276    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/053756b7-2040-430d-b260-c25ab5407609-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s\" (UID: \"053756b7-2040-430d-b260-c25ab5407609\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s"
Jul 28 18:06:20.675334 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.675308    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/053756b7-2040-430d-b260-c25ab5407609-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s\" (UID: \"053756b7-2040-430d-b260-c25ab5407609\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s"
Jul 28 18:06:20.686535 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.686514    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56spl\" (UniqueName: \"kubernetes.io/projected/053756b7-2040-430d-b260-c25ab5407609-kube-api-access-56spl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s\" (UID: \"053756b7-2040-430d-b260-c25ab5407609\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s"
Jul 28 18:06:20.827038 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.826941    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s"
Jul 28 18:06:20.961384 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.961355    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s"]
Jul 28 18:06:20.965274 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:06:20.965247    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053756b7_2040_430d_b260_c25ab5407609.slice/crio-e952056b84f92d1a1c7c21bfa978bbba96bd50eb97a7f0dba24d6bd8acd11b5d WatchSource:0}: Error finding container e952056b84f92d1a1c7c21bfa978bbba96bd50eb97a7f0dba24d6bd8acd11b5d: Status 404 returned error can't find the container with id e952056b84f92d1a1c7c21bfa978bbba96bd50eb97a7f0dba24d6bd8acd11b5d
Jul 28 18:06:20.967206 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:20.967189    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:06:21.394218 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:21.394186    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s" event={"ID":"053756b7-2040-430d-b260-c25ab5407609","Type":"ContainerStarted","Data":"e952056b84f92d1a1c7c21bfa978bbba96bd50eb97a7f0dba24d6bd8acd11b5d"}
Jul 28 18:06:28.420435 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:28.420397    2564 generic.go:356] "Generic (PLEG): container finished" podID="053756b7-2040-430d-b260-c25ab5407609" containerID="1a38cc1798800dfa3f9c816da5d442d37bb768c294edb017e4b8a7c35e702a71" exitCode=0
Jul 28 18:06:28.420794 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:28.420442    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s" event={"ID":"053756b7-2040-430d-b260-c25ab5407609","Type":"ContainerDied","Data":"1a38cc1798800dfa3f9c816da5d442d37bb768c294edb017e4b8a7c35e702a71"}
Jul 28 18:06:29.719321 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:29.719283    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9sc6p"]
Jul 28 18:06:29.722215 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:29.722196    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9sc6p"
Jul 28 18:06:29.725417 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:29.725396    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 18:06:29.726700 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:29.726683    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-p7kds\""
Jul 28 18:06:29.726790 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:29.726685    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 18:06:29.734974 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:29.734952    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9sc6p"]
Jul 28 18:06:29.748045 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:29.748022    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5j6k\" (UniqueName: \"kubernetes.io/projected/98217b11-8bb6-442b-a12f-aa2eb9fe62df-kube-api-access-n5j6k\") pod \"cert-manager-55b4d954d5-9sc6p\" (UID: \"98217b11-8bb6-442b-a12f-aa2eb9fe62df\") " pod="cert-manager/cert-manager-55b4d954d5-9sc6p"
Jul 28 18:06:29.748136 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:29.748053    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98217b11-8bb6-442b-a12f-aa2eb9fe62df-bound-sa-token\") pod \"cert-manager-55b4d954d5-9sc6p\" (UID: \"98217b11-8bb6-442b-a12f-aa2eb9fe62df\") " pod="cert-manager/cert-manager-55b4d954d5-9sc6p"
Jul 28 18:06:29.848637 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:29.848598    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98217b11-8bb6-442b-a12f-aa2eb9fe62df-bound-sa-token\") pod \"cert-manager-55b4d954d5-9sc6p\" (UID: \"98217b11-8bb6-442b-a12f-aa2eb9fe62df\") " pod="cert-manager/cert-manager-55b4d954d5-9sc6p"
Jul 28 18:06:29.848835 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:29.848691    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5j6k\" (UniqueName: \"kubernetes.io/projected/98217b11-8bb6-442b-a12f-aa2eb9fe62df-kube-api-access-n5j6k\") pod \"cert-manager-55b4d954d5-9sc6p\" (UID: \"98217b11-8bb6-442b-a12f-aa2eb9fe62df\") " pod="cert-manager/cert-manager-55b4d954d5-9sc6p"
Jul 28 18:06:29.860588 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:29.860560    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98217b11-8bb6-442b-a12f-aa2eb9fe62df-bound-sa-token\") pod \"cert-manager-55b4d954d5-9sc6p\" (UID: \"98217b11-8bb6-442b-a12f-aa2eb9fe62df\") " pod="cert-manager/cert-manager-55b4d954d5-9sc6p"
Jul 28 18:06:29.860846 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:29.860796    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5j6k\" (UniqueName: \"kubernetes.io/projected/98217b11-8bb6-442b-a12f-aa2eb9fe62df-kube-api-access-n5j6k\") pod \"cert-manager-55b4d954d5-9sc6p\" (UID: \"98217b11-8bb6-442b-a12f-aa2eb9fe62df\") " pod="cert-manager/cert-manager-55b4d954d5-9sc6p"
Jul 28 18:06:30.030968 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:30.030934    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9sc6p"
Jul 28 18:06:30.536689 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:30.536638    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9sc6p"]
Jul 28 18:06:30.539135 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:06:30.539091    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98217b11_8bb6_442b_a12f_aa2eb9fe62df.slice/crio-2d3c6696b5767e438dbc3c87d64f1e5ae10f286bcd10a31e6a6e05884297a369 WatchSource:0}: Error finding container 2d3c6696b5767e438dbc3c87d64f1e5ae10f286bcd10a31e6a6e05884297a369: Status 404 returned error can't find the container with id 2d3c6696b5767e438dbc3c87d64f1e5ae10f286bcd10a31e6a6e05884297a369
Jul 28 18:06:31.436861 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:31.436824    2564 generic.go:356] "Generic (PLEG): container finished" podID="053756b7-2040-430d-b260-c25ab5407609" containerID="b7c8c7764e1c2558f0ef8e805548233b608bfb4bf6a946811a92106063a7bf43" exitCode=0
Jul 28 18:06:31.437332 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:31.436932    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s" event={"ID":"053756b7-2040-430d-b260-c25ab5407609","Type":"ContainerDied","Data":"b7c8c7764e1c2558f0ef8e805548233b608bfb4bf6a946811a92106063a7bf43"}
Jul 28 18:06:31.438448 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:31.438402    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9sc6p" event={"ID":"98217b11-8bb6-442b-a12f-aa2eb9fe62df","Type":"ContainerStarted","Data":"2d3c6696b5767e438dbc3c87d64f1e5ae10f286bcd10a31e6a6e05884297a369"}
Jul 28 18:06:34.454504 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:34.454466    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9sc6p" event={"ID":"98217b11-8bb6-442b-a12f-aa2eb9fe62df","Type":"ContainerStarted","Data":"2bf9a2ceb5db70a826fccc556cba01b286a97f3c848de9f9b36def5f15ab7017"}
Jul 28 18:06:38.475125 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:38.475088    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s" event={"ID":"053756b7-2040-430d-b260-c25ab5407609","Type":"ContainerStarted","Data":"a154ca72ef6a48b4b780b522599ca6485cba29a2d52672018979bb7949ff942f"}
Jul 28 18:06:38.499282 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:38.499236    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-9sc6p" podStartSLOduration=6.039238981 podStartE2EDuration="9.49921933s" totalImagesPullingTime="3.459980349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:06:34.477171806 +0000 UTC m=+329.792673946" watchObservedRunningTime="2026-07-28 18:06:38.49921933 +0000 UTC m=+333.814721443"
Jul 28 18:06:38.499584 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:38.499552    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s" podStartSLOduration=2.667628063 podStartE2EDuration="18.499546869s" totalImagesPullingTime="15.831918806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:06:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:06:38.498900548 +0000 UTC m=+333.814402689" watchObservedRunningTime="2026-07-28 18:06:38.499546869 +0000 UTC m=+333.815048981"
Jul 28 18:06:39.480508 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:39.480472    2564 generic.go:356] "Generic (PLEG): container finished" podID="053756b7-2040-430d-b260-c25ab5407609" containerID="a154ca72ef6a48b4b780b522599ca6485cba29a2d52672018979bb7949ff942f" exitCode=0
Jul 28 18:06:39.480896 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:39.480514    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s" event={"ID":"053756b7-2040-430d-b260-c25ab5407609","Type":"ContainerDied","Data":"a154ca72ef6a48b4b780b522599ca6485cba29a2d52672018979bb7949ff942f"}
Jul 28 18:06:40.608382 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:40.608359    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s"
Jul 28 18:06:40.738918 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:40.738803    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56spl\" (UniqueName: \"kubernetes.io/projected/053756b7-2040-430d-b260-c25ab5407609-kube-api-access-56spl\") pod \"053756b7-2040-430d-b260-c25ab5407609\" (UID: \"053756b7-2040-430d-b260-c25ab5407609\") "
Jul 28 18:06:40.738918 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:40.738865    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/053756b7-2040-430d-b260-c25ab5407609-bundle\") pod \"053756b7-2040-430d-b260-c25ab5407609\" (UID: \"053756b7-2040-430d-b260-c25ab5407609\") "
Jul 28 18:06:40.738918 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:40.738891    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/053756b7-2040-430d-b260-c25ab5407609-util\") pod \"053756b7-2040-430d-b260-c25ab5407609\" (UID: \"053756b7-2040-430d-b260-c25ab5407609\") "
Jul 28 18:06:40.739264 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:40.739234    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053756b7-2040-430d-b260-c25ab5407609-bundle" (OuterVolumeSpecName: "bundle") pod "053756b7-2040-430d-b260-c25ab5407609" (UID: "053756b7-2040-430d-b260-c25ab5407609"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:06:40.741058 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:40.741027    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053756b7-2040-430d-b260-c25ab5407609-kube-api-access-56spl" (OuterVolumeSpecName: "kube-api-access-56spl") pod "053756b7-2040-430d-b260-c25ab5407609" (UID: "053756b7-2040-430d-b260-c25ab5407609"). InnerVolumeSpecName "kube-api-access-56spl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:06:40.743913 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:40.743893    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053756b7-2040-430d-b260-c25ab5407609-util" (OuterVolumeSpecName: "util") pod "053756b7-2040-430d-b260-c25ab5407609" (UID: "053756b7-2040-430d-b260-c25ab5407609"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:06:40.839534 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:40.839499    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56spl\" (UniqueName: \"kubernetes.io/projected/053756b7-2040-430d-b260-c25ab5407609-kube-api-access-56spl\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:06:40.839534 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:40.839526    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/053756b7-2040-430d-b260-c25ab5407609-bundle\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:06:40.839534 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:40.839537    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/053756b7-2040-430d-b260-c25ab5407609-util\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:06:41.489955 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:41.489874    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s" event={"ID":"053756b7-2040-430d-b260-c25ab5407609","Type":"ContainerDied","Data":"e952056b84f92d1a1c7c21bfa978bbba96bd50eb97a7f0dba24d6bd8acd11b5d"}
Jul 28 18:06:41.489955 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:41.489915    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7fv8s"
Jul 28 18:06:41.489955 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:06:41.489944    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e952056b84f92d1a1c7c21bfa978bbba96bd50eb97a7f0dba24d6bd8acd11b5d"
Jul 28 18:07:02.058460 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.058376    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"]
Jul 28 18:07:02.058909 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.058609    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="053756b7-2040-430d-b260-c25ab5407609" containerName="util"
Jul 28 18:07:02.058909 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.058619    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="053756b7-2040-430d-b260-c25ab5407609" containerName="util"
Jul 28 18:07:02.058909 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.058629    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="053756b7-2040-430d-b260-c25ab5407609" containerName="pull"
Jul 28 18:07:02.058909 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.058635    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="053756b7-2040-430d-b260-c25ab5407609" containerName="pull"
Jul 28 18:07:02.058909 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.058641    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="053756b7-2040-430d-b260-c25ab5407609" containerName="extract"
Jul 28 18:07:02.058909 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.058646    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="053756b7-2040-430d-b260-c25ab5407609" containerName="extract"
Jul 28 18:07:02.058909 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.058690    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="053756b7-2040-430d-b260-c25ab5407609" containerName="extract"
Jul 28 18:07:02.060635 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.060620    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"
Jul 28 18:07:02.064900 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.064876    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 18:07:02.064900 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.064898    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 18:07:02.065114 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.064928    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-d6fnj\""
Jul 28 18:07:02.065114 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.064969    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 18:07:02.065114 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.064930    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 18:07:02.065114 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.064876    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 18:07:02.075293 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.075271    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"]
Jul 28 18:07:02.187863 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.187829    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/72ad3238-44e8-44f8-b1da-3050b2256bf6-metrics-cert\") pod \"lws-controller-manager-84df6f5b9b-r6fst\" (UID: \"72ad3238-44e8-44f8-b1da-3050b2256bf6\") " pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"
Jul 28 18:07:02.187863 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.187869    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8z44\" (UniqueName: \"kubernetes.io/projected/72ad3238-44e8-44f8-b1da-3050b2256bf6-kube-api-access-v8z44\") pod \"lws-controller-manager-84df6f5b9b-r6fst\" (UID: \"72ad3238-44e8-44f8-b1da-3050b2256bf6\") " pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"
Jul 28 18:07:02.188080 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.187892    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72ad3238-44e8-44f8-b1da-3050b2256bf6-cert\") pod \"lws-controller-manager-84df6f5b9b-r6fst\" (UID: \"72ad3238-44e8-44f8-b1da-3050b2256bf6\") " pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"
Jul 28 18:07:02.188080 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.187974    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/72ad3238-44e8-44f8-b1da-3050b2256bf6-manager-config\") pod \"lws-controller-manager-84df6f5b9b-r6fst\" (UID: \"72ad3238-44e8-44f8-b1da-3050b2256bf6\") " pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"
Jul 28 18:07:02.289347 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.289314    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72ad3238-44e8-44f8-b1da-3050b2256bf6-cert\") pod \"lws-controller-manager-84df6f5b9b-r6fst\" (UID: \"72ad3238-44e8-44f8-b1da-3050b2256bf6\") " pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"
Jul 28 18:07:02.289531 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.289370    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/72ad3238-44e8-44f8-b1da-3050b2256bf6-manager-config\") pod \"lws-controller-manager-84df6f5b9b-r6fst\" (UID: \"72ad3238-44e8-44f8-b1da-3050b2256bf6\") " pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"
Jul 28 18:07:02.289531 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.289411    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/72ad3238-44e8-44f8-b1da-3050b2256bf6-metrics-cert\") pod \"lws-controller-manager-84df6f5b9b-r6fst\" (UID: \"72ad3238-44e8-44f8-b1da-3050b2256bf6\") " pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"
Jul 28 18:07:02.289531 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.289436    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8z44\" (UniqueName: \"kubernetes.io/projected/72ad3238-44e8-44f8-b1da-3050b2256bf6-kube-api-access-v8z44\") pod \"lws-controller-manager-84df6f5b9b-r6fst\" (UID: \"72ad3238-44e8-44f8-b1da-3050b2256bf6\") " pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"
Jul 28 18:07:02.290142 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.290111    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/72ad3238-44e8-44f8-b1da-3050b2256bf6-manager-config\") pod \"lws-controller-manager-84df6f5b9b-r6fst\" (UID: \"72ad3238-44e8-44f8-b1da-3050b2256bf6\") " pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"
Jul 28 18:07:02.291840 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.291817    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/72ad3238-44e8-44f8-b1da-3050b2256bf6-metrics-cert\") pod \"lws-controller-manager-84df6f5b9b-r6fst\" (UID: \"72ad3238-44e8-44f8-b1da-3050b2256bf6\") " pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"
Jul 28 18:07:02.291840 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.291831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72ad3238-44e8-44f8-b1da-3050b2256bf6-cert\") pod \"lws-controller-manager-84df6f5b9b-r6fst\" (UID: \"72ad3238-44e8-44f8-b1da-3050b2256bf6\") " pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"
Jul 28 18:07:02.305266 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.305238    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8z44\" (UniqueName: \"kubernetes.io/projected/72ad3238-44e8-44f8-b1da-3050b2256bf6-kube-api-access-v8z44\") pod \"lws-controller-manager-84df6f5b9b-r6fst\" (UID: \"72ad3238-44e8-44f8-b1da-3050b2256bf6\") " pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"
Jul 28 18:07:02.368954 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.368867    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"
Jul 28 18:07:02.498563 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.498525    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"]
Jul 28 18:07:02.501388 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:07:02.501341    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ad3238_44e8_44f8_b1da_3050b2256bf6.slice/crio-2eb813bfc010a6097af69fe9e8f4542d4ff1edae41f108c2c36beb3b138f23d0 WatchSource:0}: Error finding container 2eb813bfc010a6097af69fe9e8f4542d4ff1edae41f108c2c36beb3b138f23d0: Status 404 returned error can't find the container with id 2eb813bfc010a6097af69fe9e8f4542d4ff1edae41f108c2c36beb3b138f23d0
Jul 28 18:07:02.568803 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:02.568771    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst" event={"ID":"72ad3238-44e8-44f8-b1da-3050b2256bf6","Type":"ContainerStarted","Data":"2eb813bfc010a6097af69fe9e8f4542d4ff1edae41f108c2c36beb3b138f23d0"}
Jul 28 18:07:04.577487 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:04.577450    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst" event={"ID":"72ad3238-44e8-44f8-b1da-3050b2256bf6","Type":"ContainerStarted","Data":"81c8c1cbc3f1c83fd9d4fac3d55507979f68bfca2f9e5403ef7b966e07772c11"}
Jul 28 18:07:04.577975 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:04.577583    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"
Jul 28 18:07:04.598993 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:04.598951    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst" podStartSLOduration=0.983911234 podStartE2EDuration="2.598937255s" totalImagesPullingTime="1.615026021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:07:04.598280605 +0000 UTC m=+359.913782740" watchObservedRunningTime="2026-07-28 18:07:04.598937255 +0000 UTC m=+359.914439367"
Jul 28 18:07:08.356534 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.356496    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"]
Jul 28 18:07:08.359322 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.359305    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.361915 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.361892    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 18:07:08.361915 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.361902    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 18:07:08.362508 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.362489    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 18:07:08.362594 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.362510    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sz7wn\""
Jul 28 18:07:08.362594 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.362551    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 18:07:08.393894 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.393840    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"]
Jul 28 18:07:08.437482 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.437450    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbx85\" (UniqueName: \"kubernetes.io/projected/fde36536-6bc4-4cd7-8450-230e73b5ba59-kube-api-access-qbx85\") pod \"opendatahub-operator-controller-manager-7795fd7c74-xkjx4\" (UID: \"fde36536-6bc4-4cd7-8450-230e73b5ba59\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.437645 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.437507    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fde36536-6bc4-4cd7-8450-230e73b5ba59-manifests\") pod \"opendatahub-operator-controller-manager-7795fd7c74-xkjx4\" (UID: \"fde36536-6bc4-4cd7-8450-230e73b5ba59\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.437645 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.437526    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fde36536-6bc4-4cd7-8450-230e73b5ba59-webhook-cert\") pod \"opendatahub-operator-controller-manager-7795fd7c74-xkjx4\" (UID: \"fde36536-6bc4-4cd7-8450-230e73b5ba59\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.437645 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.437585    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fde36536-6bc4-4cd7-8450-230e73b5ba59-tmp\") pod \"opendatahub-operator-controller-manager-7795fd7c74-xkjx4\" (UID: \"fde36536-6bc4-4cd7-8450-230e73b5ba59\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.437645 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.437608    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fde36536-6bc4-4cd7-8450-230e73b5ba59-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7795fd7c74-xkjx4\" (UID: \"fde36536-6bc4-4cd7-8450-230e73b5ba59\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.538923 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.538884    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fde36536-6bc4-4cd7-8450-230e73b5ba59-webhook-cert\") pod \"opendatahub-operator-controller-manager-7795fd7c74-xkjx4\" (UID: \"fde36536-6bc4-4cd7-8450-230e73b5ba59\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.538923 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.538927    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fde36536-6bc4-4cd7-8450-230e73b5ba59-tmp\") pod \"opendatahub-operator-controller-manager-7795fd7c74-xkjx4\" (UID: \"fde36536-6bc4-4cd7-8450-230e73b5ba59\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.539111 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.538963    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fde36536-6bc4-4cd7-8450-230e73b5ba59-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7795fd7c74-xkjx4\" (UID: \"fde36536-6bc4-4cd7-8450-230e73b5ba59\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.539111 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.539001    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbx85\" (UniqueName: \"kubernetes.io/projected/fde36536-6bc4-4cd7-8450-230e73b5ba59-kube-api-access-qbx85\") pod \"opendatahub-operator-controller-manager-7795fd7c74-xkjx4\" (UID: \"fde36536-6bc4-4cd7-8450-230e73b5ba59\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.539206 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.539170    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fde36536-6bc4-4cd7-8450-230e73b5ba59-manifests\") pod \"opendatahub-operator-controller-manager-7795fd7c74-xkjx4\" (UID: \"fde36536-6bc4-4cd7-8450-230e73b5ba59\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.539490 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.539470    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fde36536-6bc4-4cd7-8450-230e73b5ba59-manifests\") pod \"opendatahub-operator-controller-manager-7795fd7c74-xkjx4\" (UID: \"fde36536-6bc4-4cd7-8450-230e73b5ba59\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.541395 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.541364    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fde36536-6bc4-4cd7-8450-230e73b5ba59-tmp\") pod \"opendatahub-operator-controller-manager-7795fd7c74-xkjx4\" (UID: \"fde36536-6bc4-4cd7-8450-230e73b5ba59\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.541500 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.541451    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fde36536-6bc4-4cd7-8450-230e73b5ba59-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7795fd7c74-xkjx4\" (UID: \"fde36536-6bc4-4cd7-8450-230e73b5ba59\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.541500 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.541479    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fde36536-6bc4-4cd7-8450-230e73b5ba59-webhook-cert\") pod \"opendatahub-operator-controller-manager-7795fd7c74-xkjx4\" (UID: \"fde36536-6bc4-4cd7-8450-230e73b5ba59\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.547868 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.547847    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbx85\" (UniqueName: \"kubernetes.io/projected/fde36536-6bc4-4cd7-8450-230e73b5ba59-kube-api-access-qbx85\") pod \"opendatahub-operator-controller-manager-7795fd7c74-xkjx4\" (UID: \"fde36536-6bc4-4cd7-8450-230e73b5ba59\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.669592 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.669486    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:08.811959 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:08.811916    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"]
Jul 28 18:07:08.816684 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:07:08.816652    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde36536_6bc4_4cd7_8450_230e73b5ba59.slice/crio-372966e8781bbf1391b7a81c167e02bf2ccb53da556c8e5e550f1b7fe1cfa3c7 WatchSource:0}: Error finding container 372966e8781bbf1391b7a81c167e02bf2ccb53da556c8e5e550f1b7fe1cfa3c7: Status 404 returned error can't find the container with id 372966e8781bbf1391b7a81c167e02bf2ccb53da556c8e5e550f1b7fe1cfa3c7
Jul 28 18:07:09.598416 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:09.598368    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4" event={"ID":"fde36536-6bc4-4cd7-8450-230e73b5ba59","Type":"ContainerStarted","Data":"372966e8781bbf1391b7a81c167e02bf2ccb53da556c8e5e550f1b7fe1cfa3c7"}
Jul 28 18:07:12.611172 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:12.611130    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4" event={"ID":"fde36536-6bc4-4cd7-8450-230e73b5ba59","Type":"ContainerStarted","Data":"89d5642e9c0f264e79c47e558ff572aa786e3a9ca7563722eed2d64a05087e04"}
Jul 28 18:07:13.616137 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:13.616096    2564 generic.go:356] "Generic (PLEG): container finished" podID="fde36536-6bc4-4cd7-8450-230e73b5ba59" containerID="89d5642e9c0f264e79c47e558ff572aa786e3a9ca7563722eed2d64a05087e04" exitCode=0
Jul 28 18:07:13.616534 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:13.616160    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4" event={"ID":"fde36536-6bc4-4cd7-8450-230e73b5ba59","Type":"ContainerDied","Data":"89d5642e9c0f264e79c47e558ff572aa786e3a9ca7563722eed2d64a05087e04"}
Jul 28 18:07:14.621578 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:14.621541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4" event={"ID":"fde36536-6bc4-4cd7-8450-230e73b5ba59","Type":"ContainerStarted","Data":"8f87ce841eb5bce81fd8d7fe6688c1ed2e699c115fbe56d53a249bd528955f55"}
Jul 28 18:07:14.621986 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:14.621627    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:14.646198 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:14.646142    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4" podStartSLOduration=2.006592112 podStartE2EDuration="6.646125061s" totalImagesPullingTime="3.566720727s" totalInitContainerRuntime="1.072812222s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:07:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:07:14.644235225 +0000 UTC m=+369.959737348" watchObservedRunningTime="2026-07-28 18:07:14.646125061 +0000 UTC m=+369.961627174"
Jul 28 18:07:15.583592 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:15.583564    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-84df6f5b9b-r6fst"
Jul 28 18:07:25.627749 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:25.627720    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-xkjx4"
Jul 28 18:07:38.499353 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:38.499319    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-79b97f775b-tcccj"]
Jul 28 18:07:38.503631 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:38.503612    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79b97f775b-tcccj"
Jul 28 18:07:38.506905 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:38.506880    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 28 18:07:38.507037 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:38.506930    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-894wp\""
Jul 28 18:07:38.508036 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:38.508017    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 18:07:38.514038 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:38.514018    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 18:07:38.514395 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:38.514381    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 28 18:07:38.535721 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:38.535690    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79b97f775b-tcccj"]
Jul 28 18:07:38.567626 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:38.567591    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xkc9\" (UniqueName: \"kubernetes.io/projected/6f52a446-8fe8-40a6-88e8-74c05a817ffc-kube-api-access-6xkc9\") pod \"kube-auth-proxy-79b97f775b-tcccj\" (UID: \"6f52a446-8fe8-40a6-88e8-74c05a817ffc\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-tcccj"
Jul 28 18:07:38.567626 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:38.567631    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f52a446-8fe8-40a6-88e8-74c05a817ffc-tls-certs\") pod \"kube-auth-proxy-79b97f775b-tcccj\" (UID: \"6f52a446-8fe8-40a6-88e8-74c05a817ffc\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-tcccj"
Jul 28 18:07:38.567830 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:38.567670    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f52a446-8fe8-40a6-88e8-74c05a817ffc-tmp\") pod \"kube-auth-proxy-79b97f775b-tcccj\" (UID: \"6f52a446-8fe8-40a6-88e8-74c05a817ffc\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-tcccj"
Jul 28 18:07:38.668845 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:38.668786    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xkc9\" (UniqueName: \"kubernetes.io/projected/6f52a446-8fe8-40a6-88e8-74c05a817ffc-kube-api-access-6xkc9\") pod \"kube-auth-proxy-79b97f775b-tcccj\" (UID: \"6f52a446-8fe8-40a6-88e8-74c05a817ffc\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-tcccj"
Jul 28 18:07:38.669002 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:38.668861    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f52a446-8fe8-40a6-88e8-74c05a817ffc-tls-certs\") pod \"kube-auth-proxy-79b97f775b-tcccj\" (UID: \"6f52a446-8fe8-40a6-88e8-74c05a817ffc\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-tcccj"
Jul 28 18:07:38.669002 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:38.668907    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f52a446-8fe8-40a6-88e8-74c05a817ffc-tmp\") pod \"kube-auth-proxy-79b97f775b-tcccj\" (UID: \"6f52a446-8fe8-40a6-88e8-74c05a817ffc\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-tcccj"
Jul 28 18:07:38.669073 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:07:38.669010    2564 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 28 18:07:38.669104 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:07:38.669093    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f52a446-8fe8-40a6-88e8-74c05a817ffc-tls-certs podName:6f52a446-8fe8-40a6-88e8-74c05a817ffc nodeName:}" failed. No retries permitted until 2026-07-28 18:07:39.169071314 +0000 UTC m=+394.484573422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6f52a446-8fe8-40a6-88e8-74c05a817ffc-tls-certs") pod "kube-auth-proxy-79b97f775b-tcccj" (UID: "6f52a446-8fe8-40a6-88e8-74c05a817ffc") : secret "kube-auth-proxy-tls" not found
Jul 28 18:07:38.671238 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:38.671215    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f52a446-8fe8-40a6-88e8-74c05a817ffc-tmp\") pod \"kube-auth-proxy-79b97f775b-tcccj\" (UID: \"6f52a446-8fe8-40a6-88e8-74c05a817ffc\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-tcccj"
Jul 28 18:07:38.679757 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:38.679732    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xkc9\" (UniqueName: \"kubernetes.io/projected/6f52a446-8fe8-40a6-88e8-74c05a817ffc-kube-api-access-6xkc9\") pod \"kube-auth-proxy-79b97f775b-tcccj\" (UID: \"6f52a446-8fe8-40a6-88e8-74c05a817ffc\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-tcccj"
Jul 28 18:07:39.172544 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:39.172502    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f52a446-8fe8-40a6-88e8-74c05a817ffc-tls-certs\") pod \"kube-auth-proxy-79b97f775b-tcccj\" (UID: \"6f52a446-8fe8-40a6-88e8-74c05a817ffc\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-tcccj"
Jul 28 18:07:39.172853 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:07:39.172643    2564 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 28 18:07:39.172853 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:07:39.172730    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f52a446-8fe8-40a6-88e8-74c05a817ffc-tls-certs podName:6f52a446-8fe8-40a6-88e8-74c05a817ffc nodeName:}" failed. No retries permitted until 2026-07-28 18:07:40.172708577 +0000 UTC m=+395.488210681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6f52a446-8fe8-40a6-88e8-74c05a817ffc-tls-certs") pod "kube-auth-proxy-79b97f775b-tcccj" (UID: "6f52a446-8fe8-40a6-88e8-74c05a817ffc") : secret "kube-auth-proxy-tls" not found
Jul 28 18:07:40.179605 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:40.179562    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f52a446-8fe8-40a6-88e8-74c05a817ffc-tls-certs\") pod \"kube-auth-proxy-79b97f775b-tcccj\" (UID: \"6f52a446-8fe8-40a6-88e8-74c05a817ffc\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-tcccj"
Jul 28 18:07:40.182046 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:40.182022    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f52a446-8fe8-40a6-88e8-74c05a817ffc-tls-certs\") pod \"kube-auth-proxy-79b97f775b-tcccj\" (UID: \"6f52a446-8fe8-40a6-88e8-74c05a817ffc\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-tcccj"
Jul 28 18:07:40.312965 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:40.312930    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79b97f775b-tcccj"
Jul 28 18:07:40.443933 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:40.443908    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79b97f775b-tcccj"]
Jul 28 18:07:40.446971 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:07:40.446931    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f52a446_8fe8_40a6_88e8_74c05a817ffc.slice/crio-c9cce91581de19b0de9dde573d8ac685b81118494d3b230aad8fd5145a203e60 WatchSource:0}: Error finding container c9cce91581de19b0de9dde573d8ac685b81118494d3b230aad8fd5145a203e60: Status 404 returned error can't find the container with id c9cce91581de19b0de9dde573d8ac685b81118494d3b230aad8fd5145a203e60
Jul 28 18:07:40.717294 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:40.717202    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79b97f775b-tcccj" event={"ID":"6f52a446-8fe8-40a6-88e8-74c05a817ffc","Type":"ContainerStarted","Data":"c9cce91581de19b0de9dde573d8ac685b81118494d3b230aad8fd5145a203e60"}
Jul 28 18:07:43.729780 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:43.729692    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79b97f775b-tcccj" event={"ID":"6f52a446-8fe8-40a6-88e8-74c05a817ffc","Type":"ContainerStarted","Data":"19a920371caf6b62556d8c5e0f1154fe6a7001005dfcfd5fe0d2c9519364332c"}
Jul 28 18:07:43.747067 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:07:43.747019    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-79b97f775b-tcccj" podStartSLOduration=3.042481614 podStartE2EDuration="5.747004067s" totalImagesPullingTime="2.704522453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:07:43.746025229 +0000 UTC m=+399.061527357" watchObservedRunningTime="2026-07-28 18:07:43.747004067 +0000 UTC m=+399.062506179"
Jul 28 18:08:59.690272 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.690234    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68ccc96945-prcbk"]
Jul 28 18:08:59.692716 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.692695    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.704020 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.703992    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68ccc96945-prcbk"]
Jul 28 18:08:59.773009 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.772977    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a74d1f-741b-431e-a60d-120aaa22e5ce-service-ca\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.773009 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.773013    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a74d1f-741b-431e-a60d-120aaa22e5ce-trusted-ca-bundle\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.773252 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.773038    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6a74d1f-741b-431e-a60d-120aaa22e5ce-console-config\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.773252 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.773136    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khpt2\" (UniqueName: \"kubernetes.io/projected/f6a74d1f-741b-431e-a60d-120aaa22e5ce-kube-api-access-khpt2\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.773252 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.773168    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6a74d1f-741b-431e-a60d-120aaa22e5ce-oauth-serving-cert\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.773252 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.773204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6a74d1f-741b-431e-a60d-120aaa22e5ce-console-serving-cert\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.773252 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.773221    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6a74d1f-741b-431e-a60d-120aaa22e5ce-console-oauth-config\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.873851 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.873773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6a74d1f-741b-431e-a60d-120aaa22e5ce-oauth-serving-cert\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.873851 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.873859    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6a74d1f-741b-431e-a60d-120aaa22e5ce-console-serving-cert\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.874110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.873879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6a74d1f-741b-431e-a60d-120aaa22e5ce-console-oauth-config\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.874110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.873919    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a74d1f-741b-431e-a60d-120aaa22e5ce-service-ca\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.874110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.873937    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a74d1f-741b-431e-a60d-120aaa22e5ce-trusted-ca-bundle\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.874110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.873963    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6a74d1f-741b-431e-a60d-120aaa22e5ce-console-config\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.874110 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.873992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khpt2\" (UniqueName: \"kubernetes.io/projected/f6a74d1f-741b-431e-a60d-120aaa22e5ce-kube-api-access-khpt2\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.874603 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.874579    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6a74d1f-741b-431e-a60d-120aaa22e5ce-oauth-serving-cert\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.874713 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.874685    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a74d1f-741b-431e-a60d-120aaa22e5ce-service-ca\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.874758 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.874742    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a74d1f-741b-431e-a60d-120aaa22e5ce-trusted-ca-bundle\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.874831 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.874786    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6a74d1f-741b-431e-a60d-120aaa22e5ce-console-config\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.876490 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.876451    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6a74d1f-741b-431e-a60d-120aaa22e5ce-console-serving-cert\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.876601 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.876541    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6a74d1f-741b-431e-a60d-120aaa22e5ce-console-oauth-config\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:08:59.883631 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:08:59.883607    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khpt2\" (UniqueName: \"kubernetes.io/projected/f6a74d1f-741b-431e-a60d-120aaa22e5ce-kube-api-access-khpt2\") pod \"console-68ccc96945-prcbk\" (UID: \"f6a74d1f-741b-431e-a60d-120aaa22e5ce\") " pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:09:00.001400 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:00.001308    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:09:00.132844 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:00.132796    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68ccc96945-prcbk"]
Jul 28 18:09:00.136230 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:09:00.136204    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a74d1f_741b_431e_a60d_120aaa22e5ce.slice/crio-96bc921083495b605fe9b4b1c27a7ea6c00d0234eafcd1752362085c5d570105 WatchSource:0}: Error finding container 96bc921083495b605fe9b4b1c27a7ea6c00d0234eafcd1752362085c5d570105: Status 404 returned error can't find the container with id 96bc921083495b605fe9b4b1c27a7ea6c00d0234eafcd1752362085c5d570105
Jul 28 18:09:01.003352 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:01.003315    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68ccc96945-prcbk" event={"ID":"f6a74d1f-741b-431e-a60d-120aaa22e5ce","Type":"ContainerStarted","Data":"3a0ac41bc7aa3517a4f244f314f997c79fcbbefa5084719fbeb78e4e833b24be"}
Jul 28 18:09:01.003844 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:01.003641    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68ccc96945-prcbk" event={"ID":"f6a74d1f-741b-431e-a60d-120aaa22e5ce","Type":"ContainerStarted","Data":"96bc921083495b605fe9b4b1c27a7ea6c00d0234eafcd1752362085c5d570105"}
Jul 28 18:09:10.001441 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:10.001400    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:09:10.002068 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:10.001466    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:09:10.006135 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:10.006113    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:09:10.027716 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:10.027668    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68ccc96945-prcbk" podStartSLOduration=11.027652154 podStartE2EDuration="11.027652154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:09:01.024212538 +0000 UTC m=+476.339714672" watchObservedRunningTime="2026-07-28 18:09:10.027652154 +0000 UTC m=+485.343154266"
Jul 28 18:09:10.040286 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:10.040258    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68ccc96945-prcbk"
Jul 28 18:09:10.095993 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:10.095960    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7474f96574-nfflq"]
Jul 28 18:09:21.271838 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.269924    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz"]
Jul 28 18:09:21.274069 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.274045    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz"
Jul 28 18:09:21.276571 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.276544    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 28 18:09:21.276691 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.276652    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 28 18:09:21.277237 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.277222    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zlqmd\""
Jul 28 18:09:21.277883 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.277868    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 18:09:21.277978 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.277902    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 18:09:21.282891 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.282868    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz"]
Jul 28 18:09:21.329701 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.329671    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb616377-2a6b-4fbf-a590-50d759166bfc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7t2mz\" (UID: \"eb616377-2a6b-4fbf-a590-50d759166bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz"
Jul 28 18:09:21.329864 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.329704    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k9pw\" (UniqueName: \"kubernetes.io/projected/eb616377-2a6b-4fbf-a590-50d759166bfc-kube-api-access-5k9pw\") pod \"kuadrant-console-plugin-66b5774985-7t2mz\" (UID: \"eb616377-2a6b-4fbf-a590-50d759166bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz"
Jul 28 18:09:21.329864 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.329734    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb616377-2a6b-4fbf-a590-50d759166bfc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7t2mz\" (UID: \"eb616377-2a6b-4fbf-a590-50d759166bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz"
Jul 28 18:09:21.430839 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.430791    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb616377-2a6b-4fbf-a590-50d759166bfc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7t2mz\" (UID: \"eb616377-2a6b-4fbf-a590-50d759166bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz"
Jul 28 18:09:21.430983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.430847    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k9pw\" (UniqueName: \"kubernetes.io/projected/eb616377-2a6b-4fbf-a590-50d759166bfc-kube-api-access-5k9pw\") pod \"kuadrant-console-plugin-66b5774985-7t2mz\" (UID: \"eb616377-2a6b-4fbf-a590-50d759166bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz"
Jul 28 18:09:21.430983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.430876    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb616377-2a6b-4fbf-a590-50d759166bfc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7t2mz\" (UID: \"eb616377-2a6b-4fbf-a590-50d759166bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz"
Jul 28 18:09:21.431068 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:09:21.430984    2564 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 28 18:09:21.431068 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:09:21.431047    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb616377-2a6b-4fbf-a590-50d759166bfc-plugin-serving-cert podName:eb616377-2a6b-4fbf-a590-50d759166bfc nodeName:}" failed. No retries permitted until 2026-07-28 18:09:21.931030969 +0000 UTC m=+497.246533060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/eb616377-2a6b-4fbf-a590-50d759166bfc-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-7t2mz" (UID: "eb616377-2a6b-4fbf-a590-50d759166bfc") : secret "plugin-serving-cert" not found
Jul 28 18:09:21.431507 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.431486    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb616377-2a6b-4fbf-a590-50d759166bfc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7t2mz\" (UID: \"eb616377-2a6b-4fbf-a590-50d759166bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz"
Jul 28 18:09:21.439573 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.439547    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k9pw\" (UniqueName: \"kubernetes.io/projected/eb616377-2a6b-4fbf-a590-50d759166bfc-kube-api-access-5k9pw\") pod \"kuadrant-console-plugin-66b5774985-7t2mz\" (UID: \"eb616377-2a6b-4fbf-a590-50d759166bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz"
Jul 28 18:09:21.933660 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.933627    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb616377-2a6b-4fbf-a590-50d759166bfc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7t2mz\" (UID: \"eb616377-2a6b-4fbf-a590-50d759166bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz"
Jul 28 18:09:21.936125 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:21.936102    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb616377-2a6b-4fbf-a590-50d759166bfc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7t2mz\" (UID: \"eb616377-2a6b-4fbf-a590-50d759166bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz"
Jul 28 18:09:22.183858 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:22.183756    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz"
Jul 28 18:09:22.311516 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:22.311487    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz"]
Jul 28 18:09:22.314270 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:09:22.314244    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb616377_2a6b_4fbf_a590_50d759166bfc.slice/crio-c5adb7fa52d4cf660af0b7e6988173f516df7e57752651261b334942760f646a WatchSource:0}: Error finding container c5adb7fa52d4cf660af0b7e6988173f516df7e57752651261b334942760f646a: Status 404 returned error can't find the container with id c5adb7fa52d4cf660af0b7e6988173f516df7e57752651261b334942760f646a
Jul 28 18:09:23.084183 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:23.084149    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz" event={"ID":"eb616377-2a6b-4fbf-a590-50d759166bfc","Type":"ContainerStarted","Data":"c5adb7fa52d4cf660af0b7e6988173f516df7e57752651261b334942760f646a"}
Jul 28 18:09:35.119200 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:35.119157    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7474f96574-nfflq" podUID="fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a" containerName="console" containerID="cri-o://f0b252a860d541b7d5e9650a8570820468b4de8bcccdf03d3a117eb829661afb" gracePeriod=15
Jul 28 18:09:35.179064 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:35.179025    2564 patch_prober.go:28] interesting pod/console-7474f96574-nfflq container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.15:8443/health\": dial tcp 10.134.0.15:8443: connect: connection refused" start-of-body=
Jul 28 18:09:35.179277 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:35.179078    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-7474f96574-nfflq" podUID="fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a" containerName="console" probeResult="failure" output="Get \"https://10.134.0.15:8443/health\": dial tcp 10.134.0.15:8443: connect: connection refused"
Jul 28 18:09:44.044227 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.044202    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:09:44.126274 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.126240    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-config\") pod \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") "
Jul 28 18:09:44.126446 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.126299    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sslvn\" (UniqueName: \"kubernetes.io/projected/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-kube-api-access-sslvn\") pod \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") "
Jul 28 18:09:44.126446 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.126326    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-trusted-ca-bundle\") pod \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") "
Jul 28 18:09:44.126446 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.126388    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-serving-cert\") pod \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") "
Jul 28 18:09:44.126609 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.126580    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-service-ca\") pod \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") "
Jul 28 18:09:44.126658 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.126624    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-oauth-config\") pod \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") "
Jul 28 18:09:44.126712 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.126684    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-oauth-serving-cert\") pod \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\" (UID: \"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a\") "
Jul 28 18:09:44.126764 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.126703    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-config" (OuterVolumeSpecName: "console-config") pod "fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a" (UID: "fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:09:44.126931 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.126893    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a" (UID: "fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:09:44.126931 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.126921    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:09:44.127050 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.126994    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-service-ca" (OuterVolumeSpecName: "service-ca") pod "fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a" (UID: "fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:09:44.127140 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.127123    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a" (UID: "fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:09:44.128616 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.128587    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-kube-api-access-sslvn" (OuterVolumeSpecName: "kube-api-access-sslvn") pod "fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a" (UID: "fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a"). InnerVolumeSpecName "kube-api-access-sslvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:09:44.128616 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.128613    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a" (UID: "fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:09:44.128911 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.128631    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a" (UID: "fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:09:44.184476 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.184389    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz" event={"ID":"eb616377-2a6b-4fbf-a590-50d759166bfc","Type":"ContainerStarted","Data":"4fdb6ecec36e599fffe820b08eff72bd3e6fd4ad839ca90ebc9325be205633f4"}
Jul 28 18:09:44.185789 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.185765    2564 generic.go:356] "Generic (PLEG): container finished" podID="fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a" containerID="f0b252a860d541b7d5e9650a8570820468b4de8bcccdf03d3a117eb829661afb" exitCode=0
Jul 28 18:09:44.185924 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.185797    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7474f96574-nfflq" event={"ID":"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a","Type":"ContainerDied","Data":"f0b252a860d541b7d5e9650a8570820468b4de8bcccdf03d3a117eb829661afb"}
Jul 28 18:09:44.185924 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.185826    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7474f96574-nfflq"
Jul 28 18:09:44.185924 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.185875    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7474f96574-nfflq" event={"ID":"fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a","Type":"ContainerDied","Data":"e0b340948d1fc653e218a0c6625bcff46797963b3134daef60d5be033c04fb70"}
Jul 28 18:09:44.185924 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.185896    2564 scope.go:117] "RemoveContainer" containerID="f0b252a860d541b7d5e9650a8570820468b4de8bcccdf03d3a117eb829661afb"
Jul 28 18:09:44.194302 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.194284    2564 scope.go:117] "RemoveContainer" containerID="f0b252a860d541b7d5e9650a8570820468b4de8bcccdf03d3a117eb829661afb"
Jul 28 18:09:44.194538 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:09:44.194523    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b252a860d541b7d5e9650a8570820468b4de8bcccdf03d3a117eb829661afb\": container with ID starting with f0b252a860d541b7d5e9650a8570820468b4de8bcccdf03d3a117eb829661afb not found: ID does not exist" containerID="f0b252a860d541b7d5e9650a8570820468b4de8bcccdf03d3a117eb829661afb"
Jul 28 18:09:44.194584 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.194544    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b252a860d541b7d5e9650a8570820468b4de8bcccdf03d3a117eb829661afb"} err="failed to get container status \"f0b252a860d541b7d5e9650a8570820468b4de8bcccdf03d3a117eb829661afb\": rpc error: code = NotFound desc = could not find container \"f0b252a860d541b7d5e9650a8570820468b4de8bcccdf03d3a117eb829661afb\": container with ID starting with f0b252a860d541b7d5e9650a8570820468b4de8bcccdf03d3a117eb829661afb not found: ID does not exist"
Jul 28 18:09:44.202432 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.202393    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7t2mz" podStartSLOduration=1.549177755 podStartE2EDuration="23.202380363s" totalImagesPullingTime="21.653202608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:09:44.20125935 +0000 UTC m=+519.516761488" watchObservedRunningTime="2026-07-28 18:09:44.202380363 +0000 UTC m=+519.517882500"
Jul 28 18:09:44.217719 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.217690    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7474f96574-nfflq"]
Jul 28 18:09:44.224591 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.224565    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7474f96574-nfflq"]
Jul 28 18:09:44.227833 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.227797    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-serving-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:09:44.227833 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.227833    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:09:44.227998 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.227844    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-console-oauth-config\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:09:44.227998 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.227853    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-oauth-serving-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:09:44.227998 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.227861    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sslvn\" (UniqueName: \"kubernetes.io/projected/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-kube-api-access-sslvn\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:09:44.227998 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:44.227870    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a-trusted-ca-bundle\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:09:45.215439 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:09:45.215359    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a" path="/var/lib/kubelet/pods/fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a/volumes"
Jul 28 18:10:17.855872 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:17.855832    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-cgvsg"]
Jul 28 18:10:17.856345 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:17.856103    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a" containerName="console"
Jul 28 18:10:17.856345 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:17.856114    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a" containerName="console"
Jul 28 18:10:17.856345 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:17.856174    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc2cfd53-d01f-4f60-b1f1-f0c2ded4b63a" containerName="console"
Jul 28 18:10:17.928677 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:17.928645    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-cgvsg"]
Jul 28 18:10:17.928885 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:17.928775    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cgvsg"
Jul 28 18:10:17.931673 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:17.931655    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jn8bg\""
Jul 28 18:10:17.962448 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:17.962412    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wckfg\" (UniqueName: \"kubernetes.io/projected/b984b571-5d4c-4b3c-8ec4-b26df5b13b3d-kube-api-access-wckfg\") pod \"authorino-58cd955675-cgvsg\" (UID: \"b984b571-5d4c-4b3c-8ec4-b26df5b13b3d\") " pod="kuadrant-system/authorino-58cd955675-cgvsg"
Jul 28 18:10:17.972242 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:17.972206    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-lx4k6"]
Jul 28 18:10:17.975356 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:17.975340    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lx4k6"
Jul 28 18:10:17.983919 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:17.983897    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lx4k6"]
Jul 28 18:10:18.062971 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:18.062935    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj8dp\" (UniqueName: \"kubernetes.io/projected/74dc8b06-7c9f-4379-867b-705589a6c9f9-kube-api-access-hj8dp\") pod \"authorino-75d556bd77-lx4k6\" (UID: \"74dc8b06-7c9f-4379-867b-705589a6c9f9\") " pod="kuadrant-system/authorino-75d556bd77-lx4k6"
Jul 28 18:10:18.063151 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:18.062996    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wckfg\" (UniqueName: \"kubernetes.io/projected/b984b571-5d4c-4b3c-8ec4-b26df5b13b3d-kube-api-access-wckfg\") pod \"authorino-58cd955675-cgvsg\" (UID: \"b984b571-5d4c-4b3c-8ec4-b26df5b13b3d\") " pod="kuadrant-system/authorino-58cd955675-cgvsg"
Jul 28 18:10:18.072479 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:18.072449    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wckfg\" (UniqueName: \"kubernetes.io/projected/b984b571-5d4c-4b3c-8ec4-b26df5b13b3d-kube-api-access-wckfg\") pod \"authorino-58cd955675-cgvsg\" (UID: \"b984b571-5d4c-4b3c-8ec4-b26df5b13b3d\") " pod="kuadrant-system/authorino-58cd955675-cgvsg"
Jul 28 18:10:18.163589 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:18.163492    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj8dp\" (UniqueName: \"kubernetes.io/projected/74dc8b06-7c9f-4379-867b-705589a6c9f9-kube-api-access-hj8dp\") pod \"authorino-75d556bd77-lx4k6\" (UID: \"74dc8b06-7c9f-4379-867b-705589a6c9f9\") " pod="kuadrant-system/authorino-75d556bd77-lx4k6"
Jul 28 18:10:18.172477 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:18.172445    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj8dp\" (UniqueName: \"kubernetes.io/projected/74dc8b06-7c9f-4379-867b-705589a6c9f9-kube-api-access-hj8dp\") pod \"authorino-75d556bd77-lx4k6\" (UID: \"74dc8b06-7c9f-4379-867b-705589a6c9f9\") " pod="kuadrant-system/authorino-75d556bd77-lx4k6"
Jul 28 18:10:18.237061 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:18.237023    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cgvsg"
Jul 28 18:10:18.284003 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:18.283968    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lx4k6"
Jul 28 18:10:18.375312 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:18.375260    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-cgvsg"]
Jul 28 18:10:18.381084 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:10:18.381038    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb984b571_5d4c_4b3c_8ec4_b26df5b13b3d.slice/crio-9b73cdd49c47b2c8f9c79bc9f1727cc7b9273a2d3f528461f9c72b950992307e WatchSource:0}: Error finding container 9b73cdd49c47b2c8f9c79bc9f1727cc7b9273a2d3f528461f9c72b950992307e: Status 404 returned error can't find the container with id 9b73cdd49c47b2c8f9c79bc9f1727cc7b9273a2d3f528461f9c72b950992307e
Jul 28 18:10:18.419967 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:18.419891    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lx4k6"]
Jul 28 18:10:18.423017 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:10:18.422990    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74dc8b06_7c9f_4379_867b_705589a6c9f9.slice/crio-85dd5ba585a6159fb91799c184c96e1a89ec304c4f0e65ae1ac811c567dc1b9c WatchSource:0}: Error finding container 85dd5ba585a6159fb91799c184c96e1a89ec304c4f0e65ae1ac811c567dc1b9c: Status 404 returned error can't find the container with id 85dd5ba585a6159fb91799c184c96e1a89ec304c4f0e65ae1ac811c567dc1b9c
Jul 28 18:10:19.324298 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:19.324260    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lx4k6" event={"ID":"74dc8b06-7c9f-4379-867b-705589a6c9f9","Type":"ContainerStarted","Data":"85dd5ba585a6159fb91799c184c96e1a89ec304c4f0e65ae1ac811c567dc1b9c"}
Jul 28 18:10:19.327421 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:19.327350    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cgvsg" event={"ID":"b984b571-5d4c-4b3c-8ec4-b26df5b13b3d","Type":"ContainerStarted","Data":"9b73cdd49c47b2c8f9c79bc9f1727cc7b9273a2d3f528461f9c72b950992307e"}
Jul 28 18:10:22.340660 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:22.340626    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cgvsg" event={"ID":"b984b571-5d4c-4b3c-8ec4-b26df5b13b3d","Type":"ContainerStarted","Data":"ade0229b3571100f8d860c4c5ab83c89a11ac4a25c6df0c603f8b30fbae491d8"}
Jul 28 18:10:22.342160 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:22.342131    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lx4k6" event={"ID":"74dc8b06-7c9f-4379-867b-705589a6c9f9","Type":"ContainerStarted","Data":"18712f4a704acf7c7b778448eb4ab3fcbea9243ace4349b7798f3c1454dcd607"}
Jul 28 18:10:22.366923 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:22.364726    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-cgvsg" podStartSLOduration=1.881106125 podStartE2EDuration="5.36469218s" totalImagesPullingTime="3.483586055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:10:22.362476585 +0000 UTC m=+557.677978691" watchObservedRunningTime="2026-07-28 18:10:22.36469218 +0000 UTC m=+557.680194298"
Jul 28 18:10:22.384324 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:22.384262    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-lx4k6" podStartSLOduration=1.930608618 podStartE2EDuration="5.384241328s" totalImagesPullingTime="3.45363271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:10:22.382660701 +0000 UTC m=+557.698162816" watchObservedRunningTime="2026-07-28 18:10:22.384241328 +0000 UTC m=+557.699743439"
Jul 28 18:10:22.419495 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:22.419466    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-cgvsg"]
Jul 28 18:10:24.350897 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:24.350857    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-cgvsg" podUID="b984b571-5d4c-4b3c-8ec4-b26df5b13b3d" containerName="authorino" containerID="cri-o://ade0229b3571100f8d860c4c5ab83c89a11ac4a25c6df0c603f8b30fbae491d8" gracePeriod=30
Jul 28 18:10:24.597928 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:24.597901    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cgvsg"
Jul 28 18:10:24.626998 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:24.626924    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wckfg\" (UniqueName: \"kubernetes.io/projected/b984b571-5d4c-4b3c-8ec4-b26df5b13b3d-kube-api-access-wckfg\") pod \"b984b571-5d4c-4b3c-8ec4-b26df5b13b3d\" (UID: \"b984b571-5d4c-4b3c-8ec4-b26df5b13b3d\") "
Jul 28 18:10:24.628975 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:24.628944    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b984b571-5d4c-4b3c-8ec4-b26df5b13b3d-kube-api-access-wckfg" (OuterVolumeSpecName: "kube-api-access-wckfg") pod "b984b571-5d4c-4b3c-8ec4-b26df5b13b3d" (UID: "b984b571-5d4c-4b3c-8ec4-b26df5b13b3d"). InnerVolumeSpecName "kube-api-access-wckfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:10:24.727577 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:24.727544    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wckfg\" (UniqueName: \"kubernetes.io/projected/b984b571-5d4c-4b3c-8ec4-b26df5b13b3d-kube-api-access-wckfg\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:10:25.356512 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:25.356477    2564 generic.go:356] "Generic (PLEG): container finished" podID="b984b571-5d4c-4b3c-8ec4-b26df5b13b3d" containerID="ade0229b3571100f8d860c4c5ab83c89a11ac4a25c6df0c603f8b30fbae491d8" exitCode=0
Jul 28 18:10:25.356972 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:25.356516    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cgvsg"
Jul 28 18:10:25.356972 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:25.356527    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cgvsg" event={"ID":"b984b571-5d4c-4b3c-8ec4-b26df5b13b3d","Type":"ContainerDied","Data":"ade0229b3571100f8d860c4c5ab83c89a11ac4a25c6df0c603f8b30fbae491d8"}
Jul 28 18:10:25.356972 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:25.356560    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cgvsg" event={"ID":"b984b571-5d4c-4b3c-8ec4-b26df5b13b3d","Type":"ContainerDied","Data":"9b73cdd49c47b2c8f9c79bc9f1727cc7b9273a2d3f528461f9c72b950992307e"}
Jul 28 18:10:25.356972 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:25.356577    2564 scope.go:117] "RemoveContainer" containerID="ade0229b3571100f8d860c4c5ab83c89a11ac4a25c6df0c603f8b30fbae491d8"
Jul 28 18:10:25.364608 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:25.364573    2564 scope.go:117] "RemoveContainer" containerID="ade0229b3571100f8d860c4c5ab83c89a11ac4a25c6df0c603f8b30fbae491d8"
Jul 28 18:10:25.364874 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:10:25.364854    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade0229b3571100f8d860c4c5ab83c89a11ac4a25c6df0c603f8b30fbae491d8\": container with ID starting with ade0229b3571100f8d860c4c5ab83c89a11ac4a25c6df0c603f8b30fbae491d8 not found: ID does not exist" containerID="ade0229b3571100f8d860c4c5ab83c89a11ac4a25c6df0c603f8b30fbae491d8"
Jul 28 18:10:25.364950 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:25.364887    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade0229b3571100f8d860c4c5ab83c89a11ac4a25c6df0c603f8b30fbae491d8"} err="failed to get container status \"ade0229b3571100f8d860c4c5ab83c89a11ac4a25c6df0c603f8b30fbae491d8\": rpc error: code = NotFound desc = could not find container \"ade0229b3571100f8d860c4c5ab83c89a11ac4a25c6df0c603f8b30fbae491d8\": container with ID starting with ade0229b3571100f8d860c4c5ab83c89a11ac4a25c6df0c603f8b30fbae491d8 not found: ID does not exist"
Jul 28 18:10:25.374481 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:25.374457    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-cgvsg"]
Jul 28 18:10:25.380336 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:25.380314    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-cgvsg"]
Jul 28 18:10:27.215935 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:27.215908    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b984b571-5d4c-4b3c-8ec4-b26df5b13b3d" path="/var/lib/kubelet/pods/b984b571-5d4c-4b3c-8ec4-b26df5b13b3d/volumes"
Jul 28 18:10:48.483373 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:48.483341    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2w7kl"]
Jul 28 18:10:48.483765 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:48.483599    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b984b571-5d4c-4b3c-8ec4-b26df5b13b3d" containerName="authorino"
Jul 28 18:10:48.483765 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:48.483609    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b984b571-5d4c-4b3c-8ec4-b26df5b13b3d" containerName="authorino"
Jul 28 18:10:48.483765 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:48.483664    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b984b571-5d4c-4b3c-8ec4-b26df5b13b3d" containerName="authorino"
Jul 28 18:10:48.485744 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:48.485729    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2w7kl"
Jul 28 18:10:48.494731 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:48.494708    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2w7kl"]
Jul 28 18:10:48.600289 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:48.600251    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrwq8\" (UniqueName: \"kubernetes.io/projected/a3456e75-5926-4558-92dd-567996346bd8-kube-api-access-rrwq8\") pod \"authorino-5d4b67c446-2w7kl\" (UID: \"a3456e75-5926-4558-92dd-567996346bd8\") " pod="kuadrant-system/authorino-5d4b67c446-2w7kl"
Jul 28 18:10:48.701204 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:48.701157    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrwq8\" (UniqueName: \"kubernetes.io/projected/a3456e75-5926-4558-92dd-567996346bd8-kube-api-access-rrwq8\") pod \"authorino-5d4b67c446-2w7kl\" (UID: \"a3456e75-5926-4558-92dd-567996346bd8\") " pod="kuadrant-system/authorino-5d4b67c446-2w7kl"
Jul 28 18:10:48.712733 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:48.712701    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrwq8\" (UniqueName: \"kubernetes.io/projected/a3456e75-5926-4558-92dd-567996346bd8-kube-api-access-rrwq8\") pod \"authorino-5d4b67c446-2w7kl\" (UID: \"a3456e75-5926-4558-92dd-567996346bd8\") " pod="kuadrant-system/authorino-5d4b67c446-2w7kl"
Jul 28 18:10:48.755553 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:48.755486    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2w7kl"]
Jul 28 18:10:48.755699 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:48.755688    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2w7kl"
Jul 28 18:10:48.783236 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:48.783210    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f7997f976-jrn7n"]
Jul 28 18:10:48.786446 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:48.786430    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7997f976-jrn7n"
Jul 28 18:10:48.796848 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:48.796824    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f7997f976-jrn7n"]
Jul 28 18:10:48.886587 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:48.886562    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2w7kl"]
Jul 28 18:10:48.889153 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:10:48.889108    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3456e75_5926_4558_92dd_567996346bd8.slice/crio-33f72c281a598b7c69d9458e3f409b4070192b8fe5c3fd3bcff69a05c745874f WatchSource:0}: Error finding container 33f72c281a598b7c69d9458e3f409b4070192b8fe5c3fd3bcff69a05c745874f: Status 404 returned error can't find the container with id 33f72c281a598b7c69d9458e3f409b4070192b8fe5c3fd3bcff69a05c745874f
Jul 28 18:10:48.903434 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:48.903412    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9rr6\" (UniqueName: \"kubernetes.io/projected/efcd4370-ffc9-4f90-a776-c8063f5863e3-kube-api-access-b9rr6\") pod \"authorino-5f7997f976-jrn7n\" (UID: \"efcd4370-ffc9-4f90-a776-c8063f5863e3\") " pod="kuadrant-system/authorino-5f7997f976-jrn7n"
Jul 28 18:10:49.004699 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.004657    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9rr6\" (UniqueName: \"kubernetes.io/projected/efcd4370-ffc9-4f90-a776-c8063f5863e3-kube-api-access-b9rr6\") pod \"authorino-5f7997f976-jrn7n\" (UID: \"efcd4370-ffc9-4f90-a776-c8063f5863e3\") " pod="kuadrant-system/authorino-5f7997f976-jrn7n"
Jul 28 18:10:49.015694 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.015659    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9rr6\" (UniqueName: \"kubernetes.io/projected/efcd4370-ffc9-4f90-a776-c8063f5863e3-kube-api-access-b9rr6\") pod \"authorino-5f7997f976-jrn7n\" (UID: \"efcd4370-ffc9-4f90-a776-c8063f5863e3\") " pod="kuadrant-system/authorino-5f7997f976-jrn7n"
Jul 28 18:10:49.043851 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.043822    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f7997f976-jrn7n"]
Jul 28 18:10:49.044068 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.044056    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7997f976-jrn7n"
Jul 28 18:10:49.074393 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.074364    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bd58cb456-ndhhc"]
Jul 28 18:10:49.078542 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.078524    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bd58cb456-ndhhc"
Jul 28 18:10:49.081410 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.081384    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 28 18:10:49.089221 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.089200    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bd58cb456-ndhhc"]
Jul 28 18:10:49.105847 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.105797    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d769d2a-e6cb-4471-8b9d-1f58544bd72b-tls-cert\") pod \"authorino-7bd58cb456-ndhhc\" (UID: \"3d769d2a-e6cb-4471-8b9d-1f58544bd72b\") " pod="kuadrant-system/authorino-7bd58cb456-ndhhc"
Jul 28 18:10:49.105962 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.105861    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsnq4\" (UniqueName: \"kubernetes.io/projected/3d769d2a-e6cb-4471-8b9d-1f58544bd72b-kube-api-access-tsnq4\") pod \"authorino-7bd58cb456-ndhhc\" (UID: \"3d769d2a-e6cb-4471-8b9d-1f58544bd72b\") " pod="kuadrant-system/authorino-7bd58cb456-ndhhc"
Jul 28 18:10:49.169020 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.168999    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f7997f976-jrn7n"]
Jul 28 18:10:49.171419 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:10:49.171396    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefcd4370_ffc9_4f90_a776_c8063f5863e3.slice/crio-eb2d84bcda276ef4e652dc205166e1eab50238dbf65c6a0d5715165764e5fa37 WatchSource:0}: Error finding container eb2d84bcda276ef4e652dc205166e1eab50238dbf65c6a0d5715165764e5fa37: Status 404 returned error can't find the container with id eb2d84bcda276ef4e652dc205166e1eab50238dbf65c6a0d5715165764e5fa37
Jul 28 18:10:49.207268 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.207240    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d769d2a-e6cb-4471-8b9d-1f58544bd72b-tls-cert\") pod \"authorino-7bd58cb456-ndhhc\" (UID: \"3d769d2a-e6cb-4471-8b9d-1f58544bd72b\") " pod="kuadrant-system/authorino-7bd58cb456-ndhhc"
Jul 28 18:10:49.207364 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.207274    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsnq4\" (UniqueName: \"kubernetes.io/projected/3d769d2a-e6cb-4471-8b9d-1f58544bd72b-kube-api-access-tsnq4\") pod \"authorino-7bd58cb456-ndhhc\" (UID: \"3d769d2a-e6cb-4471-8b9d-1f58544bd72b\") " pod="kuadrant-system/authorino-7bd58cb456-ndhhc"
Jul 28 18:10:49.209555 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.209531    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d769d2a-e6cb-4471-8b9d-1f58544bd72b-tls-cert\") pod \"authorino-7bd58cb456-ndhhc\" (UID: \"3d769d2a-e6cb-4471-8b9d-1f58544bd72b\") " pod="kuadrant-system/authorino-7bd58cb456-ndhhc"
Jul 28 18:10:49.215645 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.215628    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsnq4\" (UniqueName: \"kubernetes.io/projected/3d769d2a-e6cb-4471-8b9d-1f58544bd72b-kube-api-access-tsnq4\") pod \"authorino-7bd58cb456-ndhhc\" (UID: \"3d769d2a-e6cb-4471-8b9d-1f58544bd72b\") " pod="kuadrant-system/authorino-7bd58cb456-ndhhc"
Jul 28 18:10:49.389387 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.389302    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bd58cb456-ndhhc"
Jul 28 18:10:49.454960 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.454926    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7997f976-jrn7n" event={"ID":"efcd4370-ffc9-4f90-a776-c8063f5863e3","Type":"ContainerStarted","Data":"eb2d84bcda276ef4e652dc205166e1eab50238dbf65c6a0d5715165764e5fa37"}
Jul 28 18:10:49.456329 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.456298    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2w7kl" event={"ID":"a3456e75-5926-4558-92dd-567996346bd8","Type":"ContainerStarted","Data":"2b1915703a6971c922071885aebf0a9761c8201be7eed04ec1b86b42ffcf2819"}
Jul 28 18:10:49.456454 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.456330    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2w7kl" event={"ID":"a3456e75-5926-4558-92dd-567996346bd8","Type":"ContainerStarted","Data":"33f72c281a598b7c69d9458e3f409b4070192b8fe5c3fd3bcff69a05c745874f"}
Jul 28 18:10:49.456454 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.456362    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-2w7kl" podUID="a3456e75-5926-4558-92dd-567996346bd8" containerName="authorino" containerID="cri-o://2b1915703a6971c922071885aebf0a9761c8201be7eed04ec1b86b42ffcf2819" gracePeriod=30
Jul 28 18:10:49.475297 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.475242    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-2w7kl" podStartSLOduration=1.149552677 podStartE2EDuration="1.475220899s" totalImagesPullingTime="325.668222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:10:49.472731216 +0000 UTC m=+584.788233333" watchObservedRunningTime="2026-07-28 18:10:49.475220899 +0000 UTC m=+584.790723013"
Jul 28 18:10:49.515947 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.515914    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bd58cb456-ndhhc"]
Jul 28 18:10:49.518682 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:10:49.518655    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d769d2a_e6cb_4471_8b9d_1f58544bd72b.slice/crio-4b06743dff6fb011874572921a7d6cf3628e77d331006ebc1f67252c46e36bf5 WatchSource:0}: Error finding container 4b06743dff6fb011874572921a7d6cf3628e77d331006ebc1f67252c46e36bf5: Status 404 returned error can't find the container with id 4b06743dff6fb011874572921a7d6cf3628e77d331006ebc1f67252c46e36bf5
Jul 28 18:10:49.701920 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.701893    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2w7kl"
Jul 28 18:10:49.812522 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.812485    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrwq8\" (UniqueName: \"kubernetes.io/projected/a3456e75-5926-4558-92dd-567996346bd8-kube-api-access-rrwq8\") pod \"a3456e75-5926-4558-92dd-567996346bd8\" (UID: \"a3456e75-5926-4558-92dd-567996346bd8\") "
Jul 28 18:10:49.814549 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.814521    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3456e75-5926-4558-92dd-567996346bd8-kube-api-access-rrwq8" (OuterVolumeSpecName: "kube-api-access-rrwq8") pod "a3456e75-5926-4558-92dd-567996346bd8" (UID: "a3456e75-5926-4558-92dd-567996346bd8"). InnerVolumeSpecName "kube-api-access-rrwq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:10:49.913101 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:49.913065    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrwq8\" (UniqueName: \"kubernetes.io/projected/a3456e75-5926-4558-92dd-567996346bd8-kube-api-access-rrwq8\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:10:50.461143 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.461099    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7997f976-jrn7n" event={"ID":"efcd4370-ffc9-4f90-a776-c8063f5863e3","Type":"ContainerStarted","Data":"9ca831d6cb85ed26a4ca465ad1ffb4f0471b49482990d58216437b8c0c073691"}
Jul 28 18:10:50.461143 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.461141    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5f7997f976-jrn7n" podUID="efcd4370-ffc9-4f90-a776-c8063f5863e3" containerName="authorino" containerID="cri-o://9ca831d6cb85ed26a4ca465ad1ffb4f0471b49482990d58216437b8c0c073691" gracePeriod=30
Jul 28 18:10:50.462609 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.462583    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bd58cb456-ndhhc" event={"ID":"3d769d2a-e6cb-4471-8b9d-1f58544bd72b","Type":"ContainerStarted","Data":"f5b1535369b2bdd4ba588373f957c86a7e410174a96ae71f4acab003a65e53a9"}
Jul 28 18:10:50.462735 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.462618    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bd58cb456-ndhhc" event={"ID":"3d769d2a-e6cb-4471-8b9d-1f58544bd72b","Type":"ContainerStarted","Data":"4b06743dff6fb011874572921a7d6cf3628e77d331006ebc1f67252c46e36bf5"}
Jul 28 18:10:50.463701 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.463681    2564 generic.go:356] "Generic (PLEG): container finished" podID="a3456e75-5926-4558-92dd-567996346bd8" containerID="2b1915703a6971c922071885aebf0a9761c8201be7eed04ec1b86b42ffcf2819" exitCode=0
Jul 28 18:10:50.463799 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.463708    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2w7kl"
Jul 28 18:10:50.463799 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.463736    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2w7kl" event={"ID":"a3456e75-5926-4558-92dd-567996346bd8","Type":"ContainerDied","Data":"2b1915703a6971c922071885aebf0a9761c8201be7eed04ec1b86b42ffcf2819"}
Jul 28 18:10:50.463799 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.463767    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2w7kl" event={"ID":"a3456e75-5926-4558-92dd-567996346bd8","Type":"ContainerDied","Data":"33f72c281a598b7c69d9458e3f409b4070192b8fe5c3fd3bcff69a05c745874f"}
Jul 28 18:10:50.463799 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.463790    2564 scope.go:117] "RemoveContainer" containerID="2b1915703a6971c922071885aebf0a9761c8201be7eed04ec1b86b42ffcf2819"
Jul 28 18:10:50.471515 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.471501    2564 scope.go:117] "RemoveContainer" containerID="2b1915703a6971c922071885aebf0a9761c8201be7eed04ec1b86b42ffcf2819"
Jul 28 18:10:50.471745 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:10:50.471724    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1915703a6971c922071885aebf0a9761c8201be7eed04ec1b86b42ffcf2819\": container with ID starting with 2b1915703a6971c922071885aebf0a9761c8201be7eed04ec1b86b42ffcf2819 not found: ID does not exist" containerID="2b1915703a6971c922071885aebf0a9761c8201be7eed04ec1b86b42ffcf2819"
Jul 28 18:10:50.471784 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.471760    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1915703a6971c922071885aebf0a9761c8201be7eed04ec1b86b42ffcf2819"} err="failed to get container status \"2b1915703a6971c922071885aebf0a9761c8201be7eed04ec1b86b42ffcf2819\": rpc error: code = NotFound desc = could not find container \"2b1915703a6971c922071885aebf0a9761c8201be7eed04ec1b86b42ffcf2819\": container with ID starting with 2b1915703a6971c922071885aebf0a9761c8201be7eed04ec1b86b42ffcf2819 not found: ID does not exist"
Jul 28 18:10:50.478864 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.478820    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f7997f976-jrn7n" podStartSLOduration=2.090249893 podStartE2EDuration="2.478787872s" totalImagesPullingTime="388.537979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:10:50.477697291 +0000 UTC m=+585.793199403" watchObservedRunningTime="2026-07-28 18:10:50.478787872 +0000 UTC m=+585.794289986"
Jul 28 18:10:50.495542 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.495500    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bd58cb456-ndhhc" podStartSLOduration=1.064817728 podStartE2EDuration="1.495486082s" totalImagesPullingTime="430.668354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:10:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:10:50.493838926 +0000 UTC m=+585.809341039" watchObservedRunningTime="2026-07-28 18:10:50.495486082 +0000 UTC m=+585.810988194"
Jul 28 18:10:50.523168 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.523139    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2w7kl"]
Jul 28 18:10:50.528166 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.528144    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lx4k6"]
Jul 28 18:10:50.528307 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.528290    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-lx4k6" podUID="74dc8b06-7c9f-4379-867b-705589a6c9f9" containerName="authorino" containerID="cri-o://18712f4a704acf7c7b778448eb4ab3fcbea9243ace4349b7798f3c1454dcd607" gracePeriod=30
Jul 28 18:10:50.541392 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.541366    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2w7kl"]
Jul 28 18:10:50.762611 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.762583    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7997f976-jrn7n"
Jul 28 18:10:50.795373 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.795345    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lx4k6"
Jul 28 18:10:50.820327 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.820296    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9rr6\" (UniqueName: \"kubernetes.io/projected/efcd4370-ffc9-4f90-a776-c8063f5863e3-kube-api-access-b9rr6\") pod \"efcd4370-ffc9-4f90-a776-c8063f5863e3\" (UID: \"efcd4370-ffc9-4f90-a776-c8063f5863e3\") "
Jul 28 18:10:50.822281 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.822257    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efcd4370-ffc9-4f90-a776-c8063f5863e3-kube-api-access-b9rr6" (OuterVolumeSpecName: "kube-api-access-b9rr6") pod "efcd4370-ffc9-4f90-a776-c8063f5863e3" (UID: "efcd4370-ffc9-4f90-a776-c8063f5863e3"). InnerVolumeSpecName "kube-api-access-b9rr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:10:50.921749 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.921707    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj8dp\" (UniqueName: \"kubernetes.io/projected/74dc8b06-7c9f-4379-867b-705589a6c9f9-kube-api-access-hj8dp\") pod \"74dc8b06-7c9f-4379-867b-705589a6c9f9\" (UID: \"74dc8b06-7c9f-4379-867b-705589a6c9f9\") "
Jul 28 18:10:50.921908 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.921856    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9rr6\" (UniqueName: \"kubernetes.io/projected/efcd4370-ffc9-4f90-a776-c8063f5863e3-kube-api-access-b9rr6\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:10:50.923748 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:50.923726    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74dc8b06-7c9f-4379-867b-705589a6c9f9-kube-api-access-hj8dp" (OuterVolumeSpecName: "kube-api-access-hj8dp") pod "74dc8b06-7c9f-4379-867b-705589a6c9f9" (UID: "74dc8b06-7c9f-4379-867b-705589a6c9f9"). InnerVolumeSpecName "kube-api-access-hj8dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:10:51.022995 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.022957    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj8dp\" (UniqueName: \"kubernetes.io/projected/74dc8b06-7c9f-4379-867b-705589a6c9f9-kube-api-access-hj8dp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:10:51.215700 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.215669    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3456e75-5926-4558-92dd-567996346bd8" path="/var/lib/kubelet/pods/a3456e75-5926-4558-92dd-567996346bd8/volumes"
Jul 28 18:10:51.469884 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.469780    2564 generic.go:356] "Generic (PLEG): container finished" podID="74dc8b06-7c9f-4379-867b-705589a6c9f9" containerID="18712f4a704acf7c7b778448eb4ab3fcbea9243ace4349b7798f3c1454dcd607" exitCode=0
Jul 28 18:10:51.469884 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.469832    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lx4k6"
Jul 28 18:10:51.469884 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.469844    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lx4k6" event={"ID":"74dc8b06-7c9f-4379-867b-705589a6c9f9","Type":"ContainerDied","Data":"18712f4a704acf7c7b778448eb4ab3fcbea9243ace4349b7798f3c1454dcd607"}
Jul 28 18:10:51.469884 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.469881    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lx4k6" event={"ID":"74dc8b06-7c9f-4379-867b-705589a6c9f9","Type":"ContainerDied","Data":"85dd5ba585a6159fb91799c184c96e1a89ec304c4f0e65ae1ac811c567dc1b9c"}
Jul 28 18:10:51.470159 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.469903    2564 scope.go:117] "RemoveContainer" containerID="18712f4a704acf7c7b778448eb4ab3fcbea9243ace4349b7798f3c1454dcd607"
Jul 28 18:10:51.471177 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.471098    2564 generic.go:356] "Generic (PLEG): container finished" podID="efcd4370-ffc9-4f90-a776-c8063f5863e3" containerID="9ca831d6cb85ed26a4ca465ad1ffb4f0471b49482990d58216437b8c0c073691" exitCode=0
Jul 28 18:10:51.471177 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.471126    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7997f976-jrn7n"
Jul 28 18:10:51.471177 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.471168    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7997f976-jrn7n" event={"ID":"efcd4370-ffc9-4f90-a776-c8063f5863e3","Type":"ContainerDied","Data":"9ca831d6cb85ed26a4ca465ad1ffb4f0471b49482990d58216437b8c0c073691"}
Jul 28 18:10:51.471376 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.471204    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7997f976-jrn7n" event={"ID":"efcd4370-ffc9-4f90-a776-c8063f5863e3","Type":"ContainerDied","Data":"eb2d84bcda276ef4e652dc205166e1eab50238dbf65c6a0d5715165764e5fa37"}
Jul 28 18:10:51.478014 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.477999    2564 scope.go:117] "RemoveContainer" containerID="18712f4a704acf7c7b778448eb4ab3fcbea9243ace4349b7798f3c1454dcd607"
Jul 28 18:10:51.478266 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:10:51.478247    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18712f4a704acf7c7b778448eb4ab3fcbea9243ace4349b7798f3c1454dcd607\": container with ID starting with 18712f4a704acf7c7b778448eb4ab3fcbea9243ace4349b7798f3c1454dcd607 not found: ID does not exist" containerID="18712f4a704acf7c7b778448eb4ab3fcbea9243ace4349b7798f3c1454dcd607"
Jul 28 18:10:51.478338 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.478276    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18712f4a704acf7c7b778448eb4ab3fcbea9243ace4349b7798f3c1454dcd607"} err="failed to get container status \"18712f4a704acf7c7b778448eb4ab3fcbea9243ace4349b7798f3c1454dcd607\": rpc error: code = NotFound desc = could not find container \"18712f4a704acf7c7b778448eb4ab3fcbea9243ace4349b7798f3c1454dcd607\": container with ID starting with 18712f4a704acf7c7b778448eb4ab3fcbea9243ace4349b7798f3c1454dcd607 not found: ID does not exist"
Jul 28 18:10:51.478338 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.478301    2564 scope.go:117] "RemoveContainer" containerID="9ca831d6cb85ed26a4ca465ad1ffb4f0471b49482990d58216437b8c0c073691"
Jul 28 18:10:51.485089 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.485059    2564 scope.go:117] "RemoveContainer" containerID="9ca831d6cb85ed26a4ca465ad1ffb4f0471b49482990d58216437b8c0c073691"
Jul 28 18:10:51.485440 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:10:51.485413    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca831d6cb85ed26a4ca465ad1ffb4f0471b49482990d58216437b8c0c073691\": container with ID starting with 9ca831d6cb85ed26a4ca465ad1ffb4f0471b49482990d58216437b8c0c073691 not found: ID does not exist" containerID="9ca831d6cb85ed26a4ca465ad1ffb4f0471b49482990d58216437b8c0c073691"
Jul 28 18:10:51.485717 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.485686    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca831d6cb85ed26a4ca465ad1ffb4f0471b49482990d58216437b8c0c073691"} err="failed to get container status \"9ca831d6cb85ed26a4ca465ad1ffb4f0471b49482990d58216437b8c0c073691\": rpc error: code = NotFound desc = could not find container \"9ca831d6cb85ed26a4ca465ad1ffb4f0471b49482990d58216437b8c0c073691\": container with ID starting with 9ca831d6cb85ed26a4ca465ad1ffb4f0471b49482990d58216437b8c0c073691 not found: ID does not exist"
Jul 28 18:10:51.486961 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.486943    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lx4k6"]
Jul 28 18:10:51.494955 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.493191    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lx4k6"]
Jul 28 18:10:51.508345 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.508320    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f7997f976-jrn7n"]
Jul 28 18:10:51.511998 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:51.511973    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f7997f976-jrn7n"]
Jul 28 18:10:53.216056 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:53.216024    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74dc8b06-7c9f-4379-867b-705589a6c9f9" path="/var/lib/kubelet/pods/74dc8b06-7c9f-4379-867b-705589a6c9f9/volumes"
Jul 28 18:10:53.216412 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:53.216327    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efcd4370-ffc9-4f90-a776-c8063f5863e3" path="/var/lib/kubelet/pods/efcd4370-ffc9-4f90-a776-c8063f5863e3/volumes"
Jul 28 18:10:55.554909 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.554874    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56cfbfb6cd-vlrpt"]
Jul 28 18:10:55.555382 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.555231    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74dc8b06-7c9f-4379-867b-705589a6c9f9" containerName="authorino"
Jul 28 18:10:55.555382 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.555248    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="74dc8b06-7c9f-4379-867b-705589a6c9f9" containerName="authorino"
Jul 28 18:10:55.555382 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.555269    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3456e75-5926-4558-92dd-567996346bd8" containerName="authorino"
Jul 28 18:10:55.555382 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.555278    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3456e75-5926-4558-92dd-567996346bd8" containerName="authorino"
Jul 28 18:10:55.555382 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.555287    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efcd4370-ffc9-4f90-a776-c8063f5863e3" containerName="authorino"
Jul 28 18:10:55.555382 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.555294    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="efcd4370-ffc9-4f90-a776-c8063f5863e3" containerName="authorino"
Jul 28 18:10:55.555382 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.555367    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3456e75-5926-4558-92dd-567996346bd8" containerName="authorino"
Jul 28 18:10:55.555382 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.555380    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="efcd4370-ffc9-4f90-a776-c8063f5863e3" containerName="authorino"
Jul 28 18:10:55.555382 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.555388    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="74dc8b06-7c9f-4379-867b-705589a6c9f9" containerName="authorino"
Jul 28 18:10:55.613879 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.613767    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56cfbfb6cd-vlrpt"]
Jul 28 18:10:55.614068 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.613895    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt"
Jul 28 18:10:55.617015 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.616990    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 18:10:55.618222 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.618199    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 18:10:55.618359 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.618229    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-twsgn\""
Jul 28 18:10:55.657542 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.657501    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzctr\" (UniqueName: \"kubernetes.io/projected/06876849-cb38-4a56-98df-348ee07cbc63-kube-api-access-nzctr\") pod \"maas-controller-56cfbfb6cd-vlrpt\" (UID: \"06876849-cb38-4a56-98df-348ee07cbc63\") " pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt"
Jul 28 18:10:55.657697 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.657576    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/06876849-cb38-4a56-98df-348ee07cbc63-webhook-cert\") pod \"maas-controller-56cfbfb6cd-vlrpt\" (UID: \"06876849-cb38-4a56-98df-348ee07cbc63\") " pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt"
Jul 28 18:10:55.758885 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.758839    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/06876849-cb38-4a56-98df-348ee07cbc63-webhook-cert\") pod \"maas-controller-56cfbfb6cd-vlrpt\" (UID: \"06876849-cb38-4a56-98df-348ee07cbc63\") " pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt"
Jul 28 18:10:55.759149 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.759004    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzctr\" (UniqueName: \"kubernetes.io/projected/06876849-cb38-4a56-98df-348ee07cbc63-kube-api-access-nzctr\") pod \"maas-controller-56cfbfb6cd-vlrpt\" (UID: \"06876849-cb38-4a56-98df-348ee07cbc63\") " pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt"
Jul 28 18:10:55.761432 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.761406    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/06876849-cb38-4a56-98df-348ee07cbc63-webhook-cert\") pod \"maas-controller-56cfbfb6cd-vlrpt\" (UID: \"06876849-cb38-4a56-98df-348ee07cbc63\") " pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt"
Jul 28 18:10:55.768225 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.768200    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzctr\" (UniqueName: \"kubernetes.io/projected/06876849-cb38-4a56-98df-348ee07cbc63-kube-api-access-nzctr\") pod \"maas-controller-56cfbfb6cd-vlrpt\" (UID: \"06876849-cb38-4a56-98df-348ee07cbc63\") " pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt"
Jul 28 18:10:55.923960 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:55.923869    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt"
Jul 28 18:10:56.076329 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:56.076294    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56cfbfb6cd-vlrpt"]
Jul 28 18:10:56.079197 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:10:56.079170    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06876849_cb38_4a56_98df_348ee07cbc63.slice/crio-392a481187977dae8b2a2d0fd45c3a1aa226690a4a2c9e9eedad4718926c7e7d WatchSource:0}: Error finding container 392a481187977dae8b2a2d0fd45c3a1aa226690a4a2c9e9eedad4718926c7e7d: Status 404 returned error can't find the container with id 392a481187977dae8b2a2d0fd45c3a1aa226690a4a2c9e9eedad4718926c7e7d
Jul 28 18:10:56.496641 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:56.496604    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt" event={"ID":"06876849-cb38-4a56-98df-348ee07cbc63","Type":"ContainerStarted","Data":"392a481187977dae8b2a2d0fd45c3a1aa226690a4a2c9e9eedad4718926c7e7d"}
Jul 28 18:10:58.507218 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:58.507184    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt" event={"ID":"06876849-cb38-4a56-98df-348ee07cbc63","Type":"ContainerStarted","Data":"a128feda881a684216ccb4167ce1c7090ffc6ad2fb9fb33897f1bd6f38396ec7"}
Jul 28 18:10:58.507615 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:10:58.507307    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt"
Jul 28 18:11:09.514095 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:09.514067    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt"
Jul 28 18:11:09.533438 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:09.533390    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt" podStartSLOduration=12.390609457 podStartE2EDuration="14.53337598s" totalImagesPullingTime="2.142766523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:10:58.548956603 +0000 UTC m=+593.864458716" watchObservedRunningTime="2026-07-28 18:11:09.53337598 +0000 UTC m=+604.848878092"
Jul 28 18:11:56.919477 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:56.919441    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"]
Jul 28 18:11:56.925791 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:56.925702    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:56.929921 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:56.929898    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 28 18:11:56.929921 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:56.929918    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-87r8d\""
Jul 28 18:11:56.930102 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:56.929926    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 28 18:11:56.930102 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:56.929948    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 28 18:11:56.935633 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:56.935607    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"]
Jul 28 18:11:57.026178 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.026140    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bdebdea-05f6-4988-b87a-eef055328634-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.026178 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.026181    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bdebdea-05f6-4988-b87a-eef055328634-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.026368 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.026204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bdebdea-05f6-4988-b87a-eef055328634-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.026368 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.026256    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cxvk\" (UniqueName: \"kubernetes.io/projected/4bdebdea-05f6-4988-b87a-eef055328634-kube-api-access-4cxvk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.026368 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.026304    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bdebdea-05f6-4988-b87a-eef055328634-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.026470 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.026370    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bdebdea-05f6-4988-b87a-eef055328634-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.127544 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.127507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cxvk\" (UniqueName: \"kubernetes.io/projected/4bdebdea-05f6-4988-b87a-eef055328634-kube-api-access-4cxvk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.127544 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.127546    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bdebdea-05f6-4988-b87a-eef055328634-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.127752 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.127581    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bdebdea-05f6-4988-b87a-eef055328634-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.127752 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.127611    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bdebdea-05f6-4988-b87a-eef055328634-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.127752 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.127632    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bdebdea-05f6-4988-b87a-eef055328634-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.127752 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.127648    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bdebdea-05f6-4988-b87a-eef055328634-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.127977 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.127957    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bdebdea-05f6-4988-b87a-eef055328634-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.128038 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.128004    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bdebdea-05f6-4988-b87a-eef055328634-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.128126 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.128107    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bdebdea-05f6-4988-b87a-eef055328634-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.129860 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.129840    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bdebdea-05f6-4988-b87a-eef055328634-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.130145 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.130129    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bdebdea-05f6-4988-b87a-eef055328634-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.137012 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.136988    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cxvk\" (UniqueName: \"kubernetes.io/projected/4bdebdea-05f6-4988-b87a-eef055328634-kube-api-access-4cxvk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89\" (UID: \"4bdebdea-05f6-4988-b87a-eef055328634\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.237245 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.237170    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:11:57.370617 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.370587    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"]
Jul 28 18:11:57.374983 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:11:57.374953    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdebdea_05f6_4988_b87a_eef055328634.slice/crio-30209b8e4cb6508f4f59ffdb69d47d8fadda9a65022c85dbfaa306875c706af2 WatchSource:0}: Error finding container 30209b8e4cb6508f4f59ffdb69d47d8fadda9a65022c85dbfaa306875c706af2: Status 404 returned error can't find the container with id 30209b8e4cb6508f4f59ffdb69d47d8fadda9a65022c85dbfaa306875c706af2
Jul 28 18:11:57.376699 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.376679    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:11:57.753566 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:57.753410    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89" event={"ID":"4bdebdea-05f6-4988-b87a-eef055328634","Type":"ContainerStarted","Data":"30209b8e4cb6508f4f59ffdb69d47d8fadda9a65022c85dbfaa306875c706af2"}
Jul 28 18:11:58.759077 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:58.759046    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89" event={"ID":"4bdebdea-05f6-4988-b87a-eef055328634","Type":"ContainerStarted","Data":"0fae41ac9d18e0fbeb6f3cf12a4d763afe73247384a0574aa120e33cecb402b8"}
Jul 28 18:11:58.780931 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:11:58.780883    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89" podStartSLOduration=1.585272036 podStartE2EDuration="2.780868275s" totalImagesPullingTime="1.195596239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:11:58.779399685 +0000 UTC m=+654.094901797" watchObservedRunningTime="2026-07-28 18:11:58.780868275 +0000 UTC m=+654.096370387"
Jul 28 18:12:07.238144 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:07.238110    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:12:07.238535 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:07.238168    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:12:07.250926 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:07.250901    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:12:07.805379 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:07.805351    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89"
Jul 28 18:12:16.312515 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.312481    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"]
Jul 28 18:12:16.314921 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.314901    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.317819 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.317792    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 28 18:12:16.326939 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.326917    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"]
Jul 28 18:12:16.371452 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.371423    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtf4p\" (UniqueName: \"kubernetes.io/projected/84807c9f-a8ab-4964-94a9-6ba802476aa6-kube-api-access-xtf4p\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.371569 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.371475    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84807c9f-a8ab-4964-94a9-6ba802476aa6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.371569 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.371535    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84807c9f-a8ab-4964-94a9-6ba802476aa6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.371645 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.371575    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84807c9f-a8ab-4964-94a9-6ba802476aa6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.371645 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.371616    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84807c9f-a8ab-4964-94a9-6ba802476aa6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.371726 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.371669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84807c9f-a8ab-4964-94a9-6ba802476aa6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.472583 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.472554    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84807c9f-a8ab-4964-94a9-6ba802476aa6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.472583 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.472583    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84807c9f-a8ab-4964-94a9-6ba802476aa6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.472783 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.472606    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84807c9f-a8ab-4964-94a9-6ba802476aa6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.472783 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.472739    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84807c9f-a8ab-4964-94a9-6ba802476aa6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.472920 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.472832    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtf4p\" (UniqueName: \"kubernetes.io/projected/84807c9f-a8ab-4964-94a9-6ba802476aa6-kube-api-access-xtf4p\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.472972 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.472939    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84807c9f-a8ab-4964-94a9-6ba802476aa6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.472972 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.472948    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84807c9f-a8ab-4964-94a9-6ba802476aa6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.473185 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.473144    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84807c9f-a8ab-4964-94a9-6ba802476aa6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.473266 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.473252    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84807c9f-a8ab-4964-94a9-6ba802476aa6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.474833 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.474800    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84807c9f-a8ab-4964-94a9-6ba802476aa6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.475190 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.475173    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84807c9f-a8ab-4964-94a9-6ba802476aa6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.482303 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.482265    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtf4p\" (UniqueName: \"kubernetes.io/projected/84807c9f-a8ab-4964-94a9-6ba802476aa6-kube-api-access-xtf4p\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6z8w4\" (UID: \"84807c9f-a8ab-4964-94a9-6ba802476aa6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.624959 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.624876    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:16.751999 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.751835    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"]
Jul 28 18:12:16.754314 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:12:16.754287    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84807c9f_a8ab_4964_94a9_6ba802476aa6.slice/crio-0246454a15e83d010fea5b115cd1322d98ee48c825d070723629bb11b6cb7a48 WatchSource:0}: Error finding container 0246454a15e83d010fea5b115cd1322d98ee48c825d070723629bb11b6cb7a48: Status 404 returned error can't find the container with id 0246454a15e83d010fea5b115cd1322d98ee48c825d070723629bb11b6cb7a48
Jul 28 18:12:16.831777 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:16.831744    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4" event={"ID":"84807c9f-a8ab-4964-94a9-6ba802476aa6","Type":"ContainerStarted","Data":"0246454a15e83d010fea5b115cd1322d98ee48c825d070723629bb11b6cb7a48"}
Jul 28 18:12:17.843248 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:17.843207    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4" event={"ID":"84807c9f-a8ab-4964-94a9-6ba802476aa6","Type":"ContainerStarted","Data":"7385cf2ef87adaef45d98881f0d5d02e0dd239da8faa08ddc0e5056c07de510a"}
Jul 28 18:12:17.870496 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:17.870447    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4" podStartSLOduration=1.680565219 podStartE2EDuration="1.870431575s" totalImagesPullingTime="189.866356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:12:17.868887704 +0000 UTC m=+673.184389842" watchObservedRunningTime="2026-07-28 18:12:17.870431575 +0000 UTC m=+673.185933722"
Jul 28 18:12:26.625353 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:26.625294    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:26.625775 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:26.625402    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:26.637659 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:26.637626    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:26.895851 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:26.895758    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6z8w4"
Jul 28 18:12:39.501498 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.501462    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"]
Jul 28 18:12:39.505067 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.505047    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.507921 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.507899    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 28 18:12:39.515092 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.515072    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"]
Jul 28 18:12:39.575593 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.575547    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e933fba-818a-45e5-9825-e2d73798e776-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.575788 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.575602    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e933fba-818a-45e5-9825-e2d73798e776-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.575788 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.575635    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e933fba-818a-45e5-9825-e2d73798e776-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.575788 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.575739    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e933fba-818a-45e5-9825-e2d73798e776-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.576037 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.575828    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e933fba-818a-45e5-9825-e2d73798e776-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.576037 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.575880    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fkhg\" (UniqueName: \"kubernetes.io/projected/4e933fba-818a-45e5-9825-e2d73798e776-kube-api-access-6fkhg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.676569 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.676529    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fkhg\" (UniqueName: \"kubernetes.io/projected/4e933fba-818a-45e5-9825-e2d73798e776-kube-api-access-6fkhg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.676757 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.676592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e933fba-818a-45e5-9825-e2d73798e776-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.676757 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.676614    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e933fba-818a-45e5-9825-e2d73798e776-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.676757 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.676691    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e933fba-818a-45e5-9825-e2d73798e776-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.676964 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.676764    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e933fba-818a-45e5-9825-e2d73798e776-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.676964 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.676841    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e933fba-818a-45e5-9825-e2d73798e776-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.677074 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.677055    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e933fba-818a-45e5-9825-e2d73798e776-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.677114 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.677099    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e933fba-818a-45e5-9825-e2d73798e776-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.677149 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.677115    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e933fba-818a-45e5-9825-e2d73798e776-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.678939 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.678916    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e933fba-818a-45e5-9825-e2d73798e776-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.679207 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.679191    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e933fba-818a-45e5-9825-e2d73798e776-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.684796 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.684773    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fkhg\" (UniqueName: \"kubernetes.io/projected/4e933fba-818a-45e5-9825-e2d73798e776-kube-api-access-6fkhg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tgszt\" (UID: \"4e933fba-818a-45e5-9825-e2d73798e776\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.816305 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.816266    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:39.948091 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:39.948057    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"]
Jul 28 18:12:39.950901 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:12:39.950865    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e933fba_818a_45e5_9825_e2d73798e776.slice/crio-a31193d9132c9ff629039fdba6039ff2011a2d7c788d9e01ba90ce1af60d1ff8 WatchSource:0}: Error finding container a31193d9132c9ff629039fdba6039ff2011a2d7c788d9e01ba90ce1af60d1ff8: Status 404 returned error can't find the container with id a31193d9132c9ff629039fdba6039ff2011a2d7c788d9e01ba90ce1af60d1ff8
Jul 28 18:12:40.949139 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:40.949107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt" event={"ID":"4e933fba-818a-45e5-9825-e2d73798e776","Type":"ContainerStarted","Data":"915478f432ebd5c357bfe424af493735437fc73ea94ccf5e2e313bcfaa7fb2ff"}
Jul 28 18:12:40.949515 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:40.949149    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt" event={"ID":"4e933fba-818a-45e5-9825-e2d73798e776","Type":"ContainerStarted","Data":"a31193d9132c9ff629039fdba6039ff2011a2d7c788d9e01ba90ce1af60d1ff8"}
Jul 28 18:12:40.968431 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:40.968379    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt" podStartSLOduration=1.7809734000000002 podStartE2EDuration="1.968364293s" totalImagesPullingTime="187.390893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:12:40.967663049 +0000 UTC m=+696.283165186" watchObservedRunningTime="2026-07-28 18:12:40.968364293 +0000 UTC m=+696.283866405"
Jul 28 18:12:49.816634 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:49.816595    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:49.817076 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:49.816653    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:49.829046 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:49.829021    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:12:50.003976 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:12:50.003942    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tgszt"
Jul 28 18:13:54.130703 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:54.130670    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b756f6565-8jn68"]
Jul 28 18:13:54.133564 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:54.133548    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b756f6565-8jn68"
Jul 28 18:13:54.149989 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:54.149967    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b756f6565-8jn68"]
Jul 28 18:13:54.296314 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:54.296284    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-742l2\" (UniqueName: \"kubernetes.io/projected/d61b5f56-4792-4b6e-8e5c-29735e3bc334-kube-api-access-742l2\") pod \"authorino-b756f6565-8jn68\" (UID: \"d61b5f56-4792-4b6e-8e5c-29735e3bc334\") " pod="kuadrant-system/authorino-b756f6565-8jn68"
Jul 28 18:13:54.296469 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:54.296336    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d61b5f56-4792-4b6e-8e5c-29735e3bc334-tls-cert\") pod \"authorino-b756f6565-8jn68\" (UID: \"d61b5f56-4792-4b6e-8e5c-29735e3bc334\") " pod="kuadrant-system/authorino-b756f6565-8jn68"
Jul 28 18:13:54.397693 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:54.397607    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d61b5f56-4792-4b6e-8e5c-29735e3bc334-tls-cert\") pod \"authorino-b756f6565-8jn68\" (UID: \"d61b5f56-4792-4b6e-8e5c-29735e3bc334\") " pod="kuadrant-system/authorino-b756f6565-8jn68"
Jul 28 18:13:54.397890 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:54.397721    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-742l2\" (UniqueName: \"kubernetes.io/projected/d61b5f56-4792-4b6e-8e5c-29735e3bc334-kube-api-access-742l2\") pod \"authorino-b756f6565-8jn68\" (UID: \"d61b5f56-4792-4b6e-8e5c-29735e3bc334\") " pod="kuadrant-system/authorino-b756f6565-8jn68"
Jul 28 18:13:54.400168 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:54.400140    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d61b5f56-4792-4b6e-8e5c-29735e3bc334-tls-cert\") pod \"authorino-b756f6565-8jn68\" (UID: \"d61b5f56-4792-4b6e-8e5c-29735e3bc334\") " pod="kuadrant-system/authorino-b756f6565-8jn68"
Jul 28 18:13:54.407618 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:54.407593    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-742l2\" (UniqueName: \"kubernetes.io/projected/d61b5f56-4792-4b6e-8e5c-29735e3bc334-kube-api-access-742l2\") pod \"authorino-b756f6565-8jn68\" (UID: \"d61b5f56-4792-4b6e-8e5c-29735e3bc334\") " pod="kuadrant-system/authorino-b756f6565-8jn68"
Jul 28 18:13:54.441398 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:54.441368    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b756f6565-8jn68"
Jul 28 18:13:54.587406 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:54.587382    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b756f6565-8jn68"]
Jul 28 18:13:54.589969 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:13:54.589945    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd61b5f56_4792_4b6e_8e5c_29735e3bc334.slice/crio-9bcf1fa2c2ca49411f18e111bcc2c81d6e81a00a0f0a41cb1f5d95f7989d4677 WatchSource:0}: Error finding container 9bcf1fa2c2ca49411f18e111bcc2c81d6e81a00a0f0a41cb1f5d95f7989d4677: Status 404 returned error can't find the container with id 9bcf1fa2c2ca49411f18e111bcc2c81d6e81a00a0f0a41cb1f5d95f7989d4677
Jul 28 18:13:55.257552 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:55.257517    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b756f6565-8jn68" event={"ID":"d61b5f56-4792-4b6e-8e5c-29735e3bc334","Type":"ContainerStarted","Data":"1ec31af324caccde153c2633ad008cdcd7581269ec11dcd9cfbd259721d67bfc"}
Jul 28 18:13:55.257552 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:55.257556    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b756f6565-8jn68" event={"ID":"d61b5f56-4792-4b6e-8e5c-29735e3bc334","Type":"ContainerStarted","Data":"9bcf1fa2c2ca49411f18e111bcc2c81d6e81a00a0f0a41cb1f5d95f7989d4677"}
Jul 28 18:13:55.283822 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:55.283757    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b756f6565-8jn68" podStartSLOduration=0.810998592 podStartE2EDuration="1.283741925s" totalImagesPullingTime="472.743333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:13:55.282090057 +0000 UTC m=+770.597592171" watchObservedRunningTime="2026-07-28 18:13:55.283741925 +0000 UTC m=+770.599244038"
Jul 28 18:13:55.324347 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:55.324254    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bd58cb456-ndhhc"]
Jul 28 18:13:55.324735 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:55.324702    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bd58cb456-ndhhc" podUID="3d769d2a-e6cb-4471-8b9d-1f58544bd72b" containerName="authorino" containerID="cri-o://f5b1535369b2bdd4ba588373f957c86a7e410174a96ae71f4acab003a65e53a9" gracePeriod=30
Jul 28 18:13:55.588391 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:55.588318    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bd58cb456-ndhhc"
Jul 28 18:13:55.608956 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:55.608901    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d769d2a-e6cb-4471-8b9d-1f58544bd72b-tls-cert\") pod \"3d769d2a-e6cb-4471-8b9d-1f58544bd72b\" (UID: \"3d769d2a-e6cb-4471-8b9d-1f58544bd72b\") "
Jul 28 18:13:55.609160 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:55.608982    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsnq4\" (UniqueName: \"kubernetes.io/projected/3d769d2a-e6cb-4471-8b9d-1f58544bd72b-kube-api-access-tsnq4\") pod \"3d769d2a-e6cb-4471-8b9d-1f58544bd72b\" (UID: \"3d769d2a-e6cb-4471-8b9d-1f58544bd72b\") "
Jul 28 18:13:55.612007 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:55.611951    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d769d2a-e6cb-4471-8b9d-1f58544bd72b-kube-api-access-tsnq4" (OuterVolumeSpecName: "kube-api-access-tsnq4") pod "3d769d2a-e6cb-4471-8b9d-1f58544bd72b" (UID: "3d769d2a-e6cb-4471-8b9d-1f58544bd72b"). InnerVolumeSpecName "kube-api-access-tsnq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:13:55.620776 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:55.620749    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d769d2a-e6cb-4471-8b9d-1f58544bd72b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3d769d2a-e6cb-4471-8b9d-1f58544bd72b" (UID: "3d769d2a-e6cb-4471-8b9d-1f58544bd72b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:13:55.709876 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:55.709845    2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d769d2a-e6cb-4471-8b9d-1f58544bd72b-tls-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:13:55.709876 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:55.709872    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsnq4\" (UniqueName: \"kubernetes.io/projected/3d769d2a-e6cb-4471-8b9d-1f58544bd72b-kube-api-access-tsnq4\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:13:56.265287 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:56.265252    2564 generic.go:356] "Generic (PLEG): container finished" podID="3d769d2a-e6cb-4471-8b9d-1f58544bd72b" containerID="f5b1535369b2bdd4ba588373f957c86a7e410174a96ae71f4acab003a65e53a9" exitCode=0
Jul 28 18:13:56.265748 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:56.265298    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bd58cb456-ndhhc"
Jul 28 18:13:56.265748 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:56.265348    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bd58cb456-ndhhc" event={"ID":"3d769d2a-e6cb-4471-8b9d-1f58544bd72b","Type":"ContainerDied","Data":"f5b1535369b2bdd4ba588373f957c86a7e410174a96ae71f4acab003a65e53a9"}
Jul 28 18:13:56.265748 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:56.265397    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bd58cb456-ndhhc" event={"ID":"3d769d2a-e6cb-4471-8b9d-1f58544bd72b","Type":"ContainerDied","Data":"4b06743dff6fb011874572921a7d6cf3628e77d331006ebc1f67252c46e36bf5"}
Jul 28 18:13:56.265748 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:56.265419    2564 scope.go:117] "RemoveContainer" containerID="f5b1535369b2bdd4ba588373f957c86a7e410174a96ae71f4acab003a65e53a9"
Jul 28 18:13:56.274506 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:56.274484    2564 scope.go:117] "RemoveContainer" containerID="f5b1535369b2bdd4ba588373f957c86a7e410174a96ae71f4acab003a65e53a9"
Jul 28 18:13:56.274846 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:13:56.274801    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b1535369b2bdd4ba588373f957c86a7e410174a96ae71f4acab003a65e53a9\": container with ID starting with f5b1535369b2bdd4ba588373f957c86a7e410174a96ae71f4acab003a65e53a9 not found: ID does not exist" containerID="f5b1535369b2bdd4ba588373f957c86a7e410174a96ae71f4acab003a65e53a9"
Jul 28 18:13:56.274931 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:56.274856    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b1535369b2bdd4ba588373f957c86a7e410174a96ae71f4acab003a65e53a9"} err="failed to get container status \"f5b1535369b2bdd4ba588373f957c86a7e410174a96ae71f4acab003a65e53a9\": rpc error: code = NotFound desc = could not find container \"f5b1535369b2bdd4ba588373f957c86a7e410174a96ae71f4acab003a65e53a9\": container with ID starting with f5b1535369b2bdd4ba588373f957c86a7e410174a96ae71f4acab003a65e53a9 not found: ID does not exist"
Jul 28 18:13:56.303006 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:56.302972    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bd58cb456-ndhhc"]
Jul 28 18:13:56.322564 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:56.322532    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bd58cb456-ndhhc"]
Jul 28 18:13:57.217079 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:13:57.217050    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d769d2a-e6cb-4471-8b9d-1f58544bd72b" path="/var/lib/kubelet/pods/3d769d2a-e6cb-4471-8b9d-1f58544bd72b/volumes"
Jul 28 18:14:43.217924 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:43.217895    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56cfbfb6cd-vlrpt"]
Jul 28 18:14:43.218387 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:43.218078    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt" podUID="06876849-cb38-4a56-98df-348ee07cbc63" containerName="manager" containerID="cri-o://a128feda881a684216ccb4167ce1c7090ffc6ad2fb9fb33897f1bd6f38396ec7" gracePeriod=30
Jul 28 18:14:43.459012 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:43.458987    2564 generic.go:356] "Generic (PLEG): container finished" podID="06876849-cb38-4a56-98df-348ee07cbc63" containerID="a128feda881a684216ccb4167ce1c7090ffc6ad2fb9fb33897f1bd6f38396ec7" exitCode=0
Jul 28 18:14:43.459124 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:43.459044    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt" event={"ID":"06876849-cb38-4a56-98df-348ee07cbc63","Type":"ContainerDied","Data":"a128feda881a684216ccb4167ce1c7090ffc6ad2fb9fb33897f1bd6f38396ec7"}
Jul 28 18:14:43.459124 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:43.459090    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt" event={"ID":"06876849-cb38-4a56-98df-348ee07cbc63","Type":"ContainerDied","Data":"392a481187977dae8b2a2d0fd45c3a1aa226690a4a2c9e9eedad4718926c7e7d"}
Jul 28 18:14:43.459124 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:43.459107    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="392a481187977dae8b2a2d0fd45c3a1aa226690a4a2c9e9eedad4718926c7e7d"
Jul 28 18:14:43.459600 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:43.459587    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt"
Jul 28 18:14:43.602926 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:43.602895    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzctr\" (UniqueName: \"kubernetes.io/projected/06876849-cb38-4a56-98df-348ee07cbc63-kube-api-access-nzctr\") pod \"06876849-cb38-4a56-98df-348ee07cbc63\" (UID: \"06876849-cb38-4a56-98df-348ee07cbc63\") "
Jul 28 18:14:43.603102 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:43.602951    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/06876849-cb38-4a56-98df-348ee07cbc63-webhook-cert\") pod \"06876849-cb38-4a56-98df-348ee07cbc63\" (UID: \"06876849-cb38-4a56-98df-348ee07cbc63\") "
Jul 28 18:14:43.605073 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:43.605043    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06876849-cb38-4a56-98df-348ee07cbc63-kube-api-access-nzctr" (OuterVolumeSpecName: "kube-api-access-nzctr") pod "06876849-cb38-4a56-98df-348ee07cbc63" (UID: "06876849-cb38-4a56-98df-348ee07cbc63"). InnerVolumeSpecName "kube-api-access-nzctr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:14:43.605178 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:43.605100    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06876849-cb38-4a56-98df-348ee07cbc63-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "06876849-cb38-4a56-98df-348ee07cbc63" (UID: "06876849-cb38-4a56-98df-348ee07cbc63"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:14:43.703782 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:43.703745    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzctr\" (UniqueName: \"kubernetes.io/projected/06876849-cb38-4a56-98df-348ee07cbc63-kube-api-access-nzctr\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:14:43.703782 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:43.703776    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/06876849-cb38-4a56-98df-348ee07cbc63-webhook-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:14:44.462932 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:44.462903    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cfbfb6cd-vlrpt"
Jul 28 18:14:44.484343 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:44.484317    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56cfbfb6cd-vlrpt"]
Jul 28 18:14:44.486698 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:44.486675    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56cfbfb6cd-vlrpt"]
Jul 28 18:14:45.217147 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:14:45.217108    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06876849-cb38-4a56-98df-348ee07cbc63" path="/var/lib/kubelet/pods/06876849-cb38-4a56-98df-348ee07cbc63/volumes"
Jul 28 18:15:00.144354 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:00.144315    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754375-wbszx"]
Jul 28 18:15:00.144744 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:00.144603    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06876849-cb38-4a56-98df-348ee07cbc63" containerName="manager"
Jul 28 18:15:00.144744 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:00.144613    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="06876849-cb38-4a56-98df-348ee07cbc63" containerName="manager"
Jul 28 18:15:00.144744 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:00.144630    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d769d2a-e6cb-4471-8b9d-1f58544bd72b" containerName="authorino"
Jul 28 18:15:00.144744 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:00.144637    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d769d2a-e6cb-4471-8b9d-1f58544bd72b" containerName="authorino"
Jul 28 18:15:00.144744 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:00.144691    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="06876849-cb38-4a56-98df-348ee07cbc63" containerName="manager"
Jul 28 18:15:00.144744 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:00.144701    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d769d2a-e6cb-4471-8b9d-1f58544bd72b" containerName="authorino"
Jul 28 18:15:00.147421 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:00.147403    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754375-wbszx"
Jul 28 18:15:00.150514 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:00.150490    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\""
Jul 28 18:15:00.156093 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:00.156066    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754375-wbszx"]
Jul 28 18:15:00.158390 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:00.158370    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754375-wbszx"
Jul 28 18:15:00.290043 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:00.290015    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754375-wbszx"]
Jul 28 18:15:00.293494 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:15:00.293471    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7264ab_2557_4578_b1d0_b5b7c959c67a.slice/crio-58b47a8328e10780296ad1f719dc12d7db788b84fccc1649f8e03607b8da0813 WatchSource:0}: Error finding container 58b47a8328e10780296ad1f719dc12d7db788b84fccc1649f8e03607b8da0813: Status 404 returned error can't find the container with id 58b47a8328e10780296ad1f719dc12d7db788b84fccc1649f8e03607b8da0813
Jul 28 18:15:00.534293 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:00.534257    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754375-wbszx" event={"ID":"4c7264ab-2557-4578-b1d0-b5b7c959c67a","Type":"ContainerStarted","Data":"58b47a8328e10780296ad1f719dc12d7db788b84fccc1649f8e03607b8da0813"}
Jul 28 18:15:03.547994 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:03.547913    2564 generic.go:356] "Generic (PLEG): container finished" podID="4c7264ab-2557-4578-b1d0-b5b7c959c67a" containerID="d432cd563374beb2f1d4553517ead7f4575e98e89a65fd1eed157f728cafe009" exitCode=0
Jul 28 18:15:03.548314 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:03.548003    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754375-wbszx" event={"ID":"4c7264ab-2557-4578-b1d0-b5b7c959c67a","Type":"ContainerDied","Data":"d432cd563374beb2f1d4553517ead7f4575e98e89a65fd1eed157f728cafe009"}
Jul 28 18:15:04.682182 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:04.682156    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754375-wbszx"
Jul 28 18:15:05.563148 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:05.563124    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754375-wbszx"
Jul 28 18:15:05.563316 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:05.563117    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754375-wbszx" event={"ID":"4c7264ab-2557-4578-b1d0-b5b7c959c67a","Type":"ContainerDied","Data":"58b47a8328e10780296ad1f719dc12d7db788b84fccc1649f8e03607b8da0813"}
Jul 28 18:15:05.563316 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:05.563254    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58b47a8328e10780296ad1f719dc12d7db788b84fccc1649f8e03607b8da0813"
Jul 28 18:15:45.325448 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.325412    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56cfbfb6cd-vh4qn"]
Jul 28 18:15:45.325882 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.325693    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c7264ab-2557-4578-b1d0-b5b7c959c67a" containerName="cleanup"
Jul 28 18:15:45.325882 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.325703    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c7264ab-2557-4578-b1d0-b5b7c959c67a" containerName="cleanup"
Jul 28 18:15:45.325882 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.325774    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c7264ab-2557-4578-b1d0-b5b7c959c67a" containerName="cleanup"
Jul 28 18:15:45.328518 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.328501    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn"
Jul 28 18:15:45.331406 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.331374    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 18:15:45.332872 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.332849    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-twsgn\""
Jul 28 18:15:45.333059 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.332901    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 18:15:45.342986 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.342963    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56cfbfb6cd-vh4qn"]
Jul 28 18:15:45.389581 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.389551    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd8zg\" (UniqueName: \"kubernetes.io/projected/cbf15bcd-985f-4f7c-9933-9c744808e6c1-kube-api-access-sd8zg\") pod \"maas-controller-56cfbfb6cd-vh4qn\" (UID: \"cbf15bcd-985f-4f7c-9933-9c744808e6c1\") " pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn"
Jul 28 18:15:45.389719 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.389591    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbf15bcd-985f-4f7c-9933-9c744808e6c1-webhook-cert\") pod \"maas-controller-56cfbfb6cd-vh4qn\" (UID: \"cbf15bcd-985f-4f7c-9933-9c744808e6c1\") " pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn"
Jul 28 18:15:45.490441 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.490405    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd8zg\" (UniqueName: \"kubernetes.io/projected/cbf15bcd-985f-4f7c-9933-9c744808e6c1-kube-api-access-sd8zg\") pod \"maas-controller-56cfbfb6cd-vh4qn\" (UID: \"cbf15bcd-985f-4f7c-9933-9c744808e6c1\") " pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn"
Jul 28 18:15:45.490631 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.490462    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbf15bcd-985f-4f7c-9933-9c744808e6c1-webhook-cert\") pod \"maas-controller-56cfbfb6cd-vh4qn\" (UID: \"cbf15bcd-985f-4f7c-9933-9c744808e6c1\") " pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn"
Jul 28 18:15:45.492912 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.492888    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbf15bcd-985f-4f7c-9933-9c744808e6c1-webhook-cert\") pod \"maas-controller-56cfbfb6cd-vh4qn\" (UID: \"cbf15bcd-985f-4f7c-9933-9c744808e6c1\") " pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn"
Jul 28 18:15:45.500483 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.500456    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd8zg\" (UniqueName: \"kubernetes.io/projected/cbf15bcd-985f-4f7c-9933-9c744808e6c1-kube-api-access-sd8zg\") pod \"maas-controller-56cfbfb6cd-vh4qn\" (UID: \"cbf15bcd-985f-4f7c-9933-9c744808e6c1\") " pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn"
Jul 28 18:15:45.639855 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.639754    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn"
Jul 28 18:15:45.778037 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:45.778012    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56cfbfb6cd-vh4qn"]
Jul 28 18:15:45.780212 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:15:45.780185    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf15bcd_985f_4f7c_9933_9c744808e6c1.slice/crio-766c3690657fa89b9e12219125d33cda5366c96d00bfae191f1be2f3644f22a0 WatchSource:0}: Error finding container 766c3690657fa89b9e12219125d33cda5366c96d00bfae191f1be2f3644f22a0: Status 404 returned error can't find the container with id 766c3690657fa89b9e12219125d33cda5366c96d00bfae191f1be2f3644f22a0
Jul 28 18:15:46.735461 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:46.735420    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn" event={"ID":"cbf15bcd-985f-4f7c-9933-9c744808e6c1","Type":"ContainerStarted","Data":"c3681ba69616927e2a9a0cfeec2dcc7958af4131fbaf27171dc410b32a6be9a0"}
Jul 28 18:15:46.735844 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:46.735480    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn" event={"ID":"cbf15bcd-985f-4f7c-9933-9c744808e6c1","Type":"ContainerStarted","Data":"766c3690657fa89b9e12219125d33cda5366c96d00bfae191f1be2f3644f22a0"}
Jul 28 18:15:46.735844 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:46.735549    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn"
Jul 28 18:15:46.755997 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:46.755950    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn" podStartSLOduration=1.290295759 podStartE2EDuration="1.755936495s" totalImagesPullingTime="465.640736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:15:46.754933046 +0000 UTC m=+882.070435193" watchObservedRunningTime="2026-07-28 18:15:46.755936495 +0000 UTC m=+882.071438607"
Jul 28 18:15:57.743758 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:15:57.743681    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn"
Jul 28 18:16:02.410477 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:02.410432    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56cfbfb6cd-vh4qn"]
Jul 28 18:16:02.410963 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:02.410701    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn" podUID="cbf15bcd-985f-4f7c-9933-9c744808e6c1" containerName="manager" containerID="cri-o://c3681ba69616927e2a9a0cfeec2dcc7958af4131fbaf27171dc410b32a6be9a0" gracePeriod=30
Jul 28 18:16:02.661097 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:02.661032    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn"
Jul 28 18:16:02.803296 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:02.803260    2564 generic.go:356] "Generic (PLEG): container finished" podID="cbf15bcd-985f-4f7c-9933-9c744808e6c1" containerID="c3681ba69616927e2a9a0cfeec2dcc7958af4131fbaf27171dc410b32a6be9a0" exitCode=0
Jul 28 18:16:02.803494 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:02.803307    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn"
Jul 28 18:16:02.803494 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:02.803337    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn" event={"ID":"cbf15bcd-985f-4f7c-9933-9c744808e6c1","Type":"ContainerDied","Data":"c3681ba69616927e2a9a0cfeec2dcc7958af4131fbaf27171dc410b32a6be9a0"}
Jul 28 18:16:02.803494 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:02.803390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cfbfb6cd-vh4qn" event={"ID":"cbf15bcd-985f-4f7c-9933-9c744808e6c1","Type":"ContainerDied","Data":"766c3690657fa89b9e12219125d33cda5366c96d00bfae191f1be2f3644f22a0"}
Jul 28 18:16:02.803494 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:02.803420    2564 scope.go:117] "RemoveContainer" containerID="c3681ba69616927e2a9a0cfeec2dcc7958af4131fbaf27171dc410b32a6be9a0"
Jul 28 18:16:02.811476 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:02.811450    2564 scope.go:117] "RemoveContainer" containerID="c3681ba69616927e2a9a0cfeec2dcc7958af4131fbaf27171dc410b32a6be9a0"
Jul 28 18:16:02.811718 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:16:02.811700    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3681ba69616927e2a9a0cfeec2dcc7958af4131fbaf27171dc410b32a6be9a0\": container with ID starting with c3681ba69616927e2a9a0cfeec2dcc7958af4131fbaf27171dc410b32a6be9a0 not found: ID does not exist" containerID="c3681ba69616927e2a9a0cfeec2dcc7958af4131fbaf27171dc410b32a6be9a0"
Jul 28 18:16:02.811767 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:02.811727    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3681ba69616927e2a9a0cfeec2dcc7958af4131fbaf27171dc410b32a6be9a0"} err="failed to get container status \"c3681ba69616927e2a9a0cfeec2dcc7958af4131fbaf27171dc410b32a6be9a0\": rpc error: code = NotFound desc = could not find container \"c3681ba69616927e2a9a0cfeec2dcc7958af4131fbaf27171dc410b32a6be9a0\": container with ID starting with c3681ba69616927e2a9a0cfeec2dcc7958af4131fbaf27171dc410b32a6be9a0 not found: ID does not exist"
Jul 28 18:16:02.838064 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:02.838029    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd8zg\" (UniqueName: \"kubernetes.io/projected/cbf15bcd-985f-4f7c-9933-9c744808e6c1-kube-api-access-sd8zg\") pod \"cbf15bcd-985f-4f7c-9933-9c744808e6c1\" (UID: \"cbf15bcd-985f-4f7c-9933-9c744808e6c1\") "
Jul 28 18:16:02.838183 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:02.838083    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbf15bcd-985f-4f7c-9933-9c744808e6c1-webhook-cert\") pod \"cbf15bcd-985f-4f7c-9933-9c744808e6c1\" (UID: \"cbf15bcd-985f-4f7c-9933-9c744808e6c1\") "
Jul 28 18:16:02.840195 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:02.840171    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf15bcd-985f-4f7c-9933-9c744808e6c1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "cbf15bcd-985f-4f7c-9933-9c744808e6c1" (UID: "cbf15bcd-985f-4f7c-9933-9c744808e6c1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:16:02.840305 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:02.840173    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf15bcd-985f-4f7c-9933-9c744808e6c1-kube-api-access-sd8zg" (OuterVolumeSpecName: "kube-api-access-sd8zg") pod "cbf15bcd-985f-4f7c-9933-9c744808e6c1" (UID: "cbf15bcd-985f-4f7c-9933-9c744808e6c1"). InnerVolumeSpecName "kube-api-access-sd8zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:16:02.939416 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:02.939329    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd8zg\" (UniqueName: \"kubernetes.io/projected/cbf15bcd-985f-4f7c-9933-9c744808e6c1-kube-api-access-sd8zg\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:16:02.939416 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:02.939362    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbf15bcd-985f-4f7c-9933-9c744808e6c1-webhook-cert\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:16:03.126427 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:03.126397    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56cfbfb6cd-vh4qn"]
Jul 28 18:16:03.130682 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:03.130655    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56cfbfb6cd-vh4qn"]
Jul 28 18:16:03.215664 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:03.215592    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbf15bcd-985f-4f7c-9933-9c744808e6c1" path="/var/lib/kubelet/pods/cbf15bcd-985f-4f7c-9933-9c744808e6c1/volumes"
Jul 28 18:16:03.941737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:03.941704    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56cfbfb6cd-xs7wh"]
Jul 28 18:16:03.942154 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:03.941996    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf15bcd-985f-4f7c-9933-9c744808e6c1" containerName="manager"
Jul 28 18:16:03.942154 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:03.942006    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf15bcd-985f-4f7c-9933-9c744808e6c1" containerName="manager"
Jul 28 18:16:03.942154 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:03.942058    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbf15bcd-985f-4f7c-9933-9c744808e6c1" containerName="manager"
Jul 28 18:16:03.946417 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:03.946399    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cfbfb6cd-xs7wh"
Jul 28 18:16:03.950700 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:03.950681    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 18:16:03.950834 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:03.950682    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 18:16:03.950834 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:03.950681    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-twsgn\""
Jul 28 18:16:03.956880 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:03.956856    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56cfbfb6cd-xs7wh"]
Jul 28 18:16:04.048928 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:04.048878    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1ef0fed9-0db5-440c-a8bf-513f51f8b6a8-webhook-cert\") pod \"maas-controller-56cfbfb6cd-xs7wh\" (UID: \"1ef0fed9-0db5-440c-a8bf-513f51f8b6a8\") " pod="opendatahub/maas-controller-56cfbfb6cd-xs7wh"
Jul 28 18:16:04.049105 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:04.048950    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjmdl\" (UniqueName: \"kubernetes.io/projected/1ef0fed9-0db5-440c-a8bf-513f51f8b6a8-kube-api-access-vjmdl\") pod \"maas-controller-56cfbfb6cd-xs7wh\" (UID: \"1ef0fed9-0db5-440c-a8bf-513f51f8b6a8\") " pod="opendatahub/maas-controller-56cfbfb6cd-xs7wh"
Jul 28 18:16:04.149693 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:04.149658    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1ef0fed9-0db5-440c-a8bf-513f51f8b6a8-webhook-cert\") pod \"maas-controller-56cfbfb6cd-xs7wh\" (UID: \"1ef0fed9-0db5-440c-a8bf-513f51f8b6a8\") " pod="opendatahub/maas-controller-56cfbfb6cd-xs7wh"
Jul 28 18:16:04.149797 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:04.149706    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjmdl\" (UniqueName: \"kubernetes.io/projected/1ef0fed9-0db5-440c-a8bf-513f51f8b6a8-kube-api-access-vjmdl\") pod \"maas-controller-56cfbfb6cd-xs7wh\" (UID: \"1ef0fed9-0db5-440c-a8bf-513f51f8b6a8\") " pod="opendatahub/maas-controller-56cfbfb6cd-xs7wh"
Jul 28 18:16:04.152226 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:04.152198    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1ef0fed9-0db5-440c-a8bf-513f51f8b6a8-webhook-cert\") pod \"maas-controller-56cfbfb6cd-xs7wh\" (UID: \"1ef0fed9-0db5-440c-a8bf-513f51f8b6a8\") " pod="opendatahub/maas-controller-56cfbfb6cd-xs7wh"
Jul 28 18:16:04.168368 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:04.168342    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjmdl\" (UniqueName: \"kubernetes.io/projected/1ef0fed9-0db5-440c-a8bf-513f51f8b6a8-kube-api-access-vjmdl\") pod \"maas-controller-56cfbfb6cd-xs7wh\" (UID: \"1ef0fed9-0db5-440c-a8bf-513f51f8b6a8\") " pod="opendatahub/maas-controller-56cfbfb6cd-xs7wh"
Jul 28 18:16:04.256641 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:04.256609    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cfbfb6cd-xs7wh"
Jul 28 18:16:04.389071 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:04.389048    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56cfbfb6cd-xs7wh"]
Jul 28 18:16:04.391324 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:16:04.391299    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef0fed9_0db5_440c_a8bf_513f51f8b6a8.slice/crio-7f1f6f237d42b71bf4719760ce1cffe79f786cf327feb64ba37b22d9a6ef36f0 WatchSource:0}: Error finding container 7f1f6f237d42b71bf4719760ce1cffe79f786cf327feb64ba37b22d9a6ef36f0: Status 404 returned error can't find the container with id 7f1f6f237d42b71bf4719760ce1cffe79f786cf327feb64ba37b22d9a6ef36f0
Jul 28 18:16:04.813238 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:04.813204    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cfbfb6cd-xs7wh" event={"ID":"1ef0fed9-0db5-440c-a8bf-513f51f8b6a8","Type":"ContainerStarted","Data":"7f1f6f237d42b71bf4719760ce1cffe79f786cf327feb64ba37b22d9a6ef36f0"}
Jul 28 18:16:05.818502 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:05.818462    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cfbfb6cd-xs7wh" event={"ID":"1ef0fed9-0db5-440c-a8bf-513f51f8b6a8","Type":"ContainerStarted","Data":"fea5cfd80066973d477cfffc7acb9852caf39ae7bccb0b23165f8aeaf6cad24d"}
Jul 28 18:16:05.818989 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:05.818525    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56cfbfb6cd-xs7wh"
Jul 28 18:16:05.838885 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:05.838831    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-56cfbfb6cd-xs7wh" podStartSLOduration=2.405453945 podStartE2EDuration="2.838800265s" totalImagesPullingTime="433.34632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:16:05.83754532 +0000 UTC m=+901.153047433" watchObservedRunningTime="2026-07-28 18:16:05.838800265 +0000 UTC m=+901.154302379"
Jul 28 18:16:16.824873 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:16:16.824839    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56cfbfb6cd-xs7wh"
Jul 28 18:17:05.181983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:17:05.181950    2564 scope.go:117] "RemoveContainer" containerID="a128feda881a684216ccb4167ce1c7090ffc6ad2fb9fb33897f1bd6f38396ec7"
Jul 28 18:25:31.700393 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.700357    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"]
Jul 28 18:25:31.703783 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.703764    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.706816 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.706785    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-6ed4c9b0-kserve-self-signed-certs\""
Jul 28 18:25:31.713138 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.713108    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"]
Jul 28 18:25:31.753451 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.753424    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-dshm\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.753580 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.753463    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9zdd\" (UniqueName: \"kubernetes.io/projected/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-kube-api-access-t9zdd\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.753580 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.753504    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-tmp-dir\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.753689 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.753580    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-home\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.753689 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.753616    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-tls-certs\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.753689 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.753651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-model-cache\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.854767 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.854740    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-dshm\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.854913 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.854784    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9zdd\" (UniqueName: \"kubernetes.io/projected/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-kube-api-access-t9zdd\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.854913 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.854861    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-tmp-dir\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.854913 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.854902    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-home\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.855078 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.854925    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-tls-certs\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.855078 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.854953    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-model-cache\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.855337 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.855301    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-home\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.855337 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.855317    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-tmp-dir\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.855490 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.855345    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-model-cache\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.857027 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.856999    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-dshm\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.857316 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.857295    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-tls-certs\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:31.863566 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:31.863542    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9zdd\" (UniqueName: \"kubernetes.io/projected/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-kube-api-access-t9zdd\") pod \"e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") " pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:32.013460 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:32.013434    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:32.146055 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:32.146027    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"]
Jul 28 18:25:32.149038 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:25:32.149005    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4dd2e61_93f5_4bb4_a230_0ce6b7ff1915.slice/crio-3e5038965480c67d2e1c78166e6d5a4e5edde7673b95a1cc1776ac07154ebd8a WatchSource:0}: Error finding container 3e5038965480c67d2e1c78166e6d5a4e5edde7673b95a1cc1776ac07154ebd8a: Status 404 returned error can't find the container with id 3e5038965480c67d2e1c78166e6d5a4e5edde7673b95a1cc1776ac07154ebd8a
Jul 28 18:25:32.175623 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:32.175600    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl" event={"ID":"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915","Type":"ContainerStarted","Data":"3e5038965480c67d2e1c78166e6d5a4e5edde7673b95a1cc1776ac07154ebd8a"}
Jul 28 18:25:33.180949 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:33.180915    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl" event={"ID":"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915","Type":"ContainerStarted","Data":"c44b3a456e4bf8d1fdcfb34f212bda0fb6e8170dcc44a75409fa8d61c6a74ace"}
Jul 28 18:25:33.200856 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:33.200792    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl" podStartSLOduration=2.200775214 podStartE2EDuration="2.200775214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:25:33.19904995 +0000 UTC m=+1468.514552062" watchObservedRunningTime="2026-07-28 18:25:33.200775214 +0000 UTC m=+1468.516277328"
Jul 28 18:25:42.014204 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:42.014164    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:42.014204 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:42.014221    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:42.026495 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:42.026471    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:42.229566 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:42.229535    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:25:49.607078 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:25:49.607045    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"]
Jul 28 18:26:04.626176 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:04.626111    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl" podUID="d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915" containerName="main" containerID="cri-o://c44b3a456e4bf8d1fdcfb34f212bda0fb6e8170dcc44a75409fa8d61c6a74ace" gracePeriod=45
Jul 28 18:26:04.869754 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:04.869733    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:26:05.033740 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.033704    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9zdd\" (UniqueName: \"kubernetes.io/projected/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-kube-api-access-t9zdd\") pod \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") "
Jul 28 18:26:05.033952 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.033767    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-home\") pod \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") "
Jul 28 18:26:05.033952 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.033835    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-dshm\") pod \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") "
Jul 28 18:26:05.033952 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.033880    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-tmp-dir\") pod \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") "
Jul 28 18:26:05.033952 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.033905    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-model-cache\") pod \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") "
Jul 28 18:26:05.033952 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.033944    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-tls-certs\") pod \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\" (UID: \"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915\") "
Jul 28 18:26:05.034178 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.034057    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-home" (OuterVolumeSpecName: "home") pod "d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915" (UID: "d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:05.034178 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.034116    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915" (UID: "d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:05.034291 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.034213    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915" (UID: "d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:05.034344 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.034309    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:26:05.034344 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.034329    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:26:05.034344 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.034342    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:26:05.036017 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.035986    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-dshm" (OuterVolumeSpecName: "dshm") pod "d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915" (UID: "d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:05.036132 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.036063    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-kube-api-access-t9zdd" (OuterVolumeSpecName: "kube-api-access-t9zdd") pod "d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915" (UID: "d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915"). InnerVolumeSpecName "kube-api-access-t9zdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:26:05.036132 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.036066    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915" (UID: "d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:26:05.135016 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.134986    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:26:05.135016 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.135010    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:26:05.135016 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.135021    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9zdd\" (UniqueName: \"kubernetes.io/projected/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915-kube-api-access-t9zdd\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:26:05.312233 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.312162    2564 generic.go:356] "Generic (PLEG): container finished" podID="d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915" containerID="c44b3a456e4bf8d1fdcfb34f212bda0fb6e8170dcc44a75409fa8d61c6a74ace" exitCode=0
Jul 28 18:26:05.312233 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.312204    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"
Jul 28 18:26:05.312402 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.312200    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl" event={"ID":"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915","Type":"ContainerDied","Data":"c44b3a456e4bf8d1fdcfb34f212bda0fb6e8170dcc44a75409fa8d61c6a74ace"}
Jul 28 18:26:05.312402 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.312312    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl" event={"ID":"d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915","Type":"ContainerDied","Data":"3e5038965480c67d2e1c78166e6d5a4e5edde7673b95a1cc1776ac07154ebd8a"}
Jul 28 18:26:05.312402 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.312330    2564 scope.go:117] "RemoveContainer" containerID="c44b3a456e4bf8d1fdcfb34f212bda0fb6e8170dcc44a75409fa8d61c6a74ace"
Jul 28 18:26:05.320168 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.320151    2564 scope.go:117] "RemoveContainer" containerID="c44b3a456e4bf8d1fdcfb34f212bda0fb6e8170dcc44a75409fa8d61c6a74ace"
Jul 28 18:26:05.320390 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:26:05.320370    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44b3a456e4bf8d1fdcfb34f212bda0fb6e8170dcc44a75409fa8d61c6a74ace\": container with ID starting with c44b3a456e4bf8d1fdcfb34f212bda0fb6e8170dcc44a75409fa8d61c6a74ace not found: ID does not exist" containerID="c44b3a456e4bf8d1fdcfb34f212bda0fb6e8170dcc44a75409fa8d61c6a74ace"
Jul 28 18:26:05.320460 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.320396    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44b3a456e4bf8d1fdcfb34f212bda0fb6e8170dcc44a75409fa8d61c6a74ace"} err="failed to get container status \"c44b3a456e4bf8d1fdcfb34f212bda0fb6e8170dcc44a75409fa8d61c6a74ace\": rpc error: code = NotFound desc = could not find container \"c44b3a456e4bf8d1fdcfb34f212bda0fb6e8170dcc44a75409fa8d61c6a74ace\": container with ID starting with c44b3a456e4bf8d1fdcfb34f212bda0fb6e8170dcc44a75409fa8d61c6a74ace not found: ID does not exist"
Jul 28 18:26:05.331383 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.331364    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"]
Jul 28 18:26:05.335903 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:05.335884    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-6ed4c9b0-kserve-5c656876c9-gvfgl"]
Jul 28 18:26:07.217428 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:26:07.217397    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915" path="/var/lib/kubelet/pods/d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915/volumes"
Jul 28 18:29:29.820693 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:29.820620    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"]
Jul 28 18:29:29.821171 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:29.820963    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915" containerName="main"
Jul 28 18:29:29.821171 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:29.820975    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915" containerName="main"
Jul 28 18:29:29.821171 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:29.821041    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4dd2e61-93f5-4bb4-a230-0ce6b7ff1915" containerName="main"
Jul 28 18:29:29.823926 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:29.823909    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"
Jul 28 18:29:29.827091 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:29.827065    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-314f4856\""
Jul 28 18:29:29.827091 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:29.827065    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-314f4856-dockercfg-cgm8v\""
Jul 28 18:29:29.831888 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:29.831862    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"]
Jul 28 18:29:29.906876 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:29.906842    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6acab5-81e5-42e6-85bc-7e71c6463dcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj\" (UID: \"0c6acab5-81e5-42e6-85bc-7e71c6463dcc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"
Jul 28 18:29:29.906876 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:29.906881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfhws\" (UniqueName: \"kubernetes.io/projected/0c6acab5-81e5-42e6-85bc-7e71c6463dcc-kube-api-access-nfhws\") pod \"payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj\" (UID: \"0c6acab5-81e5-42e6-85bc-7e71c6463dcc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"
Jul 28 18:29:30.008134 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:30.008098    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6acab5-81e5-42e6-85bc-7e71c6463dcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj\" (UID: \"0c6acab5-81e5-42e6-85bc-7e71c6463dcc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"
Jul 28 18:29:30.008292 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:30.008140    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfhws\" (UniqueName: \"kubernetes.io/projected/0c6acab5-81e5-42e6-85bc-7e71c6463dcc-kube-api-access-nfhws\") pod \"payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj\" (UID: \"0c6acab5-81e5-42e6-85bc-7e71c6463dcc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"
Jul 28 18:29:30.008777 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:30.008757    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6acab5-81e5-42e6-85bc-7e71c6463dcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj\" (UID: \"0c6acab5-81e5-42e6-85bc-7e71c6463dcc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"
Jul 28 18:29:30.017769 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:30.017738    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfhws\" (UniqueName: \"kubernetes.io/projected/0c6acab5-81e5-42e6-85bc-7e71c6463dcc-kube-api-access-nfhws\") pod \"payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj\" (UID: \"0c6acab5-81e5-42e6-85bc-7e71c6463dcc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"
Jul 28 18:29:30.135330 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:30.135223    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"
Jul 28 18:29:30.268493 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:30.268467    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"]
Jul 28 18:29:30.271394 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:29:30.271367    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6acab5_81e5_42e6_85bc_7e71c6463dcc.slice/crio-8436293e9ce1d3201cfc54b341578cdfb50990a4b46f640749cbb62916151e68 WatchSource:0}: Error finding container 8436293e9ce1d3201cfc54b341578cdfb50990a4b46f640749cbb62916151e68: Status 404 returned error can't find the container with id 8436293e9ce1d3201cfc54b341578cdfb50990a4b46f640749cbb62916151e68
Jul 28 18:29:30.273418 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:30.273401    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:29:31.175566 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:31.175529    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj" event={"ID":"0c6acab5-81e5-42e6-85bc-7e71c6463dcc","Type":"ContainerStarted","Data":"8436293e9ce1d3201cfc54b341578cdfb50990a4b46f640749cbb62916151e68"}
Jul 28 18:29:32.181179 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:32.181093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj" event={"ID":"0c6acab5-81e5-42e6-85bc-7e71c6463dcc","Type":"ContainerStarted","Data":"73151f540585fe671c3635a5c5dd1e8bfc4c8d8b794a0e7cfbd826f1d992c4f0"}
Jul 28 18:29:32.181551 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:32.181221    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"
Jul 28 18:29:32.198056 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:32.197969    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj" podStartSLOduration=1.601631185 podStartE2EDuration="3.197951013s" totalImagesPullingTime="1.596319828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:29:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:29:32.19728702 +0000 UTC m=+1707.512789133" watchObservedRunningTime="2026-07-28 18:29:32.197951013 +0000 UTC m=+1707.513453130"
Jul 28 18:29:43.187989 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:43.187958    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"
Jul 28 18:29:54.585412 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:54.585374    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-314f4856-gdfc2"]
Jul 28 18:29:54.588466 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:54.588449    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-314f4856-gdfc2"
Jul 28 18:29:54.591123 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:54.591099    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\""
Jul 28 18:29:54.591249 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:54.591129    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 18:29:54.596351 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:54.596008    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-314f4856-gdfc2"]
Jul 28 18:29:54.712982 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:54.712948    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e092a978-957a-43d8-876c-09d46f456a83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-314f4856-gdfc2\" (UID: \"e092a978-957a-43d8-876c-09d46f456a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-314f4856-gdfc2"
Jul 28 18:29:54.814044 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:54.814009    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e092a978-957a-43d8-876c-09d46f456a83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-314f4856-gdfc2\" (UID: \"e092a978-957a-43d8-876c-09d46f456a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-314f4856-gdfc2"
Jul 28 18:29:54.814672 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:54.814653    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e092a978-957a-43d8-876c-09d46f456a83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-314f4856-gdfc2\" (UID: \"e092a978-957a-43d8-876c-09d46f456a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-314f4856-gdfc2"
Jul 28 18:29:54.901099 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:54.901016    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-314f4856-gdfc2"
Jul 28 18:29:55.043754 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:55.043726    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-314f4856-gdfc2"]
Jul 28 18:29:55.046600 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:29:55.046570    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode092a978_957a_43d8_876c_09d46f456a83.slice/crio-1aa4f4c36c7ca140cc546f71eb4ae472c04d760b22c6f059ab487ceb7f9dce3c WatchSource:0}: Error finding container 1aa4f4c36c7ca140cc546f71eb4ae472c04d760b22c6f059ab487ceb7f9dce3c: Status 404 returned error can't find the container with id 1aa4f4c36c7ca140cc546f71eb4ae472c04d760b22c6f059ab487ceb7f9dce3c
Jul 28 18:29:55.277885 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:55.277847    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-314f4856-gdfc2" event={"ID":"e092a978-957a-43d8-876c-09d46f456a83","Type":"ContainerStarted","Data":"1aa4f4c36c7ca140cc546f71eb4ae472c04d760b22c6f059ab487ceb7f9dce3c"}
Jul 28 18:29:56.284130 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:56.284098    2564 generic.go:356] "Generic (PLEG): container finished" podID="e092a978-957a-43d8-876c-09d46f456a83" containerID="a105724971a55dc7a5c4efe1ca1385a1e55a45372a4936a377428a17a3e6d3a6" exitCode=0
Jul 28 18:29:56.284511 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:56.284150    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-314f4856-gdfc2" event={"ID":"e092a978-957a-43d8-876c-09d46f456a83","Type":"ContainerDied","Data":"a105724971a55dc7a5c4efe1ca1385a1e55a45372a4936a377428a17a3e6d3a6"}
Jul 28 18:29:57.409272 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:57.409251    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-314f4856-gdfc2"
Jul 28 18:29:57.538500 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:57.538416    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e092a978-957a-43d8-876c-09d46f456a83-maas-api-service-ca\") pod \"e092a978-957a-43d8-876c-09d46f456a83\" (UID: \"e092a978-957a-43d8-876c-09d46f456a83\") "
Jul 28 18:29:57.538776 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:57.538754    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e092a978-957a-43d8-876c-09d46f456a83-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e092a978-957a-43d8-876c-09d46f456a83" (UID: "e092a978-957a-43d8-876c-09d46f456a83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:29:57.639260 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:57.639222    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e092a978-957a-43d8-876c-09d46f456a83-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:29:58.294672 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:58.294646    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-314f4856-gdfc2"
Jul 28 18:29:58.294850 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:58.294643    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-314f4856-gdfc2" event={"ID":"e092a978-957a-43d8-876c-09d46f456a83","Type":"ContainerDied","Data":"1aa4f4c36c7ca140cc546f71eb4ae472c04d760b22c6f059ab487ceb7f9dce3c"}
Jul 28 18:29:58.294850 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:58.294774    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aa4f4c36c7ca140cc546f71eb4ae472c04d760b22c6f059ab487ceb7f9dce3c"
Jul 28 18:29:59.640074 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:59.640045    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"]
Jul 28 18:29:59.640486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:59.640277    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj" podUID="0c6acab5-81e5-42e6-85bc-7e71c6463dcc" containerName="payload-pre-processing" containerID="cri-o://73151f540585fe671c3635a5c5dd1e8bfc4c8d8b794a0e7cfbd826f1d992c4f0" gracePeriod=30
Jul 28 18:29:59.762758 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:59.762719    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-314f4856-gdfc2"]
Jul 28 18:29:59.764816 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:59.764783    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-314f4856-gdfc2"]
Jul 28 18:29:59.883170 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:29:59.883147    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"
Jul 28 18:30:00.060318 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.060281    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfhws\" (UniqueName: \"kubernetes.io/projected/0c6acab5-81e5-42e6-85bc-7e71c6463dcc-kube-api-access-nfhws\") pod \"0c6acab5-81e5-42e6-85bc-7e71c6463dcc\" (UID: \"0c6acab5-81e5-42e6-85bc-7e71c6463dcc\") "
Jul 28 18:30:00.060512 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.060327    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6acab5-81e5-42e6-85bc-7e71c6463dcc-plugins-config-volume\") pod \"0c6acab5-81e5-42e6-85bc-7e71c6463dcc\" (UID: \"0c6acab5-81e5-42e6-85bc-7e71c6463dcc\") "
Jul 28 18:30:00.060731 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.060702    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6acab5-81e5-42e6-85bc-7e71c6463dcc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0c6acab5-81e5-42e6-85bc-7e71c6463dcc" (UID: "0c6acab5-81e5-42e6-85bc-7e71c6463dcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:30:00.062515 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.062494    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6acab5-81e5-42e6-85bc-7e71c6463dcc-kube-api-access-nfhws" (OuterVolumeSpecName: "kube-api-access-nfhws") pod "0c6acab5-81e5-42e6-85bc-7e71c6463dcc" (UID: "0c6acab5-81e5-42e6-85bc-7e71c6463dcc"). InnerVolumeSpecName "kube-api-access-nfhws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:30:00.129936 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.129904    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754390-5ww2d"]
Jul 28 18:30:00.130187 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.130176    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e092a978-957a-43d8-876c-09d46f456a83" containerName="revoke-keys"
Jul 28 18:30:00.130226 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.130189    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e092a978-957a-43d8-876c-09d46f456a83" containerName="revoke-keys"
Jul 28 18:30:00.130226 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.130207    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6acab5-81e5-42e6-85bc-7e71c6463dcc" containerName="payload-pre-processing"
Jul 28 18:30:00.130226 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.130212    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6acab5-81e5-42e6-85bc-7e71c6463dcc" containerName="payload-pre-processing"
Jul 28 18:30:00.130319 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.130261    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c6acab5-81e5-42e6-85bc-7e71c6463dcc" containerName="payload-pre-processing"
Jul 28 18:30:00.130319 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.130273    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e092a978-957a-43d8-876c-09d46f456a83" containerName="revoke-keys"
Jul 28 18:30:00.133172 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.133157    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-5ww2d"
Jul 28 18:30:00.135621 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.135597    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\""
Jul 28 18:30:00.140715 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.140692    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754390-5ww2d"]
Jul 28 18:30:00.143329 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.143309    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-5ww2d"
Jul 28 18:30:00.161051 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.161029    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfhws\" (UniqueName: \"kubernetes.io/projected/0c6acab5-81e5-42e6-85bc-7e71c6463dcc-kube-api-access-nfhws\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:30:00.161141 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.161057    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6acab5-81e5-42e6-85bc-7e71c6463dcc-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:30:00.265409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.265385    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754390-5ww2d"]
Jul 28 18:30:00.267883 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:30:00.267859    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7660f4_d709_4018_99cb_5fea6c35c65f.slice/crio-6af027a2d91d31b87bdfbe775b1a5b54ab280968e8ddad8db719adcee5666e05 WatchSource:0}: Error finding container 6af027a2d91d31b87bdfbe775b1a5b54ab280968e8ddad8db719adcee5666e05: Status 404 returned error can't find the container with id 6af027a2d91d31b87bdfbe775b1a5b54ab280968e8ddad8db719adcee5666e05
Jul 28 18:30:00.303798 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.303769    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-5ww2d" event={"ID":"bc7660f4-d709-4018-99cb-5fea6c35c65f","Type":"ContainerStarted","Data":"6af027a2d91d31b87bdfbe775b1a5b54ab280968e8ddad8db719adcee5666e05"}
Jul 28 18:30:00.304934 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.304909    2564 generic.go:356] "Generic (PLEG): container finished" podID="0c6acab5-81e5-42e6-85bc-7e71c6463dcc" containerID="73151f540585fe671c3635a5c5dd1e8bfc4c8d8b794a0e7cfbd826f1d992c4f0" exitCode=0
Jul 28 18:30:00.305032 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.304949    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"
Jul 28 18:30:00.305032 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.304971    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj" event={"ID":"0c6acab5-81e5-42e6-85bc-7e71c6463dcc","Type":"ContainerDied","Data":"73151f540585fe671c3635a5c5dd1e8bfc4c8d8b794a0e7cfbd826f1d992c4f0"}
Jul 28 18:30:00.305032 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.305007    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj" event={"ID":"0c6acab5-81e5-42e6-85bc-7e71c6463dcc","Type":"ContainerDied","Data":"8436293e9ce1d3201cfc54b341578cdfb50990a4b46f640749cbb62916151e68"}
Jul 28 18:30:00.305032 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.305028    2564 scope.go:117] "RemoveContainer" containerID="73151f540585fe671c3635a5c5dd1e8bfc4c8d8b794a0e7cfbd826f1d992c4f0"
Jul 28 18:30:00.312654 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.312563    2564 scope.go:117] "RemoveContainer" containerID="73151f540585fe671c3635a5c5dd1e8bfc4c8d8b794a0e7cfbd826f1d992c4f0"
Jul 28 18:30:00.313051 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:30:00.313024    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73151f540585fe671c3635a5c5dd1e8bfc4c8d8b794a0e7cfbd826f1d992c4f0\": container with ID starting with 73151f540585fe671c3635a5c5dd1e8bfc4c8d8b794a0e7cfbd826f1d992c4f0 not found: ID does not exist" containerID="73151f540585fe671c3635a5c5dd1e8bfc4c8d8b794a0e7cfbd826f1d992c4f0"
Jul 28 18:30:00.313155 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.313061    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73151f540585fe671c3635a5c5dd1e8bfc4c8d8b794a0e7cfbd826f1d992c4f0"} err="failed to get container status \"73151f540585fe671c3635a5c5dd1e8bfc4c8d8b794a0e7cfbd826f1d992c4f0\": rpc error: code = NotFound desc = could not find container \"73151f540585fe671c3635a5c5dd1e8bfc4c8d8b794a0e7cfbd826f1d992c4f0\": container with ID starting with 73151f540585fe671c3635a5c5dd1e8bfc4c8d8b794a0e7cfbd826f1d992c4f0 not found: ID does not exist"
Jul 28 18:30:00.329506 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.329480    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"]
Jul 28 18:30:00.333224 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:00.333203    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-314f4856-69c6498456-mmljj"]
Jul 28 18:30:01.216610 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:01.216582    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c6acab5-81e5-42e6-85bc-7e71c6463dcc" path="/var/lib/kubelet/pods/0c6acab5-81e5-42e6-85bc-7e71c6463dcc/volumes"
Jul 28 18:30:01.217010 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:01.216897    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e092a978-957a-43d8-876c-09d46f456a83" path="/var/lib/kubelet/pods/e092a978-957a-43d8-876c-09d46f456a83/volumes"
Jul 28 18:30:01.310924 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:01.310893    2564 generic.go:356] "Generic (PLEG): container finished" podID="bc7660f4-d709-4018-99cb-5fea6c35c65f" containerID="bfd58d90baf7f31c1d7ed7d577feb4aff985a53775e444f18f63c4a8d61ae477" exitCode=0
Jul 28 18:30:01.311103 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:01.310947    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-5ww2d" event={"ID":"bc7660f4-d709-4018-99cb-5fea6c35c65f","Type":"ContainerDied","Data":"bfd58d90baf7f31c1d7ed7d577feb4aff985a53775e444f18f63c4a8d61ae477"}
Jul 28 18:30:02.443613 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:02.443589    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-5ww2d"
Jul 28 18:30:03.320088 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.320055    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-5ww2d" event={"ID":"bc7660f4-d709-4018-99cb-5fea6c35c65f","Type":"ContainerDied","Data":"6af027a2d91d31b87bdfbe775b1a5b54ab280968e8ddad8db719adcee5666e05"}
Jul 28 18:30:03.320088 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.320081    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-5ww2d"
Jul 28 18:30:03.320289 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.320105    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6af027a2d91d31b87bdfbe775b1a5b54ab280968e8ddad8db719adcee5666e05"
Jul 28 18:30:03.440478 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.440449    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"]
Jul 28 18:30:03.440748 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.440736    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7660f4-d709-4018-99cb-5fea6c35c65f" containerName="cleanup"
Jul 28 18:30:03.440791 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.440750    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7660f4-d709-4018-99cb-5fea6c35c65f" containerName="cleanup"
Jul 28 18:30:03.440840 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.440802    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc7660f4-d709-4018-99cb-5fea6c35c65f" containerName="cleanup"
Jul 28 18:30:03.443722 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.443697    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"
Jul 28 18:30:03.447133 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.446790    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ba40d93a-serving-cert\""
Jul 28 18:30:03.447133 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.446862    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gffgz\""
Jul 28 18:30:03.452337 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.447428    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 18:30:03.452337 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.447703    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 18:30:03.456608 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.456580    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"]
Jul 28 18:30:03.591092 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.591046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05acca15-b94c-4d1f-9df2-cdce685acd8c-maas-api-tls\") pod \"maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6\" (UID: \"05acca15-b94c-4d1f-9df2-cdce685acd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"
Jul 28 18:30:03.591092 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.591089    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05acca15-b94c-4d1f-9df2-cdce685acd8c-openshift-service-ca\") pod \"maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6\" (UID: \"05acca15-b94c-4d1f-9df2-cdce685acd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"
Jul 28 18:30:03.591299 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.591166    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq9xb\" (UniqueName: \"kubernetes.io/projected/05acca15-b94c-4d1f-9df2-cdce685acd8c-kube-api-access-fq9xb\") pod \"maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6\" (UID: \"05acca15-b94c-4d1f-9df2-cdce685acd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"
Jul 28 18:30:03.692195 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.692155    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05acca15-b94c-4d1f-9df2-cdce685acd8c-maas-api-tls\") pod \"maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6\" (UID: \"05acca15-b94c-4d1f-9df2-cdce685acd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"
Jul 28 18:30:03.692195 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.692192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05acca15-b94c-4d1f-9df2-cdce685acd8c-openshift-service-ca\") pod \"maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6\" (UID: \"05acca15-b94c-4d1f-9df2-cdce685acd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"
Jul 28 18:30:03.692406 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.692223    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9xb\" (UniqueName: \"kubernetes.io/projected/05acca15-b94c-4d1f-9df2-cdce685acd8c-kube-api-access-fq9xb\") pod \"maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6\" (UID: \"05acca15-b94c-4d1f-9df2-cdce685acd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"
Jul 28 18:30:03.692858 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.692800    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05acca15-b94c-4d1f-9df2-cdce685acd8c-openshift-service-ca\") pod \"maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6\" (UID: \"05acca15-b94c-4d1f-9df2-cdce685acd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"
Jul 28 18:30:03.694585 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.694556    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05acca15-b94c-4d1f-9df2-cdce685acd8c-maas-api-tls\") pod \"maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6\" (UID: \"05acca15-b94c-4d1f-9df2-cdce685acd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"
Jul 28 18:30:03.701676 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.701655    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq9xb\" (UniqueName: \"kubernetes.io/projected/05acca15-b94c-4d1f-9df2-cdce685acd8c-kube-api-access-fq9xb\") pod \"maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6\" (UID: \"05acca15-b94c-4d1f-9df2-cdce685acd8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"
Jul 28 18:30:03.762381 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.762343    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"
Jul 28 18:30:03.899156 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:03.899122    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"]
Jul 28 18:30:03.902321 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:30:03.902294    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05acca15_b94c_4d1f_9df2_cdce685acd8c.slice/crio-f365511228c79fb98813b0b633fd5c28c78a45cdb54bfea4b486a4dea0684311 WatchSource:0}: Error finding container f365511228c79fb98813b0b633fd5c28c78a45cdb54bfea4b486a4dea0684311: Status 404 returned error can't find the container with id f365511228c79fb98813b0b633fd5c28c78a45cdb54bfea4b486a4dea0684311
Jul 28 18:30:04.073415 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.073377    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"]
Jul 28 18:30:04.077787 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.077766    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"
Jul 28 18:30:04.080916 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.080896    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-ba40d93a-dockercfg-rt299\""
Jul 28 18:30:04.081014 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.080939    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-ba40d93a\""
Jul 28 18:30:04.087869 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.087846    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"]
Jul 28 18:30:04.096838 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.096759    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"]
Jul 28 18:30:04.099917 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.099896    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"
Jul 28 18:30:04.109475 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.109452    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"]
Jul 28 18:30:04.197617 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.197586    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e719176-9a75-4923-bbf1-1fafa24d3cb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd\" (UID: \"2e719176-9a75-4923-bbf1-1fafa24d3cb0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"
Jul 28 18:30:04.197777 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.197704    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dsdp\" (UniqueName: \"kubernetes.io/projected/2e719176-9a75-4923-bbf1-1fafa24d3cb0-kube-api-access-2dsdp\") pod \"payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd\" (UID: \"2e719176-9a75-4923-bbf1-1fafa24d3cb0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"
Jul 28 18:30:04.197777 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.197766    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ecd31d-9a08-4903-bef0-76ed5218764f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74\" (UID: \"c3ecd31d-9a08-4903-bef0-76ed5218764f\") " pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"
Jul 28 18:30:04.197889 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.197835    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crgtz\" (UniqueName: \"kubernetes.io/projected/c3ecd31d-9a08-4903-bef0-76ed5218764f-kube-api-access-crgtz\") pod \"payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74\" (UID: \"c3ecd31d-9a08-4903-bef0-76ed5218764f\") " pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"
Jul 28 18:30:04.298453 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.298412    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dsdp\" (UniqueName: \"kubernetes.io/projected/2e719176-9a75-4923-bbf1-1fafa24d3cb0-kube-api-access-2dsdp\") pod \"payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd\" (UID: \"2e719176-9a75-4923-bbf1-1fafa24d3cb0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"
Jul 28 18:30:04.298637 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.298465    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ecd31d-9a08-4903-bef0-76ed5218764f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74\" (UID: \"c3ecd31d-9a08-4903-bef0-76ed5218764f\") " pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"
Jul 28 18:30:04.298637 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.298504    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crgtz\" (UniqueName: \"kubernetes.io/projected/c3ecd31d-9a08-4903-bef0-76ed5218764f-kube-api-access-crgtz\") pod \"payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74\" (UID: \"c3ecd31d-9a08-4903-bef0-76ed5218764f\") " pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"
Jul 28 18:30:04.298637 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.298542    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e719176-9a75-4923-bbf1-1fafa24d3cb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd\" (UID: \"2e719176-9a75-4923-bbf1-1fafa24d3cb0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"
Jul 28 18:30:04.299286 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.299265    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e719176-9a75-4923-bbf1-1fafa24d3cb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd\" (UID: \"2e719176-9a75-4923-bbf1-1fafa24d3cb0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"
Jul 28 18:30:04.299286 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.299277    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ecd31d-9a08-4903-bef0-76ed5218764f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74\" (UID: \"c3ecd31d-9a08-4903-bef0-76ed5218764f\") " pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"
Jul 28 18:30:04.307283 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.307259    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crgtz\" (UniqueName: \"kubernetes.io/projected/c3ecd31d-9a08-4903-bef0-76ed5218764f-kube-api-access-crgtz\") pod \"payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74\" (UID: \"c3ecd31d-9a08-4903-bef0-76ed5218764f\") " pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"
Jul 28 18:30:04.307566 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.307549    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dsdp\" (UniqueName: \"kubernetes.io/projected/2e719176-9a75-4923-bbf1-1fafa24d3cb0-kube-api-access-2dsdp\") pod \"payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd\" (UID: \"2e719176-9a75-4923-bbf1-1fafa24d3cb0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"
Jul 28 18:30:04.325357 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.325333    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6" event={"ID":"05acca15-b94c-4d1f-9df2-cdce685acd8c","Type":"ContainerStarted","Data":"f365511228c79fb98813b0b633fd5c28c78a45cdb54bfea4b486a4dea0684311"}
Jul 28 18:30:04.388706 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.388634    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"
Jul 28 18:30:04.411039 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.411013    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"
Jul 28 18:30:04.530642 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.530619    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"]
Jul 28 18:30:04.533903 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:30:04.533854    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ecd31d_9a08_4903_bef0_76ed5218764f.slice/crio-f1a2e52ac0eb11b4d03e4e14da8646954971537c8b2ab322c46fade43eb660b2 WatchSource:0}: Error finding container f1a2e52ac0eb11b4d03e4e14da8646954971537c8b2ab322c46fade43eb660b2: Status 404 returned error can't find the container with id f1a2e52ac0eb11b4d03e4e14da8646954971537c8b2ab322c46fade43eb660b2
Jul 28 18:30:04.558453 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:04.558432    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"]
Jul 28 18:30:04.563982 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:30:04.563955    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e719176_9a75_4923_bbf1_1fafa24d3cb0.slice/crio-350e4dd294d639d0efb4ef472fdbc2ba5b489a1f3c41b7b7b50528da3aebdb0f WatchSource:0}: Error finding container 350e4dd294d639d0efb4ef472fdbc2ba5b489a1f3c41b7b7b50528da3aebdb0f: Status 404 returned error can't find the container with id 350e4dd294d639d0efb4ef472fdbc2ba5b489a1f3c41b7b7b50528da3aebdb0f
Jul 28 18:30:05.333129 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:05.333091    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd" event={"ID":"2e719176-9a75-4923-bbf1-1fafa24d3cb0","Type":"ContainerStarted","Data":"73815a775a94ade7f840ede1521c256e78f5769f53835a03aa15a9fc828ef3ce"}
Jul 28 18:30:05.333310 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:05.333152    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd" event={"ID":"2e719176-9a75-4923-bbf1-1fafa24d3cb0","Type":"ContainerStarted","Data":"350e4dd294d639d0efb4ef472fdbc2ba5b489a1f3c41b7b7b50528da3aebdb0f"}
Jul 28 18:30:05.333310 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:05.333183    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"
Jul 28 18:30:05.334873 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:05.334846    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74" event={"ID":"c3ecd31d-9a08-4903-bef0-76ed5218764f","Type":"ContainerStarted","Data":"37a7b615ad8abbd13aed1dcf58a007ec399d5b797c1beb3fe89d3adfec87bbb7"}
Jul 28 18:30:05.335006 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:05.334899    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"
Jul 28 18:30:05.335006 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:05.334921    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74" event={"ID":"c3ecd31d-9a08-4903-bef0-76ed5218764f","Type":"ContainerStarted","Data":"f1a2e52ac0eb11b4d03e4e14da8646954971537c8b2ab322c46fade43eb660b2"}
Jul 28 18:30:05.351777 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:05.351714    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd" podStartSLOduration=1.351691816 podStartE2EDuration="1.351691816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:30:05.349360274 +0000 UTC m=+1740.664862411" watchObservedRunningTime="2026-07-28 18:30:05.351691816 +0000 UTC m=+1740.667193930"
Jul 28 18:30:05.367996 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:05.367933    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74" podStartSLOduration=1.367911573 podStartE2EDuration="1.367911573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:30:05.36510328 +0000 UTC m=+1740.680605435" watchObservedRunningTime="2026-07-28 18:30:05.367911573 +0000 UTC m=+1740.683413687"
Jul 28 18:30:06.341467 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:06.341431    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6" event={"ID":"05acca15-b94c-4d1f-9df2-cdce685acd8c","Type":"ContainerStarted","Data":"ca0f1d6b985a3d0c9d53972fe499ede9a57c8370b5ff84c071023f11b7b82d07"}
Jul 28 18:30:06.341887 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:06.341772    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"
Jul 28 18:30:06.359078 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:06.359031    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6" podStartSLOduration=1.6345585379999998 podStartE2EDuration="3.359016721s" totalImagesPullingTime="1.724458183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:30:06.357706449 +0000 UTC m=+1741.673208564" watchObservedRunningTime="2026-07-28 18:30:06.359016721 +0000 UTC m=+1741.674518833"
Jul 28 18:30:12.351380 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:12.351347    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"
Jul 28 18:30:16.343720 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:16.343687    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"
Jul 28 18:30:16.344190 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:16.343764    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"
Jul 28 18:30:44.692435 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:44.692399    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"]
Jul 28 18:30:44.692935 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:44.692609    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6" podUID="05acca15-b94c-4d1f-9df2-cdce685acd8c" containerName="maas-api" containerID="cri-o://ca0f1d6b985a3d0c9d53972fe499ede9a57c8370b5ff84c071023f11b7b82d07" gracePeriod=30
Jul 28 18:30:44.745349 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:44.745318    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"]
Jul 28 18:30:44.745543 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:44.745507    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74" podUID="c3ecd31d-9a08-4903-bef0-76ed5218764f" containerName="payload-processing" containerID="cri-o://37a7b615ad8abbd13aed1dcf58a007ec399d5b797c1beb3fe89d3adfec87bbb7" gracePeriod=30
Jul 28 18:30:44.752483 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:44.752455    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"]
Jul 28 18:30:44.752839 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:44.752668    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd" podUID="2e719176-9a75-4923-bbf1-1fafa24d3cb0" containerName="payload-pre-processing" containerID="cri-o://73815a775a94ade7f840ede1521c256e78f5769f53835a03aa15a9fc828ef3ce" gracePeriod=30
Jul 28 18:30:45.051738 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.051713    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"
Jul 28 18:30:45.052341 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.052304    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05acca15-b94c-4d1f-9df2-cdce685acd8c-openshift-service-ca\") pod \"05acca15-b94c-4d1f-9df2-cdce685acd8c\" (UID: \"05acca15-b94c-4d1f-9df2-cdce685acd8c\") "
Jul 28 18:30:45.052469 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.052372    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05acca15-b94c-4d1f-9df2-cdce685acd8c-maas-api-tls\") pod \"05acca15-b94c-4d1f-9df2-cdce685acd8c\" (UID: \"05acca15-b94c-4d1f-9df2-cdce685acd8c\") "
Jul 28 18:30:45.052533 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.052474    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq9xb\" (UniqueName: \"kubernetes.io/projected/05acca15-b94c-4d1f-9df2-cdce685acd8c-kube-api-access-fq9xb\") pod \"05acca15-b94c-4d1f-9df2-cdce685acd8c\" (UID: \"05acca15-b94c-4d1f-9df2-cdce685acd8c\") "
Jul 28 18:30:45.052642 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.052616    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05acca15-b94c-4d1f-9df2-cdce685acd8c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "05acca15-b94c-4d1f-9df2-cdce685acd8c" (UID: "05acca15-b94c-4d1f-9df2-cdce685acd8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:30:45.052786 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.052764    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05acca15-b94c-4d1f-9df2-cdce685acd8c-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:30:45.055386 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.055362    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"
Jul 28 18:30:45.055956 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.055927    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05acca15-b94c-4d1f-9df2-cdce685acd8c-kube-api-access-fq9xb" (OuterVolumeSpecName: "kube-api-access-fq9xb") pod "05acca15-b94c-4d1f-9df2-cdce685acd8c" (UID: "05acca15-b94c-4d1f-9df2-cdce685acd8c"). InnerVolumeSpecName "kube-api-access-fq9xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:30:45.055956 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.055933    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05acca15-b94c-4d1f-9df2-cdce685acd8c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "05acca15-b94c-4d1f-9df2-cdce685acd8c" (UID: "05acca15-b94c-4d1f-9df2-cdce685acd8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:30:45.058579 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.058562    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"
Jul 28 18:30:45.153542 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.153503    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dsdp\" (UniqueName: \"kubernetes.io/projected/2e719176-9a75-4923-bbf1-1fafa24d3cb0-kube-api-access-2dsdp\") pod \"2e719176-9a75-4923-bbf1-1fafa24d3cb0\" (UID: \"2e719176-9a75-4923-bbf1-1fafa24d3cb0\") "
Jul 28 18:30:45.153722 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.153553    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e719176-9a75-4923-bbf1-1fafa24d3cb0-plugins-config-volume\") pod \"2e719176-9a75-4923-bbf1-1fafa24d3cb0\" (UID: \"2e719176-9a75-4923-bbf1-1fafa24d3cb0\") "
Jul 28 18:30:45.153722 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.153686    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ecd31d-9a08-4903-bef0-76ed5218764f-plugins-config-volume\") pod \"c3ecd31d-9a08-4903-bef0-76ed5218764f\" (UID: \"c3ecd31d-9a08-4903-bef0-76ed5218764f\") "
Jul 28 18:30:45.153866 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.153740    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crgtz\" (UniqueName: \"kubernetes.io/projected/c3ecd31d-9a08-4903-bef0-76ed5218764f-kube-api-access-crgtz\") pod \"c3ecd31d-9a08-4903-bef0-76ed5218764f\" (UID: \"c3ecd31d-9a08-4903-bef0-76ed5218764f\") "
Jul 28 18:30:45.153998 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.153982    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq9xb\" (UniqueName: \"kubernetes.io/projected/05acca15-b94c-4d1f-9df2-cdce685acd8c-kube-api-access-fq9xb\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:30:45.154052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.154003    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05acca15-b94c-4d1f-9df2-cdce685acd8c-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:30:45.154106 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.154086    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ecd31d-9a08-4903-bef0-76ed5218764f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3ecd31d-9a08-4903-bef0-76ed5218764f" (UID: "c3ecd31d-9a08-4903-bef0-76ed5218764f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:30:45.154174 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.154086    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e719176-9a75-4923-bbf1-1fafa24d3cb0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2e719176-9a75-4923-bbf1-1fafa24d3cb0" (UID: "2e719176-9a75-4923-bbf1-1fafa24d3cb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:30:45.155653 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.155631    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e719176-9a75-4923-bbf1-1fafa24d3cb0-kube-api-access-2dsdp" (OuterVolumeSpecName: "kube-api-access-2dsdp") pod "2e719176-9a75-4923-bbf1-1fafa24d3cb0" (UID: "2e719176-9a75-4923-bbf1-1fafa24d3cb0"). InnerVolumeSpecName "kube-api-access-2dsdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:30:45.155708 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.155676    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ecd31d-9a08-4903-bef0-76ed5218764f-kube-api-access-crgtz" (OuterVolumeSpecName: "kube-api-access-crgtz") pod "c3ecd31d-9a08-4903-bef0-76ed5218764f" (UID: "c3ecd31d-9a08-4903-bef0-76ed5218764f"). InnerVolumeSpecName "kube-api-access-crgtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:30:45.254816 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.254776    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ecd31d-9a08-4903-bef0-76ed5218764f-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:30:45.254926 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.254829    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crgtz\" (UniqueName: \"kubernetes.io/projected/c3ecd31d-9a08-4903-bef0-76ed5218764f-kube-api-access-crgtz\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:30:45.254926 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.254846    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dsdp\" (UniqueName: \"kubernetes.io/projected/2e719176-9a75-4923-bbf1-1fafa24d3cb0-kube-api-access-2dsdp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:30:45.254926 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.254860    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e719176-9a75-4923-bbf1-1fafa24d3cb0-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:30:45.519580 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.519550    2564 generic.go:356] "Generic (PLEG): container finished" podID="c3ecd31d-9a08-4903-bef0-76ed5218764f" containerID="37a7b615ad8abbd13aed1dcf58a007ec399d5b797c1beb3fe89d3adfec87bbb7" exitCode=0
Jul 28 18:30:45.519746 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.519595    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"
Jul 28 18:30:45.519746 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.519637    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74" event={"ID":"c3ecd31d-9a08-4903-bef0-76ed5218764f","Type":"ContainerDied","Data":"37a7b615ad8abbd13aed1dcf58a007ec399d5b797c1beb3fe89d3adfec87bbb7"}
Jul 28 18:30:45.519746 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.519686    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74" event={"ID":"c3ecd31d-9a08-4903-bef0-76ed5218764f","Type":"ContainerDied","Data":"f1a2e52ac0eb11b4d03e4e14da8646954971537c8b2ab322c46fade43eb660b2"}
Jul 28 18:30:45.519746 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.519715    2564 scope.go:117] "RemoveContainer" containerID="37a7b615ad8abbd13aed1dcf58a007ec399d5b797c1beb3fe89d3adfec87bbb7"
Jul 28 18:30:45.520949 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.520930    2564 generic.go:356] "Generic (PLEG): container finished" podID="05acca15-b94c-4d1f-9df2-cdce685acd8c" containerID="ca0f1d6b985a3d0c9d53972fe499ede9a57c8370b5ff84c071023f11b7b82d07" exitCode=0
Jul 28 18:30:45.521051 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.520959    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6" event={"ID":"05acca15-b94c-4d1f-9df2-cdce685acd8c","Type":"ContainerDied","Data":"ca0f1d6b985a3d0c9d53972fe499ede9a57c8370b5ff84c071023f11b7b82d07"}
Jul 28 18:30:45.521051 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.520979    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"
Jul 28 18:30:45.521051 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.521007    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6" event={"ID":"05acca15-b94c-4d1f-9df2-cdce685acd8c","Type":"ContainerDied","Data":"f365511228c79fb98813b0b633fd5c28c78a45cdb54bfea4b486a4dea0684311"}
Jul 28 18:30:45.522545 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.522525    2564 generic.go:356] "Generic (PLEG): container finished" podID="2e719176-9a75-4923-bbf1-1fafa24d3cb0" containerID="73815a775a94ade7f840ede1521c256e78f5769f53835a03aa15a9fc828ef3ce" exitCode=0
Jul 28 18:30:45.522645 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.522554    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd" event={"ID":"2e719176-9a75-4923-bbf1-1fafa24d3cb0","Type":"ContainerDied","Data":"73815a775a94ade7f840ede1521c256e78f5769f53835a03aa15a9fc828ef3ce"}
Jul 28 18:30:45.522645 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.522569    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"
Jul 28 18:30:45.522645 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.522590    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd" event={"ID":"2e719176-9a75-4923-bbf1-1fafa24d3cb0","Type":"ContainerDied","Data":"350e4dd294d639d0efb4ef472fdbc2ba5b489a1f3c41b7b7b50528da3aebdb0f"}
Jul 28 18:30:45.529091 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.529071    2564 scope.go:117] "RemoveContainer" containerID="37a7b615ad8abbd13aed1dcf58a007ec399d5b797c1beb3fe89d3adfec87bbb7"
Jul 28 18:30:45.529334 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:30:45.529315    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a7b615ad8abbd13aed1dcf58a007ec399d5b797c1beb3fe89d3adfec87bbb7\": container with ID starting with 37a7b615ad8abbd13aed1dcf58a007ec399d5b797c1beb3fe89d3adfec87bbb7 not found: ID does not exist" containerID="37a7b615ad8abbd13aed1dcf58a007ec399d5b797c1beb3fe89d3adfec87bbb7"
Jul 28 18:30:45.529390 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.529342    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a7b615ad8abbd13aed1dcf58a007ec399d5b797c1beb3fe89d3adfec87bbb7"} err="failed to get container status \"37a7b615ad8abbd13aed1dcf58a007ec399d5b797c1beb3fe89d3adfec87bbb7\": rpc error: code = NotFound desc = could not find container \"37a7b615ad8abbd13aed1dcf58a007ec399d5b797c1beb3fe89d3adfec87bbb7\": container with ID starting with 37a7b615ad8abbd13aed1dcf58a007ec399d5b797c1beb3fe89d3adfec87bbb7 not found: ID does not exist"
Jul 28 18:30:45.529390 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.529358    2564 scope.go:117] "RemoveContainer" containerID="ca0f1d6b985a3d0c9d53972fe499ede9a57c8370b5ff84c071023f11b7b82d07"
Jul 28 18:30:45.536717 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.536700    2564 scope.go:117] "RemoveContainer" containerID="ca0f1d6b985a3d0c9d53972fe499ede9a57c8370b5ff84c071023f11b7b82d07"
Jul 28 18:30:45.536981 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:30:45.536963    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0f1d6b985a3d0c9d53972fe499ede9a57c8370b5ff84c071023f11b7b82d07\": container with ID starting with ca0f1d6b985a3d0c9d53972fe499ede9a57c8370b5ff84c071023f11b7b82d07 not found: ID does not exist" containerID="ca0f1d6b985a3d0c9d53972fe499ede9a57c8370b5ff84c071023f11b7b82d07"
Jul 28 18:30:45.537041 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.536989    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0f1d6b985a3d0c9d53972fe499ede9a57c8370b5ff84c071023f11b7b82d07"} err="failed to get container status \"ca0f1d6b985a3d0c9d53972fe499ede9a57c8370b5ff84c071023f11b7b82d07\": rpc error: code = NotFound desc = could not find container \"ca0f1d6b985a3d0c9d53972fe499ede9a57c8370b5ff84c071023f11b7b82d07\": container with ID starting with ca0f1d6b985a3d0c9d53972fe499ede9a57c8370b5ff84c071023f11b7b82d07 not found: ID does not exist"
Jul 28 18:30:45.537041 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.537006    2564 scope.go:117] "RemoveContainer" containerID="73815a775a94ade7f840ede1521c256e78f5769f53835a03aa15a9fc828ef3ce"
Jul 28 18:30:45.548872 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.548654    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"]
Jul 28 18:30:45.549697 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.549670    2564 scope.go:117] "RemoveContainer" containerID="73815a775a94ade7f840ede1521c256e78f5769f53835a03aa15a9fc828ef3ce"
Jul 28 18:30:45.551379 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:30:45.551348    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73815a775a94ade7f840ede1521c256e78f5769f53835a03aa15a9fc828ef3ce\": container with ID starting with 73815a775a94ade7f840ede1521c256e78f5769f53835a03aa15a9fc828ef3ce not found: ID does not exist" containerID="73815a775a94ade7f840ede1521c256e78f5769f53835a03aa15a9fc828ef3ce"
Jul 28 18:30:45.551493 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.551387    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73815a775a94ade7f840ede1521c256e78f5769f53835a03aa15a9fc828ef3ce"} err="failed to get container status \"73815a775a94ade7f840ede1521c256e78f5769f53835a03aa15a9fc828ef3ce\": rpc error: code = NotFound desc = could not find container \"73815a775a94ade7f840ede1521c256e78f5769f53835a03aa15a9fc828ef3ce\": container with ID starting with 73815a775a94ade7f840ede1521c256e78f5769f53835a03aa15a9fc828ef3ce not found: ID does not exist"
Jul 28 18:30:45.552071 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.552048    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ba40d93a-6dfd676b78-k5kl6"]
Jul 28 18:30:45.562573 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.562551    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"]
Jul 28 18:30:45.567598 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.567578    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ba40d93a-9c7dfb4b9-rdc74"]
Jul 28 18:30:45.577852 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.577833    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"]
Jul 28 18:30:45.583215 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:45.583191    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ba40d93a-c6b5d989b-9rmrd"]
Jul 28 18:30:47.215758 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:47.215715    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05acca15-b94c-4d1f-9df2-cdce685acd8c" path="/var/lib/kubelet/pods/05acca15-b94c-4d1f-9df2-cdce685acd8c/volumes"
Jul 28 18:30:47.216253 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:47.216237    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e719176-9a75-4923-bbf1-1fafa24d3cb0" path="/var/lib/kubelet/pods/2e719176-9a75-4923-bbf1-1fafa24d3cb0/volumes"
Jul 28 18:30:47.216605 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:47.216585    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3ecd31d-9a08-4903-bef0-76ed5218764f" path="/var/lib/kubelet/pods/c3ecd31d-9a08-4903-bef0-76ed5218764f/volumes"
Jul 28 18:30:48.416644 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.416614    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"]
Jul 28 18:30:48.417104 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.417040    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ecd31d-9a08-4903-bef0-76ed5218764f" containerName="payload-processing"
Jul 28 18:30:48.417104 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.417058    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ecd31d-9a08-4903-bef0-76ed5218764f" containerName="payload-processing"
Jul 28 18:30:48.417104 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.417071    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e719176-9a75-4923-bbf1-1fafa24d3cb0" containerName="payload-pre-processing"
Jul 28 18:30:48.417104 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.417080    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e719176-9a75-4923-bbf1-1fafa24d3cb0" containerName="payload-pre-processing"
Jul 28 18:30:48.417104 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.417099    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05acca15-b94c-4d1f-9df2-cdce685acd8c" containerName="maas-api"
Jul 28 18:30:48.417337 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.417109    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="05acca15-b94c-4d1f-9df2-cdce685acd8c" containerName="maas-api"
Jul 28 18:30:48.417337 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.417192    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e719176-9a75-4923-bbf1-1fafa24d3cb0" containerName="payload-pre-processing"
Jul 28 18:30:48.417337 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.417204    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3ecd31d-9a08-4903-bef0-76ed5218764f" containerName="payload-processing"
Jul 28 18:30:48.417337 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.417215    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="05acca15-b94c-4d1f-9df2-cdce685acd8c" containerName="maas-api"
Jul 28 18:30:48.421944 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.421922    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"
Jul 28 18:30:48.425981 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.425961    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b13f7e6e-serving-cert\""
Jul 28 18:30:48.426129 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.425983    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 18:30:48.426129 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.425998    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gffgz\""
Jul 28 18:30:48.427718 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.427537    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 18:30:48.431043 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.431011    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"]
Jul 28 18:30:48.481496 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.481460    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghlzv\" (UniqueName: \"kubernetes.io/projected/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-kube-api-access-ghlzv\") pod \"maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr\" (UID: \"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"
Jul 28 18:30:48.481673 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.481509    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-maas-api-tls\") pod \"maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr\" (UID: \"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"
Jul 28 18:30:48.481673 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.481566    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-openshift-service-ca\") pod \"maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr\" (UID: \"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"
Jul 28 18:30:48.582191 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.582157    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-openshift-service-ca\") pod \"maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr\" (UID: \"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"
Jul 28 18:30:48.582389 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.582215    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghlzv\" (UniqueName: \"kubernetes.io/projected/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-kube-api-access-ghlzv\") pod \"maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr\" (UID: \"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"
Jul 28 18:30:48.582389 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.582251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-maas-api-tls\") pod \"maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr\" (UID: \"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"
Jul 28 18:30:48.582389 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:30:48.582364    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-serving-cert: secret "maas-api-e2e-ait-b13f7e6e-serving-cert" not found
Jul 28 18:30:48.582555 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:30:48.582446    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-maas-api-tls podName:85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb nodeName:}" failed. No retries permitted until 2026-07-28 18:30:49.082423124 +0000 UTC m=+1784.397925220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-maas-api-tls") pod "maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr" (UID: "85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb") : secret "maas-api-e2e-ait-b13f7e6e-serving-cert" not found
Jul 28 18:30:48.582990 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.582966    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-openshift-service-ca\") pod \"maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr\" (UID: \"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"
Jul 28 18:30:48.591403 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:48.591378    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghlzv\" (UniqueName: \"kubernetes.io/projected/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-kube-api-access-ghlzv\") pod \"maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr\" (UID: \"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"
Jul 28 18:30:49.044362 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.044327    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"]
Jul 28 18:30:49.047696 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.047676    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"
Jul 28 18:30:49.050411 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.050392    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b13f7e6e\""
Jul 28 18:30:49.050411 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.050390    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b13f7e6e-dockercfg-nnchp\""
Jul 28 18:30:49.056491 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.056467    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"]
Jul 28 18:30:49.059635 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.059612    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"
Jul 28 18:30:49.060367 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.060347    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"]
Jul 28 18:30:49.073625 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.073602    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"]
Jul 28 18:30:49.086599 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.086565    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-maas-api-tls\") pod \"maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr\" (UID: \"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"
Jul 28 18:30:49.086731 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.086605    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rxb7\" (UniqueName: \"kubernetes.io/projected/409cc8f6-2684-456e-b229-07b4afe7b9d3-kube-api-access-5rxb7\") pod \"payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p\" (UID: \"409cc8f6-2684-456e-b229-07b4afe7b9d3\") " pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"
Jul 28 18:30:49.086731 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.086666    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm6lf\" (UniqueName: \"kubernetes.io/projected/3440bd30-9f7e-4792-8bd1-22544dbc82be-kube-api-access-qm6lf\") pod \"payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm\" (UID: \"3440bd30-9f7e-4792-8bd1-22544dbc82be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"
Jul 28 18:30:49.086731 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.086723    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409cc8f6-2684-456e-b229-07b4afe7b9d3-plugins-config-volume\") pod \"payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p\" (UID: \"409cc8f6-2684-456e-b229-07b4afe7b9d3\") " pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"
Jul 28 18:30:49.086900 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.086759    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3440bd30-9f7e-4792-8bd1-22544dbc82be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm\" (UID: \"3440bd30-9f7e-4792-8bd1-22544dbc82be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"
Jul 28 18:30:49.088940 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.088920    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-maas-api-tls\") pod \"maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr\" (UID: \"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"
Jul 28 18:30:49.187242 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.187206    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm6lf\" (UniqueName: \"kubernetes.io/projected/3440bd30-9f7e-4792-8bd1-22544dbc82be-kube-api-access-qm6lf\") pod \"payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm\" (UID: \"3440bd30-9f7e-4792-8bd1-22544dbc82be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"
Jul 28 18:30:49.187242 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.187245    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409cc8f6-2684-456e-b229-07b4afe7b9d3-plugins-config-volume\") pod \"payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p\" (UID: \"409cc8f6-2684-456e-b229-07b4afe7b9d3\") " pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"
Jul 28 18:30:49.187522 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.187268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3440bd30-9f7e-4792-8bd1-22544dbc82be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm\" (UID: \"3440bd30-9f7e-4792-8bd1-22544dbc82be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"
Jul 28 18:30:49.187522 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.187350    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rxb7\" (UniqueName: \"kubernetes.io/projected/409cc8f6-2684-456e-b229-07b4afe7b9d3-kube-api-access-5rxb7\") pod \"payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p\" (UID: \"409cc8f6-2684-456e-b229-07b4afe7b9d3\") " pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"
Jul 28 18:30:49.188074 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.188052    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409cc8f6-2684-456e-b229-07b4afe7b9d3-plugins-config-volume\") pod \"payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p\" (UID: \"409cc8f6-2684-456e-b229-07b4afe7b9d3\") " pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"
Jul 28 18:30:49.188254 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.188233    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3440bd30-9f7e-4792-8bd1-22544dbc82be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm\" (UID: \"3440bd30-9f7e-4792-8bd1-22544dbc82be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"
Jul 28 18:30:49.197035 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.197011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm6lf\" (UniqueName: \"kubernetes.io/projected/3440bd30-9f7e-4792-8bd1-22544dbc82be-kube-api-access-qm6lf\") pod \"payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm\" (UID: \"3440bd30-9f7e-4792-8bd1-22544dbc82be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"
Jul 28 18:30:49.197486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.197463    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rxb7\" (UniqueName: \"kubernetes.io/projected/409cc8f6-2684-456e-b229-07b4afe7b9d3-kube-api-access-5rxb7\") pod \"payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p\" (UID: \"409cc8f6-2684-456e-b229-07b4afe7b9d3\") " pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"
Jul 28 18:30:49.333939 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.333853    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"
Jul 28 18:30:49.358154 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.358129    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"
Jul 28 18:30:49.369154 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.369130    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"
Jul 28 18:30:49.502901 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.502870    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"]
Jul 28 18:30:49.505154 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:30:49.505113    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d32dcd_67dd_4d2f_a82a_20c0af8bc1eb.slice/crio-f7d70a3b19981ddee5ad03d782328d0336605d3ead9233ec549fb6eef3e48b13 WatchSource:0}: Error finding container f7d70a3b19981ddee5ad03d782328d0336605d3ead9233ec549fb6eef3e48b13: Status 404 returned error can't find the container with id f7d70a3b19981ddee5ad03d782328d0336605d3ead9233ec549fb6eef3e48b13
Jul 28 18:30:49.518617 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.518588    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"]
Jul 28 18:30:49.520136 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:30:49.520110    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3440bd30_9f7e_4792_8bd1_22544dbc82be.slice/crio-3fed5707eab38b2169e728576ee62a3254a017659f4e406093dc22a7e22c8aa7 WatchSource:0}: Error finding container 3fed5707eab38b2169e728576ee62a3254a017659f4e406093dc22a7e22c8aa7: Status 404 returned error can't find the container with id 3fed5707eab38b2169e728576ee62a3254a017659f4e406093dc22a7e22c8aa7
Jul 28 18:30:49.551323 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.547085    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"]
Jul 28 18:30:49.551323 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.547372    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm" event={"ID":"3440bd30-9f7e-4792-8bd1-22544dbc82be","Type":"ContainerStarted","Data":"3fed5707eab38b2169e728576ee62a3254a017659f4e406093dc22a7e22c8aa7"}
Jul 28 18:30:49.552391 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:30:49.552362    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409cc8f6_2684_456e_b229_07b4afe7b9d3.slice/crio-a2848cbc7fe2085b13ef8392f66e82c1ae4b3c9bcb58943b3261add7c42bf6f2 WatchSource:0}: Error finding container a2848cbc7fe2085b13ef8392f66e82c1ae4b3c9bcb58943b3261add7c42bf6f2: Status 404 returned error can't find the container with id a2848cbc7fe2085b13ef8392f66e82c1ae4b3c9bcb58943b3261add7c42bf6f2
Jul 28 18:30:49.552957 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:49.552851    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr" event={"ID":"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb","Type":"ContainerStarted","Data":"f7d70a3b19981ddee5ad03d782328d0336605d3ead9233ec549fb6eef3e48b13"}
Jul 28 18:30:50.559485 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:50.559447    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm" event={"ID":"3440bd30-9f7e-4792-8bd1-22544dbc82be","Type":"ContainerStarted","Data":"990fcae42da86cf1de973c0cf5e14d986521d99831cd0effef9ebdaab3555eca"}
Jul 28 18:30:50.559965 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:50.559586    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"
Jul 28 18:30:50.560895 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:50.560872    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr" event={"ID":"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb","Type":"ContainerStarted","Data":"acb53fb47203e088f88b1fb406fefc1756fa109fb3ca0fa127555e11f9454b1d"}
Jul 28 18:30:50.561019 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:50.560943    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"
Jul 28 18:30:50.562320 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:50.562302    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p" event={"ID":"409cc8f6-2684-456e-b229-07b4afe7b9d3","Type":"ContainerStarted","Data":"dbfd23787066b0fb1bd15ee9d462ffa2f9a4b002a65e9a1c49df1afdda168ce5"}
Jul 28 18:30:50.562409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:50.562335    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p" event={"ID":"409cc8f6-2684-456e-b229-07b4afe7b9d3","Type":"ContainerStarted","Data":"a2848cbc7fe2085b13ef8392f66e82c1ae4b3c9bcb58943b3261add7c42bf6f2"}
Jul 28 18:30:50.562409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:50.562365    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"
Jul 28 18:30:50.578696 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:50.578652    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm" podStartSLOduration=1.578637626 podStartE2EDuration="1.578637626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:30:50.577369026 +0000 UTC m=+1785.892871139" watchObservedRunningTime="2026-07-28 18:30:50.578637626 +0000 UTC m=+1785.894139738"
Jul 28 18:30:50.594087 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:50.594044    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p" podStartSLOduration=1.594031093 podStartE2EDuration="1.594031093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:30:50.592334119 +0000 UTC m=+1785.907836256" watchObservedRunningTime="2026-07-28 18:30:50.594031093 +0000 UTC m=+1785.909533206"
Jul 28 18:30:50.612193 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:50.612148    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr" podStartSLOduration=2.131384281 podStartE2EDuration="2.612134237s" totalImagesPullingTime="480.749956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:30:50.61087145 +0000 UTC m=+1785.926373584" watchObservedRunningTime="2026-07-28 18:30:50.612134237 +0000 UTC m=+1785.927636349"
Jul 28 18:30:56.572491 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:30:56.572415    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"
Jul 28 18:31:01.571973 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:01.571937    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"
Jul 28 18:31:01.573241 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:01.573219    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"
Jul 28 18:31:19.036417 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.036373    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"]
Jul 28 18:31:19.036887 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.036630    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr" podUID="85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb" containerName="maas-api" containerID="cri-o://acb53fb47203e088f88b1fb406fefc1756fa109fb3ca0fa127555e11f9454b1d" gracePeriod=30
Jul 28 18:31:19.097094 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.097060    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"]
Jul 28 18:31:19.097317 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.097270    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p" podUID="409cc8f6-2684-456e-b229-07b4afe7b9d3" containerName="payload-processing" containerID="cri-o://dbfd23787066b0fb1bd15ee9d462ffa2f9a4b002a65e9a1c49df1afdda168ce5" gracePeriod=30
Jul 28 18:31:19.103708 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.103680    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"]
Jul 28 18:31:19.103949 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.103915    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm" podUID="3440bd30-9f7e-4792-8bd1-22544dbc82be" containerName="payload-pre-processing" containerID="cri-o://990fcae42da86cf1de973c0cf5e14d986521d99831cd0effef9ebdaab3555eca" gracePeriod=30
Jul 28 18:31:19.190226 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:31:19.190188    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409cc8f6_2684_456e_b229_07b4afe7b9d3.slice/crio-conmon-dbfd23787066b0fb1bd15ee9d462ffa2f9a4b002a65e9a1c49df1afdda168ce5.scope\": RecentStats: unable to find data in memory cache]"
Jul 28 18:31:19.370082 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.370052    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"
Jul 28 18:31:19.382876 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.382854    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"
Jul 28 18:31:19.392300 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.392276    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"
Jul 28 18:31:19.439098 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.439062    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409cc8f6-2684-456e-b229-07b4afe7b9d3-plugins-config-volume\") pod \"409cc8f6-2684-456e-b229-07b4afe7b9d3\" (UID: \"409cc8f6-2684-456e-b229-07b4afe7b9d3\") "
Jul 28 18:31:19.439273 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.439111    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rxb7\" (UniqueName: \"kubernetes.io/projected/409cc8f6-2684-456e-b229-07b4afe7b9d3-kube-api-access-5rxb7\") pod \"409cc8f6-2684-456e-b229-07b4afe7b9d3\" (UID: \"409cc8f6-2684-456e-b229-07b4afe7b9d3\") "
Jul 28 18:31:19.439447 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.439425    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409cc8f6-2684-456e-b229-07b4afe7b9d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "409cc8f6-2684-456e-b229-07b4afe7b9d3" (UID: "409cc8f6-2684-456e-b229-07b4afe7b9d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:31:19.441139 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.441118    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409cc8f6-2684-456e-b229-07b4afe7b9d3-kube-api-access-5rxb7" (OuterVolumeSpecName: "kube-api-access-5rxb7") pod "409cc8f6-2684-456e-b229-07b4afe7b9d3" (UID: "409cc8f6-2684-456e-b229-07b4afe7b9d3"). InnerVolumeSpecName "kube-api-access-5rxb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:31:19.539865 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.539832    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm6lf\" (UniqueName: \"kubernetes.io/projected/3440bd30-9f7e-4792-8bd1-22544dbc82be-kube-api-access-qm6lf\") pod \"3440bd30-9f7e-4792-8bd1-22544dbc82be\" (UID: \"3440bd30-9f7e-4792-8bd1-22544dbc82be\") "
Jul 28 18:31:19.540034 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.539878    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-openshift-service-ca\") pod \"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb\" (UID: \"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb\") "
Jul 28 18:31:19.540034 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.539896    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghlzv\" (UniqueName: \"kubernetes.io/projected/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-kube-api-access-ghlzv\") pod \"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb\" (UID: \"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb\") "
Jul 28 18:31:19.540034 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.539931    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3440bd30-9f7e-4792-8bd1-22544dbc82be-plugins-config-volume\") pod \"3440bd30-9f7e-4792-8bd1-22544dbc82be\" (UID: \"3440bd30-9f7e-4792-8bd1-22544dbc82be\") "
Jul 28 18:31:19.540034 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.539993    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-maas-api-tls\") pod \"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb\" (UID: \"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb\") "
Jul 28 18:31:19.540263 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.540244    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409cc8f6-2684-456e-b229-07b4afe7b9d3-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:31:19.540315 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.540266    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rxb7\" (UniqueName: \"kubernetes.io/projected/409cc8f6-2684-456e-b229-07b4afe7b9d3-kube-api-access-5rxb7\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:31:19.540315 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.540262    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb" (UID: "85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:31:19.540409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.540355    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3440bd30-9f7e-4792-8bd1-22544dbc82be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3440bd30-9f7e-4792-8bd1-22544dbc82be" (UID: "3440bd30-9f7e-4792-8bd1-22544dbc82be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:31:19.542056 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.542033    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-kube-api-access-ghlzv" (OuterVolumeSpecName: "kube-api-access-ghlzv") pod "85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb" (UID: "85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb"). InnerVolumeSpecName "kube-api-access-ghlzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:31:19.542056 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.542046    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb" (UID: "85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:31:19.542184 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.542062    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3440bd30-9f7e-4792-8bd1-22544dbc82be-kube-api-access-qm6lf" (OuterVolumeSpecName: "kube-api-access-qm6lf") pod "3440bd30-9f7e-4792-8bd1-22544dbc82be" (UID: "3440bd30-9f7e-4792-8bd1-22544dbc82be"). InnerVolumeSpecName "kube-api-access-qm6lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:31:19.641393 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.641305    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghlzv\" (UniqueName: \"kubernetes.io/projected/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-kube-api-access-ghlzv\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:31:19.641393 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.641339    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3440bd30-9f7e-4792-8bd1-22544dbc82be-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:31:19.641393 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.641348    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:31:19.641393 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.641356    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm6lf\" (UniqueName: \"kubernetes.io/projected/3440bd30-9f7e-4792-8bd1-22544dbc82be-kube-api-access-qm6lf\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:31:19.641393 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.641365    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:31:19.698284 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.698248    2564 generic.go:356] "Generic (PLEG): container finished" podID="3440bd30-9f7e-4792-8bd1-22544dbc82be" containerID="990fcae42da86cf1de973c0cf5e14d986521d99831cd0effef9ebdaab3555eca" exitCode=0
Jul 28 18:31:19.698438 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.698292    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"
Jul 28 18:31:19.698438 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.698323    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm" event={"ID":"3440bd30-9f7e-4792-8bd1-22544dbc82be","Type":"ContainerDied","Data":"990fcae42da86cf1de973c0cf5e14d986521d99831cd0effef9ebdaab3555eca"}
Jul 28 18:31:19.698438 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.698375    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm" event={"ID":"3440bd30-9f7e-4792-8bd1-22544dbc82be","Type":"ContainerDied","Data":"3fed5707eab38b2169e728576ee62a3254a017659f4e406093dc22a7e22c8aa7"}
Jul 28 18:31:19.698438 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.698399    2564 scope.go:117] "RemoveContainer" containerID="990fcae42da86cf1de973c0cf5e14d986521d99831cd0effef9ebdaab3555eca"
Jul 28 18:31:19.699639 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.699614    2564 generic.go:356] "Generic (PLEG): container finished" podID="85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb" containerID="acb53fb47203e088f88b1fb406fefc1756fa109fb3ca0fa127555e11f9454b1d" exitCode=0
Jul 28 18:31:19.699768 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.699657    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"
Jul 28 18:31:19.699768 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.699697    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr" event={"ID":"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb","Type":"ContainerDied","Data":"acb53fb47203e088f88b1fb406fefc1756fa109fb3ca0fa127555e11f9454b1d"}
Jul 28 18:31:19.699768 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.699765    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr" event={"ID":"85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb","Type":"ContainerDied","Data":"f7d70a3b19981ddee5ad03d782328d0336605d3ead9233ec549fb6eef3e48b13"}
Jul 28 18:31:19.701183 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.701158    2564 generic.go:356] "Generic (PLEG): container finished" podID="409cc8f6-2684-456e-b229-07b4afe7b9d3" containerID="dbfd23787066b0fb1bd15ee9d462ffa2f9a4b002a65e9a1c49df1afdda168ce5" exitCode=0
Jul 28 18:31:19.701378 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.701190    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"
Jul 28 18:31:19.701378 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.701199    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p" event={"ID":"409cc8f6-2684-456e-b229-07b4afe7b9d3","Type":"ContainerDied","Data":"dbfd23787066b0fb1bd15ee9d462ffa2f9a4b002a65e9a1c49df1afdda168ce5"}
Jul 28 18:31:19.701378 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.701241    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p" event={"ID":"409cc8f6-2684-456e-b229-07b4afe7b9d3","Type":"ContainerDied","Data":"a2848cbc7fe2085b13ef8392f66e82c1ae4b3c9bcb58943b3261add7c42bf6f2"}
Jul 28 18:31:19.708385 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.708340    2564 scope.go:117] "RemoveContainer" containerID="990fcae42da86cf1de973c0cf5e14d986521d99831cd0effef9ebdaab3555eca"
Jul 28 18:31:19.708706 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:31:19.708670    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990fcae42da86cf1de973c0cf5e14d986521d99831cd0effef9ebdaab3555eca\": container with ID starting with 990fcae42da86cf1de973c0cf5e14d986521d99831cd0effef9ebdaab3555eca not found: ID does not exist" containerID="990fcae42da86cf1de973c0cf5e14d986521d99831cd0effef9ebdaab3555eca"
Jul 28 18:31:19.708784 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.708714    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990fcae42da86cf1de973c0cf5e14d986521d99831cd0effef9ebdaab3555eca"} err="failed to get container status \"990fcae42da86cf1de973c0cf5e14d986521d99831cd0effef9ebdaab3555eca\": rpc error: code = NotFound desc = could not find container \"990fcae42da86cf1de973c0cf5e14d986521d99831cd0effef9ebdaab3555eca\": container with ID starting with 990fcae42da86cf1de973c0cf5e14d986521d99831cd0effef9ebdaab3555eca not found: ID does not exist"
Jul 28 18:31:19.708784 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.708731    2564 scope.go:117] "RemoveContainer" containerID="acb53fb47203e088f88b1fb406fefc1756fa109fb3ca0fa127555e11f9454b1d"
Jul 28 18:31:19.716903 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.716887    2564 scope.go:117] "RemoveContainer" containerID="acb53fb47203e088f88b1fb406fefc1756fa109fb3ca0fa127555e11f9454b1d"
Jul 28 18:31:19.717146 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:31:19.717128    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb53fb47203e088f88b1fb406fefc1756fa109fb3ca0fa127555e11f9454b1d\": container with ID starting with acb53fb47203e088f88b1fb406fefc1756fa109fb3ca0fa127555e11f9454b1d not found: ID does not exist" containerID="acb53fb47203e088f88b1fb406fefc1756fa109fb3ca0fa127555e11f9454b1d"
Jul 28 18:31:19.717208 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.717152    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb53fb47203e088f88b1fb406fefc1756fa109fb3ca0fa127555e11f9454b1d"} err="failed to get container status \"acb53fb47203e088f88b1fb406fefc1756fa109fb3ca0fa127555e11f9454b1d\": rpc error: code = NotFound desc = could not find container \"acb53fb47203e088f88b1fb406fefc1756fa109fb3ca0fa127555e11f9454b1d\": container with ID starting with acb53fb47203e088f88b1fb406fefc1756fa109fb3ca0fa127555e11f9454b1d not found: ID does not exist"
Jul 28 18:31:19.717208 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.717169    2564 scope.go:117] "RemoveContainer" containerID="dbfd23787066b0fb1bd15ee9d462ffa2f9a4b002a65e9a1c49df1afdda168ce5"
Jul 28 18:31:19.722207 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.722187    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"]
Jul 28 18:31:19.725227 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.725210    2564 scope.go:117] "RemoveContainer" containerID="dbfd23787066b0fb1bd15ee9d462ffa2f9a4b002a65e9a1c49df1afdda168ce5"
Jul 28 18:31:19.725522 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:31:19.725497    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbfd23787066b0fb1bd15ee9d462ffa2f9a4b002a65e9a1c49df1afdda168ce5\": container with ID starting with dbfd23787066b0fb1bd15ee9d462ffa2f9a4b002a65e9a1c49df1afdda168ce5 not found: ID does not exist" containerID="dbfd23787066b0fb1bd15ee9d462ffa2f9a4b002a65e9a1c49df1afdda168ce5"
Jul 28 18:31:19.725585 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.725534    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbfd23787066b0fb1bd15ee9d462ffa2f9a4b002a65e9a1c49df1afdda168ce5"} err="failed to get container status \"dbfd23787066b0fb1bd15ee9d462ffa2f9a4b002a65e9a1c49df1afdda168ce5\": rpc error: code = NotFound desc = could not find container \"dbfd23787066b0fb1bd15ee9d462ffa2f9a4b002a65e9a1c49df1afdda168ce5\": container with ID starting with dbfd23787066b0fb1bd15ee9d462ffa2f9a4b002a65e9a1c49df1afdda168ce5 not found: ID does not exist"
Jul 28 18:31:19.726623 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.726606    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b13f7e6e-84ccf7fcfb-rlvvm"]
Jul 28 18:31:19.738960 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.738938    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"]
Jul 28 18:31:19.742911 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.742891    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b13f7e6e-ff5988c4b-v2d7p"]
Jul 28 18:31:19.753291 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.753272    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"]
Jul 28 18:31:19.759080 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:19.759062    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b13f7e6e-999d4fd68-x5qzr"]
Jul 28 18:31:21.215895 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:21.215855    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3440bd30-9f7e-4792-8bd1-22544dbc82be" path="/var/lib/kubelet/pods/3440bd30-9f7e-4792-8bd1-22544dbc82be/volumes"
Jul 28 18:31:21.216376 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:21.216358    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="409cc8f6-2684-456e-b229-07b4afe7b9d3" path="/var/lib/kubelet/pods/409cc8f6-2684-456e-b229-07b4afe7b9d3/volumes"
Jul 28 18:31:21.216754 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:21.216735    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb" path="/var/lib/kubelet/pods/85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb/volumes"
Jul 28 18:31:23.712290 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.712254    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"]
Jul 28 18:31:23.712726 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.712699    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb" containerName="maas-api"
Jul 28 18:31:23.712726 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.712717    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb" containerName="maas-api"
Jul 28 18:31:23.712850 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.712738    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3440bd30-9f7e-4792-8bd1-22544dbc82be" containerName="payload-pre-processing"
Jul 28 18:31:23.712850 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.712748    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3440bd30-9f7e-4792-8bd1-22544dbc82be" containerName="payload-pre-processing"
Jul 28 18:31:23.712850 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.712770    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="409cc8f6-2684-456e-b229-07b4afe7b9d3" containerName="payload-processing"
Jul 28 18:31:23.712850 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.712779    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="409cc8f6-2684-456e-b229-07b4afe7b9d3" containerName="payload-processing"
Jul 28 18:31:23.712983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.712870    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="85d32dcd-67dd-4d2f-a82a-20c0af8bc1eb" containerName="maas-api"
Jul 28 18:31:23.712983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.712884    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="409cc8f6-2684-456e-b229-07b4afe7b9d3" containerName="payload-processing"
Jul 28 18:31:23.712983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.712894    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3440bd30-9f7e-4792-8bd1-22544dbc82be" containerName="payload-pre-processing"
Jul 28 18:31:23.717930 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.717907    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"
Jul 28 18:31:23.721122 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.721097    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 18:31:23.721336 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.721321    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gffgz\""
Jul 28 18:31:23.722314 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.722296    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 18:31:23.722412 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.722324    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-12084f6d-serving-cert\""
Jul 28 18:31:23.732476 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.732453    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"]
Jul 28 18:31:23.876345 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.876305    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db31e462-d99b-4176-8706-575feb13424c-openshift-service-ca\") pod \"maas-api-e2e-derive-12084f6d-67d954b44f-spzwg\" (UID: \"db31e462-d99b-4176-8706-575feb13424c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"
Jul 28 18:31:23.876529 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.876401    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfxlz\" (UniqueName: \"kubernetes.io/projected/db31e462-d99b-4176-8706-575feb13424c-kube-api-access-cfxlz\") pod \"maas-api-e2e-derive-12084f6d-67d954b44f-spzwg\" (UID: \"db31e462-d99b-4176-8706-575feb13424c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"
Jul 28 18:31:23.876529 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.876459    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db31e462-d99b-4176-8706-575feb13424c-maas-api-tls\") pod \"maas-api-e2e-derive-12084f6d-67d954b44f-spzwg\" (UID: \"db31e462-d99b-4176-8706-575feb13424c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"
Jul 28 18:31:23.977641 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.977454    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxlz\" (UniqueName: \"kubernetes.io/projected/db31e462-d99b-4176-8706-575feb13424c-kube-api-access-cfxlz\") pod \"maas-api-e2e-derive-12084f6d-67d954b44f-spzwg\" (UID: \"db31e462-d99b-4176-8706-575feb13424c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"
Jul 28 18:31:23.977984 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.977923    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db31e462-d99b-4176-8706-575feb13424c-maas-api-tls\") pod \"maas-api-e2e-derive-12084f6d-67d954b44f-spzwg\" (UID: \"db31e462-d99b-4176-8706-575feb13424c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"
Jul 28 18:31:23.978094 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.978023    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db31e462-d99b-4176-8706-575feb13424c-openshift-service-ca\") pod \"maas-api-e2e-derive-12084f6d-67d954b44f-spzwg\" (UID: \"db31e462-d99b-4176-8706-575feb13424c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"
Jul 28 18:31:23.978682 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.978658    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db31e462-d99b-4176-8706-575feb13424c-openshift-service-ca\") pod \"maas-api-e2e-derive-12084f6d-67d954b44f-spzwg\" (UID: \"db31e462-d99b-4176-8706-575feb13424c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"
Jul 28 18:31:23.980316 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.980295    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db31e462-d99b-4176-8706-575feb13424c-maas-api-tls\") pod \"maas-api-e2e-derive-12084f6d-67d954b44f-spzwg\" (UID: \"db31e462-d99b-4176-8706-575feb13424c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"
Jul 28 18:31:23.986270 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:23.986250    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfxlz\" (UniqueName: \"kubernetes.io/projected/db31e462-d99b-4176-8706-575feb13424c-kube-api-access-cfxlz\") pod \"maas-api-e2e-derive-12084f6d-67d954b44f-spzwg\" (UID: \"db31e462-d99b-4176-8706-575feb13424c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"
Jul 28 18:31:24.029259 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.029229    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"
Jul 28 18:31:24.158237 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.158204    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"]
Jul 28 18:31:24.161330 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:31:24.161299    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb31e462_d99b_4176_8706_575feb13424c.slice/crio-9195ec7b83bdb913d77749443cc510fc72a5c74f16032e63565662db4d8acd8d WatchSource:0}: Error finding container 9195ec7b83bdb913d77749443cc510fc72a5c74f16032e63565662db4d8acd8d: Status 404 returned error can't find the container with id 9195ec7b83bdb913d77749443cc510fc72a5c74f16032e63565662db4d8acd8d
Jul 28 18:31:24.403124 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.403092    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"]
Jul 28 18:31:24.407337 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.407318    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"
Jul 28 18:31:24.410161 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.410138    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-12084f6d\""
Jul 28 18:31:24.410272 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.410195    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-12084f6d-dockercfg-s98ht\""
Jul 28 18:31:24.416590 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.416566    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"]
Jul 28 18:31:24.438581 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.438553    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"]
Jul 28 18:31:24.441632 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.441616    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"
Jul 28 18:31:24.452023 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.452000    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"]
Jul 28 18:31:24.583481 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.583445    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76zsb\" (UniqueName: \"kubernetes.io/projected/b8becfd8-66a0-4286-9e18-a37a8a569533-kube-api-access-76zsb\") pod \"payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq\" (UID: \"b8becfd8-66a0-4286-9e18-a37a8a569533\") " pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"
Jul 28 18:31:24.583659 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.583502    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8becfd8-66a0-4286-9e18-a37a8a569533-plugins-config-volume\") pod \"payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq\" (UID: \"b8becfd8-66a0-4286-9e18-a37a8a569533\") " pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"
Jul 28 18:31:24.583659 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.583593    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60577033-6c68-48b7-afe2-fbfda687a9fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz\" (UID: \"60577033-6c68-48b7-afe2-fbfda687a9fd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"
Jul 28 18:31:24.583659 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.583638    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx7n6\" (UniqueName: \"kubernetes.io/projected/60577033-6c68-48b7-afe2-fbfda687a9fd-kube-api-access-rx7n6\") pod \"payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz\" (UID: \"60577033-6c68-48b7-afe2-fbfda687a9fd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"
Jul 28 18:31:24.685038 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.684949    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76zsb\" (UniqueName: \"kubernetes.io/projected/b8becfd8-66a0-4286-9e18-a37a8a569533-kube-api-access-76zsb\") pod \"payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq\" (UID: \"b8becfd8-66a0-4286-9e18-a37a8a569533\") " pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"
Jul 28 18:31:24.685038 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.685012    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8becfd8-66a0-4286-9e18-a37a8a569533-plugins-config-volume\") pod \"payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq\" (UID: \"b8becfd8-66a0-4286-9e18-a37a8a569533\") " pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"
Jul 28 18:31:24.685228 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.685059    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60577033-6c68-48b7-afe2-fbfda687a9fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz\" (UID: \"60577033-6c68-48b7-afe2-fbfda687a9fd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"
Jul 28 18:31:24.685228 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.685092    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx7n6\" (UniqueName: \"kubernetes.io/projected/60577033-6c68-48b7-afe2-fbfda687a9fd-kube-api-access-rx7n6\") pod \"payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz\" (UID: \"60577033-6c68-48b7-afe2-fbfda687a9fd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"
Jul 28 18:31:24.685741 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.685720    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8becfd8-66a0-4286-9e18-a37a8a569533-plugins-config-volume\") pod \"payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq\" (UID: \"b8becfd8-66a0-4286-9e18-a37a8a569533\") " pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"
Jul 28 18:31:24.685774 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.685741    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60577033-6c68-48b7-afe2-fbfda687a9fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz\" (UID: \"60577033-6c68-48b7-afe2-fbfda687a9fd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"
Jul 28 18:31:24.693521 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.693490    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx7n6\" (UniqueName: \"kubernetes.io/projected/60577033-6c68-48b7-afe2-fbfda687a9fd-kube-api-access-rx7n6\") pod \"payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz\" (UID: \"60577033-6c68-48b7-afe2-fbfda687a9fd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"
Jul 28 18:31:24.693521 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.693509    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76zsb\" (UniqueName: \"kubernetes.io/projected/b8becfd8-66a0-4286-9e18-a37a8a569533-kube-api-access-76zsb\") pod \"payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq\" (UID: \"b8becfd8-66a0-4286-9e18-a37a8a569533\") " pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"
Jul 28 18:31:24.719065 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.719037    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"
Jul 28 18:31:24.733990 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.733940    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg" event={"ID":"db31e462-d99b-4176-8706-575feb13424c","Type":"ContainerStarted","Data":"9195ec7b83bdb913d77749443cc510fc72a5c74f16032e63565662db4d8acd8d"}
Jul 28 18:31:24.751652 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.751613    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"
Jul 28 18:31:24.862424 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.862339    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"]
Jul 28 18:31:24.864965 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:31:24.864923    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60577033_6c68_48b7_afe2_fbfda687a9fd.slice/crio-bda21650a3909be9ca41ea688488e39288b8e5fe334f1fa2d8d0eb6690b575d2 WatchSource:0}: Error finding container bda21650a3909be9ca41ea688488e39288b8e5fe334f1fa2d8d0eb6690b575d2: Status 404 returned error can't find the container with id bda21650a3909be9ca41ea688488e39288b8e5fe334f1fa2d8d0eb6690b575d2
Jul 28 18:31:24.899632 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:24.899609    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"]
Jul 28 18:31:24.908896 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:31:24.908868    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8becfd8_66a0_4286_9e18_a37a8a569533.slice/crio-cb96b78a19edc57c21b6ac903f372963369120261d456938c62256d748605966 WatchSource:0}: Error finding container cb96b78a19edc57c21b6ac903f372963369120261d456938c62256d748605966: Status 404 returned error can't find the container with id cb96b78a19edc57c21b6ac903f372963369120261d456938c62256d748605966
Jul 28 18:31:25.739698 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:25.739660    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg" event={"ID":"db31e462-d99b-4176-8706-575feb13424c","Type":"ContainerStarted","Data":"e99469fe1793ed06180828b97d72c76fa263dbfc0f573bdde93d82a6ebcdf70d"}
Jul 28 18:31:25.740200 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:25.739754    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"
Jul 28 18:31:25.741206 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:25.741188    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq" event={"ID":"b8becfd8-66a0-4286-9e18-a37a8a569533","Type":"ContainerStarted","Data":"28fac9f43fbe22b7681771663e787c6f7e768d00d78b8b0daf1d0cf5204f921d"}
Jul 28 18:31:25.741269 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:25.741221    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq" event={"ID":"b8becfd8-66a0-4286-9e18-a37a8a569533","Type":"ContainerStarted","Data":"cb96b78a19edc57c21b6ac903f372963369120261d456938c62256d748605966"}
Jul 28 18:31:25.741269 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:25.741257    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"
Jul 28 18:31:25.742408 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:25.742387    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz" event={"ID":"60577033-6c68-48b7-afe2-fbfda687a9fd","Type":"ContainerStarted","Data":"3b906f08cdfc6fc824a2a079847b78e8897216184a2a24d83aa9985764dcce3a"}
Jul 28 18:31:25.742537 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:25.742435    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"
Jul 28 18:31:25.742537 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:25.742458    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz" event={"ID":"60577033-6c68-48b7-afe2-fbfda687a9fd","Type":"ContainerStarted","Data":"bda21650a3909be9ca41ea688488e39288b8e5fe334f1fa2d8d0eb6690b575d2"}
Jul 28 18:31:25.757382 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:25.757335    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg" podStartSLOduration=2.198656201 podStartE2EDuration="2.757322534s" totalImagesPullingTime="558.666333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:31:25.7567632 +0000 UTC m=+1821.072265337" watchObservedRunningTime="2026-07-28 18:31:25.757322534 +0000 UTC m=+1821.072824647"
Jul 28 18:31:25.774699 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:25.774653    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq" podStartSLOduration=1.774639774 podStartE2EDuration="1.774639774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:31:25.772937584 +0000 UTC m=+1821.088439720" watchObservedRunningTime="2026-07-28 18:31:25.774639774 +0000 UTC m=+1821.090141889"
Jul 28 18:31:25.790379 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:25.790326    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz" podStartSLOduration=1.790307312 podStartE2EDuration="1.790307312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:31:25.788400829 +0000 UTC m=+1821.103902974" watchObservedRunningTime="2026-07-28 18:31:25.790307312 +0000 UTC m=+1821.105809427"
Jul 28 18:31:31.752661 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:31.752630    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"
Jul 28 18:31:36.749291 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:36.749189    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"
Jul 28 18:31:36.749291 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:36.749281    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"
Jul 28 18:31:43.302687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.302635    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"]
Jul 28 18:31:43.303475 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.303442    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg" podUID="db31e462-d99b-4176-8706-575feb13424c" containerName="maas-api" containerID="cri-o://e99469fe1793ed06180828b97d72c76fa263dbfc0f573bdde93d82a6ebcdf70d" gracePeriod=30
Jul 28 18:31:43.352685 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.352654    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"]
Jul 28 18:31:43.352918 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.352880    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq" podUID="b8becfd8-66a0-4286-9e18-a37a8a569533" containerName="payload-processing" containerID="cri-o://28fac9f43fbe22b7681771663e787c6f7e768d00d78b8b0daf1d0cf5204f921d" gracePeriod=30
Jul 28 18:31:43.358228 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.358192    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"]
Jul 28 18:31:43.358431 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.358406    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz" podUID="60577033-6c68-48b7-afe2-fbfda687a9fd" containerName="payload-pre-processing" containerID="cri-o://3b906f08cdfc6fc824a2a079847b78e8897216184a2a24d83aa9985764dcce3a" gracePeriod=30
Jul 28 18:31:43.623991 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.623962    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"
Jul 28 18:31:43.640873 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.640846    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8becfd8-66a0-4286-9e18-a37a8a569533-plugins-config-volume\") pod \"b8becfd8-66a0-4286-9e18-a37a8a569533\" (UID: \"b8becfd8-66a0-4286-9e18-a37a8a569533\") "
Jul 28 18:31:43.641049 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.640901    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76zsb\" (UniqueName: \"kubernetes.io/projected/b8becfd8-66a0-4286-9e18-a37a8a569533-kube-api-access-76zsb\") pod \"b8becfd8-66a0-4286-9e18-a37a8a569533\" (UID: \"b8becfd8-66a0-4286-9e18-a37a8a569533\") "
Jul 28 18:31:43.641298 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.641241    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8becfd8-66a0-4286-9e18-a37a8a569533-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b8becfd8-66a0-4286-9e18-a37a8a569533" (UID: "b8becfd8-66a0-4286-9e18-a37a8a569533"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:31:43.644358 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.644325    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8becfd8-66a0-4286-9e18-a37a8a569533-kube-api-access-76zsb" (OuterVolumeSpecName: "kube-api-access-76zsb") pod "b8becfd8-66a0-4286-9e18-a37a8a569533" (UID: "b8becfd8-66a0-4286-9e18-a37a8a569533"). InnerVolumeSpecName "kube-api-access-76zsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:31:43.650653 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.650631    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"
Jul 28 18:31:43.660104 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.660082    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"
Jul 28 18:31:43.742230 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.742195    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60577033-6c68-48b7-afe2-fbfda687a9fd-plugins-config-volume\") pod \"60577033-6c68-48b7-afe2-fbfda687a9fd\" (UID: \"60577033-6c68-48b7-afe2-fbfda687a9fd\") "
Jul 28 18:31:43.742402 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.742240    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db31e462-d99b-4176-8706-575feb13424c-openshift-service-ca\") pod \"db31e462-d99b-4176-8706-575feb13424c\" (UID: \"db31e462-d99b-4176-8706-575feb13424c\") "
Jul 28 18:31:43.742402 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.742259    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx7n6\" (UniqueName: \"kubernetes.io/projected/60577033-6c68-48b7-afe2-fbfda687a9fd-kube-api-access-rx7n6\") pod \"60577033-6c68-48b7-afe2-fbfda687a9fd\" (UID: \"60577033-6c68-48b7-afe2-fbfda687a9fd\") "
Jul 28 18:31:43.742402 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.742291    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db31e462-d99b-4176-8706-575feb13424c-maas-api-tls\") pod \"db31e462-d99b-4176-8706-575feb13424c\" (UID: \"db31e462-d99b-4176-8706-575feb13424c\") "
Jul 28 18:31:43.742402 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.742379    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfxlz\" (UniqueName: \"kubernetes.io/projected/db31e462-d99b-4176-8706-575feb13424c-kube-api-access-cfxlz\") pod \"db31e462-d99b-4176-8706-575feb13424c\" (UID: \"db31e462-d99b-4176-8706-575feb13424c\") "
Jul 28 18:31:43.742633 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.742605    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60577033-6c68-48b7-afe2-fbfda687a9fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60577033-6c68-48b7-afe2-fbfda687a9fd" (UID: "60577033-6c68-48b7-afe2-fbfda687a9fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:31:43.742698 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.742622    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db31e462-d99b-4176-8706-575feb13424c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "db31e462-d99b-4176-8706-575feb13424c" (UID: "db31e462-d99b-4176-8706-575feb13424c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:31:43.742698 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.742637    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8becfd8-66a0-4286-9e18-a37a8a569533-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:31:43.742698 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.742656    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76zsb\" (UniqueName: \"kubernetes.io/projected/b8becfd8-66a0-4286-9e18-a37a8a569533-kube-api-access-76zsb\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:31:43.744430 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.744406    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db31e462-d99b-4176-8706-575feb13424c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "db31e462-d99b-4176-8706-575feb13424c" (UID: "db31e462-d99b-4176-8706-575feb13424c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:31:43.744522 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.744471    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db31e462-d99b-4176-8706-575feb13424c-kube-api-access-cfxlz" (OuterVolumeSpecName: "kube-api-access-cfxlz") pod "db31e462-d99b-4176-8706-575feb13424c" (UID: "db31e462-d99b-4176-8706-575feb13424c"). InnerVolumeSpecName "kube-api-access-cfxlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:31:43.744558 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.744517    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60577033-6c68-48b7-afe2-fbfda687a9fd-kube-api-access-rx7n6" (OuterVolumeSpecName: "kube-api-access-rx7n6") pod "60577033-6c68-48b7-afe2-fbfda687a9fd" (UID: "60577033-6c68-48b7-afe2-fbfda687a9fd"). InnerVolumeSpecName "kube-api-access-rx7n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:31:43.831457 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.831368    2564 generic.go:356] "Generic (PLEG): container finished" podID="60577033-6c68-48b7-afe2-fbfda687a9fd" containerID="3b906f08cdfc6fc824a2a079847b78e8897216184a2a24d83aa9985764dcce3a" exitCode=0
Jul 28 18:31:43.831457 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.831407    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"
Jul 28 18:31:43.831685 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.831454    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz" event={"ID":"60577033-6c68-48b7-afe2-fbfda687a9fd","Type":"ContainerDied","Data":"3b906f08cdfc6fc824a2a079847b78e8897216184a2a24d83aa9985764dcce3a"}
Jul 28 18:31:43.831685 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.831515    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz" event={"ID":"60577033-6c68-48b7-afe2-fbfda687a9fd","Type":"ContainerDied","Data":"bda21650a3909be9ca41ea688488e39288b8e5fe334f1fa2d8d0eb6690b575d2"}
Jul 28 18:31:43.831685 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.831544    2564 scope.go:117] "RemoveContainer" containerID="3b906f08cdfc6fc824a2a079847b78e8897216184a2a24d83aa9985764dcce3a"
Jul 28 18:31:43.832840 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.832790    2564 generic.go:356] "Generic (PLEG): container finished" podID="db31e462-d99b-4176-8706-575feb13424c" containerID="e99469fe1793ed06180828b97d72c76fa263dbfc0f573bdde93d82a6ebcdf70d" exitCode=0
Jul 28 18:31:43.832959 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.832850    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"
Jul 28 18:31:43.832959 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.832883    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg" event={"ID":"db31e462-d99b-4176-8706-575feb13424c","Type":"ContainerDied","Data":"e99469fe1793ed06180828b97d72c76fa263dbfc0f573bdde93d82a6ebcdf70d"}
Jul 28 18:31:43.832959 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.832945    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg" event={"ID":"db31e462-d99b-4176-8706-575feb13424c","Type":"ContainerDied","Data":"9195ec7b83bdb913d77749443cc510fc72a5c74f16032e63565662db4d8acd8d"}
Jul 28 18:31:43.834508 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.834484    2564 generic.go:356] "Generic (PLEG): container finished" podID="b8becfd8-66a0-4286-9e18-a37a8a569533" containerID="28fac9f43fbe22b7681771663e787c6f7e768d00d78b8b0daf1d0cf5204f921d" exitCode=0
Jul 28 18:31:43.834628 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.834509    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq" event={"ID":"b8becfd8-66a0-4286-9e18-a37a8a569533","Type":"ContainerDied","Data":"28fac9f43fbe22b7681771663e787c6f7e768d00d78b8b0daf1d0cf5204f921d"}
Jul 28 18:31:43.834628 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.834532    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"
Jul 28 18:31:43.834628 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.834552    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq" event={"ID":"b8becfd8-66a0-4286-9e18-a37a8a569533","Type":"ContainerDied","Data":"cb96b78a19edc57c21b6ac903f372963369120261d456938c62256d748605966"}
Jul 28 18:31:43.840505 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.840488    2564 scope.go:117] "RemoveContainer" containerID="3b906f08cdfc6fc824a2a079847b78e8897216184a2a24d83aa9985764dcce3a"
Jul 28 18:31:43.840771 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:31:43.840750    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b906f08cdfc6fc824a2a079847b78e8897216184a2a24d83aa9985764dcce3a\": container with ID starting with 3b906f08cdfc6fc824a2a079847b78e8897216184a2a24d83aa9985764dcce3a not found: ID does not exist" containerID="3b906f08cdfc6fc824a2a079847b78e8897216184a2a24d83aa9985764dcce3a"
Jul 28 18:31:43.840873 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.840779    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b906f08cdfc6fc824a2a079847b78e8897216184a2a24d83aa9985764dcce3a"} err="failed to get container status \"3b906f08cdfc6fc824a2a079847b78e8897216184a2a24d83aa9985764dcce3a\": rpc error: code = NotFound desc = could not find container \"3b906f08cdfc6fc824a2a079847b78e8897216184a2a24d83aa9985764dcce3a\": container with ID starting with 3b906f08cdfc6fc824a2a079847b78e8897216184a2a24d83aa9985764dcce3a not found: ID does not exist"
Jul 28 18:31:43.840873 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.840796    2564 scope.go:117] "RemoveContainer" containerID="e99469fe1793ed06180828b97d72c76fa263dbfc0f573bdde93d82a6ebcdf70d"
Jul 28 18:31:43.843316 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.843290    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx7n6\" (UniqueName: \"kubernetes.io/projected/60577033-6c68-48b7-afe2-fbfda687a9fd-kube-api-access-rx7n6\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:31:43.843389 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.843323    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db31e462-d99b-4176-8706-575feb13424c-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:31:43.843389 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.843338    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfxlz\" (UniqueName: \"kubernetes.io/projected/db31e462-d99b-4176-8706-575feb13424c-kube-api-access-cfxlz\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:31:43.843389 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.843348    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60577033-6c68-48b7-afe2-fbfda687a9fd-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:31:43.843389 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.843356    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db31e462-d99b-4176-8706-575feb13424c-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:31:43.850201 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.850179    2564 scope.go:117] "RemoveContainer" containerID="e99469fe1793ed06180828b97d72c76fa263dbfc0f573bdde93d82a6ebcdf70d"
Jul 28 18:31:43.850423 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:31:43.850406    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99469fe1793ed06180828b97d72c76fa263dbfc0f573bdde93d82a6ebcdf70d\": container with ID starting with e99469fe1793ed06180828b97d72c76fa263dbfc0f573bdde93d82a6ebcdf70d not found: ID does not exist" containerID="e99469fe1793ed06180828b97d72c76fa263dbfc0f573bdde93d82a6ebcdf70d"
Jul 28 18:31:43.850496 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.850430    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99469fe1793ed06180828b97d72c76fa263dbfc0f573bdde93d82a6ebcdf70d"} err="failed to get container status \"e99469fe1793ed06180828b97d72c76fa263dbfc0f573bdde93d82a6ebcdf70d\": rpc error: code = NotFound desc = could not find container \"e99469fe1793ed06180828b97d72c76fa263dbfc0f573bdde93d82a6ebcdf70d\": container with ID starting with e99469fe1793ed06180828b97d72c76fa263dbfc0f573bdde93d82a6ebcdf70d not found: ID does not exist"
Jul 28 18:31:43.850496 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.850447    2564 scope.go:117] "RemoveContainer" containerID="28fac9f43fbe22b7681771663e787c6f7e768d00d78b8b0daf1d0cf5204f921d"
Jul 28 18:31:43.858045 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.858015    2564 scope.go:117] "RemoveContainer" containerID="28fac9f43fbe22b7681771663e787c6f7e768d00d78b8b0daf1d0cf5204f921d"
Jul 28 18:31:43.858300 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:31:43.858283    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fac9f43fbe22b7681771663e787c6f7e768d00d78b8b0daf1d0cf5204f921d\": container with ID starting with 28fac9f43fbe22b7681771663e787c6f7e768d00d78b8b0daf1d0cf5204f921d not found: ID does not exist" containerID="28fac9f43fbe22b7681771663e787c6f7e768d00d78b8b0daf1d0cf5204f921d"
Jul 28 18:31:43.858361 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.858306    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fac9f43fbe22b7681771663e787c6f7e768d00d78b8b0daf1d0cf5204f921d"} err="failed to get container status \"28fac9f43fbe22b7681771663e787c6f7e768d00d78b8b0daf1d0cf5204f921d\": rpc error: code = NotFound desc = could not find container \"28fac9f43fbe22b7681771663e787c6f7e768d00d78b8b0daf1d0cf5204f921d\": container with ID starting with 28fac9f43fbe22b7681771663e787c6f7e768d00d78b8b0daf1d0cf5204f921d not found: ID does not exist"
Jul 28 18:31:43.858361 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.858334    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"]
Jul 28 18:31:43.863783 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.863763    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-12084f6d-67d954b44f-spzwg"]
Jul 28 18:31:43.873846 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.873827    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"]
Jul 28 18:31:43.877774 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.877755    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-12084f6d-6b54559cb-wqlrz"]
Jul 28 18:31:43.890623 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.890599    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"]
Jul 28 18:31:43.894476 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:43.894457    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-12084f6d-658dcfb95d-z64zq"]
Jul 28 18:31:45.216968 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:45.216937    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60577033-6c68-48b7-afe2-fbfda687a9fd" path="/var/lib/kubelet/pods/60577033-6c68-48b7-afe2-fbfda687a9fd/volumes"
Jul 28 18:31:45.217339 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:45.217299    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8becfd8-66a0-4286-9e18-a37a8a569533" path="/var/lib/kubelet/pods/b8becfd8-66a0-4286-9e18-a37a8a569533/volumes"
Jul 28 18:31:45.217632 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:31:45.217619    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db31e462-d99b-4176-8706-575feb13424c" path="/var/lib/kubelet/pods/db31e462-d99b-4176-8706-575feb13424c/volumes"
Jul 28 18:33:29.938955 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:29.938923    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"]
Jul 28 18:33:29.939395 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:29.939244    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8becfd8-66a0-4286-9e18-a37a8a569533" containerName="payload-processing"
Jul 28 18:33:29.939395 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:29.939256    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8becfd8-66a0-4286-9e18-a37a8a569533" containerName="payload-processing"
Jul 28 18:33:29.939395 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:29.939277    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db31e462-d99b-4176-8706-575feb13424c" containerName="maas-api"
Jul 28 18:33:29.939395 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:29.939289    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="db31e462-d99b-4176-8706-575feb13424c" containerName="maas-api"
Jul 28 18:33:29.939395 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:29.939304    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60577033-6c68-48b7-afe2-fbfda687a9fd" containerName="payload-pre-processing"
Jul 28 18:33:29.939395 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:29.939310    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="60577033-6c68-48b7-afe2-fbfda687a9fd" containerName="payload-pre-processing"
Jul 28 18:33:29.939395 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:29.939360    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="60577033-6c68-48b7-afe2-fbfda687a9fd" containerName="payload-pre-processing"
Jul 28 18:33:29.939395 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:29.939373    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="db31e462-d99b-4176-8706-575feb13424c" containerName="maas-api"
Jul 28 18:33:29.939395 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:29.939378    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8becfd8-66a0-4286-9e18-a37a8a569533" containerName="payload-processing"
Jul 28 18:33:29.942347 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:29.942326    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:33:29.945064 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:29.945041    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 18:33:29.946280 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:29.946254    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fec3f09a-serving-cert\""
Jul 28 18:33:29.946410 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:29.946263    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gffgz\""
Jul 28 18:33:29.946410 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:29.946265    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 18:33:29.954243 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:29.954220    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"]
Jul 28 18:33:30.025329 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.025287    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7397e292-6db3-4298-bfdd-ff025e32ee08-openshift-service-ca\") pod \"maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv\" (UID: \"7397e292-6db3-4298-bfdd-ff025e32ee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:33:30.025513 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.025358    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2vh4\" (UniqueName: \"kubernetes.io/projected/7397e292-6db3-4298-bfdd-ff025e32ee08-kube-api-access-m2vh4\") pod \"maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv\" (UID: \"7397e292-6db3-4298-bfdd-ff025e32ee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:33:30.025513 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.025385    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7397e292-6db3-4298-bfdd-ff025e32ee08-maas-api-tls\") pod \"maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv\" (UID: \"7397e292-6db3-4298-bfdd-ff025e32ee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:33:30.126021 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.125987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2vh4\" (UniqueName: \"kubernetes.io/projected/7397e292-6db3-4298-bfdd-ff025e32ee08-kube-api-access-m2vh4\") pod \"maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv\" (UID: \"7397e292-6db3-4298-bfdd-ff025e32ee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:33:30.126199 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.126031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7397e292-6db3-4298-bfdd-ff025e32ee08-maas-api-tls\") pod \"maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv\" (UID: \"7397e292-6db3-4298-bfdd-ff025e32ee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:33:30.126199 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.126064    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7397e292-6db3-4298-bfdd-ff025e32ee08-openshift-service-ca\") pod \"maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv\" (UID: \"7397e292-6db3-4298-bfdd-ff025e32ee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:33:30.126280 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:33:30.126199    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-serving-cert: secret "maas-api-e2e-mt-fec3f09a-serving-cert" not found
Jul 28 18:33:30.126322 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:33:30.126294    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7397e292-6db3-4298-bfdd-ff025e32ee08-maas-api-tls podName:7397e292-6db3-4298-bfdd-ff025e32ee08 nodeName:}" failed. No retries permitted until 2026-07-28 18:33:30.626273728 +0000 UTC m=+1945.941775822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7397e292-6db3-4298-bfdd-ff025e32ee08-maas-api-tls") pod "maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv" (UID: "7397e292-6db3-4298-bfdd-ff025e32ee08") : secret "maas-api-e2e-mt-fec3f09a-serving-cert" not found
Jul 28 18:33:30.126781 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.126763    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7397e292-6db3-4298-bfdd-ff025e32ee08-openshift-service-ca\") pod \"maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv\" (UID: \"7397e292-6db3-4298-bfdd-ff025e32ee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:33:30.134966 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.134937    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2vh4\" (UniqueName: \"kubernetes.io/projected/7397e292-6db3-4298-bfdd-ff025e32ee08-kube-api-access-m2vh4\") pod \"maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv\" (UID: \"7397e292-6db3-4298-bfdd-ff025e32ee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:33:30.612586 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.612554    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"]
Jul 28 18:33:30.615815 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.615788    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"
Jul 28 18:33:30.618490 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.618468    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fec3f09a\""
Jul 28 18:33:30.618490 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.618479    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fec3f09a-dockercfg-dr624\""
Jul 28 18:33:30.625235 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.625212    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"]
Jul 28 18:33:30.629691 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.629667    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb70a42-8539-414a-ab83-d9aad0eb4003-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz\" (UID: \"0cb70a42-8539-414a-ab83-d9aad0eb4003\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"
Jul 28 18:33:30.629819 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.629708    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qb8f\" (UniqueName: \"kubernetes.io/projected/0cb70a42-8539-414a-ab83-d9aad0eb4003-kube-api-access-5qb8f\") pod \"payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz\" (UID: \"0cb70a42-8539-414a-ab83-d9aad0eb4003\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"
Jul 28 18:33:30.629892 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.629844    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7397e292-6db3-4298-bfdd-ff025e32ee08-maas-api-tls\") pod \"maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv\" (UID: \"7397e292-6db3-4298-bfdd-ff025e32ee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:33:30.629983 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:33:30.629967    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-serving-cert: secret "maas-api-e2e-mt-fec3f09a-serving-cert" not found
Jul 28 18:33:30.630067 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:33:30.630056    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7397e292-6db3-4298-bfdd-ff025e32ee08-maas-api-tls podName:7397e292-6db3-4298-bfdd-ff025e32ee08 nodeName:}" failed. No retries permitted until 2026-07-28 18:33:31.630038464 +0000 UTC m=+1946.945540557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7397e292-6db3-4298-bfdd-ff025e32ee08-maas-api-tls") pod "maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv" (UID: "7397e292-6db3-4298-bfdd-ff025e32ee08") : secret "maas-api-e2e-mt-fec3f09a-serving-cert" not found
Jul 28 18:33:30.648176 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.648150    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"]
Jul 28 18:33:30.651390 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.651374    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"
Jul 28 18:33:30.659624 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.659599    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"]
Jul 28 18:33:30.730570 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.730524    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdmr\" (UniqueName: \"kubernetes.io/projected/6d051f7b-c58a-47f9-a98b-8ebf2b7039a0-kube-api-access-4jdmr\") pod \"payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k\" (UID: \"6d051f7b-c58a-47f9-a98b-8ebf2b7039a0\") " pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"
Jul 28 18:33:30.730751 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.730600    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qb8f\" (UniqueName: \"kubernetes.io/projected/0cb70a42-8539-414a-ab83-d9aad0eb4003-kube-api-access-5qb8f\") pod \"payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz\" (UID: \"0cb70a42-8539-414a-ab83-d9aad0eb4003\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"
Jul 28 18:33:30.730751 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.730640    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d051f7b-c58a-47f9-a98b-8ebf2b7039a0-plugins-config-volume\") pod \"payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k\" (UID: \"6d051f7b-c58a-47f9-a98b-8ebf2b7039a0\") " pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"
Jul 28 18:33:30.730858 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.730789    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb70a42-8539-414a-ab83-d9aad0eb4003-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz\" (UID: \"0cb70a42-8539-414a-ab83-d9aad0eb4003\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"
Jul 28 18:33:30.731453 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.731436    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb70a42-8539-414a-ab83-d9aad0eb4003-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz\" (UID: \"0cb70a42-8539-414a-ab83-d9aad0eb4003\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"
Jul 28 18:33:30.738941 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.738910    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qb8f\" (UniqueName: \"kubernetes.io/projected/0cb70a42-8539-414a-ab83-d9aad0eb4003-kube-api-access-5qb8f\") pod \"payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz\" (UID: \"0cb70a42-8539-414a-ab83-d9aad0eb4003\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"
Jul 28 18:33:30.831666 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.831630    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdmr\" (UniqueName: \"kubernetes.io/projected/6d051f7b-c58a-47f9-a98b-8ebf2b7039a0-kube-api-access-4jdmr\") pod \"payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k\" (UID: \"6d051f7b-c58a-47f9-a98b-8ebf2b7039a0\") " pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"
Jul 28 18:33:30.831855 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.831679    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d051f7b-c58a-47f9-a98b-8ebf2b7039a0-plugins-config-volume\") pod \"payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k\" (UID: \"6d051f7b-c58a-47f9-a98b-8ebf2b7039a0\") " pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"
Jul 28 18:33:30.832396 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.832377    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d051f7b-c58a-47f9-a98b-8ebf2b7039a0-plugins-config-volume\") pod \"payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k\" (UID: \"6d051f7b-c58a-47f9-a98b-8ebf2b7039a0\") " pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"
Jul 28 18:33:30.842506 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.842483    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdmr\" (UniqueName: \"kubernetes.io/projected/6d051f7b-c58a-47f9-a98b-8ebf2b7039a0-kube-api-access-4jdmr\") pod \"payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k\" (UID: \"6d051f7b-c58a-47f9-a98b-8ebf2b7039a0\") " pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"
Jul 28 18:33:30.927332 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.927240    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"
Jul 28 18:33:30.961865 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:30.961799    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"
Jul 28 18:33:31.072899 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:31.072863    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"]
Jul 28 18:33:31.074927 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:33:31.074898    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb70a42_8539_414a_ab83_d9aad0eb4003.slice/crio-b564929e59443215bf58705135eda9d13db9ee3dbe489ac74061a97439cb49ae WatchSource:0}: Error finding container b564929e59443215bf58705135eda9d13db9ee3dbe489ac74061a97439cb49ae: Status 404 returned error can't find the container with id b564929e59443215bf58705135eda9d13db9ee3dbe489ac74061a97439cb49ae
Jul 28 18:33:31.108079 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:31.108053    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"]
Jul 28 18:33:31.110390 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:33:31.110363    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d051f7b_c58a_47f9_a98b_8ebf2b7039a0.slice/crio-e2526bf6797528f784a2edd6b76ec493034eb37b626c1a066e89dee6faf9e96e WatchSource:0}: Error finding container e2526bf6797528f784a2edd6b76ec493034eb37b626c1a066e89dee6faf9e96e: Status 404 returned error can't find the container with id e2526bf6797528f784a2edd6b76ec493034eb37b626c1a066e89dee6faf9e96e
Jul 28 18:33:31.302764 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:31.302725    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz" event={"ID":"0cb70a42-8539-414a-ab83-d9aad0eb4003","Type":"ContainerStarted","Data":"d0e5468ae00d05b67fcfa567bbe5b65277ce00f11a9e40f329c371f71db6d266"}
Jul 28 18:33:31.302987 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:31.302801    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz" event={"ID":"0cb70a42-8539-414a-ab83-d9aad0eb4003","Type":"ContainerStarted","Data":"b564929e59443215bf58705135eda9d13db9ee3dbe489ac74061a97439cb49ae"}
Jul 28 18:33:31.302987 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:31.302929    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"
Jul 28 18:33:31.304351 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:31.304328    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k" event={"ID":"6d051f7b-c58a-47f9-a98b-8ebf2b7039a0","Type":"ContainerStarted","Data":"6fc06a9e29031b65f1babaee0e296c310bc1aebc51c58a672337ca565ac23755"}
Jul 28 18:33:31.304472 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:31.304373    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k" event={"ID":"6d051f7b-c58a-47f9-a98b-8ebf2b7039a0","Type":"ContainerStarted","Data":"e2526bf6797528f784a2edd6b76ec493034eb37b626c1a066e89dee6faf9e96e"}
Jul 28 18:33:31.304472 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:31.304405    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"
Jul 28 18:33:31.321364 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:31.321316    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz" podStartSLOduration=1.321302661 podStartE2EDuration="1.321302661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:33:31.319238048 +0000 UTC m=+1946.634740185" watchObservedRunningTime="2026-07-28 18:33:31.321302661 +0000 UTC m=+1946.636804774"
Jul 28 18:33:31.338831 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:31.335685    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k" podStartSLOduration=1.335662442 podStartE2EDuration="1.335662442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:33:31.334457667 +0000 UTC m=+1946.649959815" watchObservedRunningTime="2026-07-28 18:33:31.335662442 +0000 UTC m=+1946.651164555"
Jul 28 18:33:31.638329 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:31.638235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7397e292-6db3-4298-bfdd-ff025e32ee08-maas-api-tls\") pod \"maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv\" (UID: \"7397e292-6db3-4298-bfdd-ff025e32ee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:33:31.640665 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:31.640645    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7397e292-6db3-4298-bfdd-ff025e32ee08-maas-api-tls\") pod \"maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv\" (UID: \"7397e292-6db3-4298-bfdd-ff025e32ee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:33:31.753523 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:31.753489    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:33:31.884962 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:31.884927    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"]
Jul 28 18:33:31.887690 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:33:31.887665    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7397e292_6db3_4298_bfdd_ff025e32ee08.slice/crio-a7c2b0c94954cf1e0c885356c5a630f2e4b0cca7b238483cabba22ba187bb830 WatchSource:0}: Error finding container a7c2b0c94954cf1e0c885356c5a630f2e4b0cca7b238483cabba22ba187bb830: Status 404 returned error can't find the container with id a7c2b0c94954cf1e0c885356c5a630f2e4b0cca7b238483cabba22ba187bb830
Jul 28 18:33:32.310326 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:32.310291    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv" event={"ID":"7397e292-6db3-4298-bfdd-ff025e32ee08","Type":"ContainerStarted","Data":"a7c2b0c94954cf1e0c885356c5a630f2e4b0cca7b238483cabba22ba187bb830"}
Jul 28 18:33:33.316059 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:33.316025    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv" event={"ID":"7397e292-6db3-4298-bfdd-ff025e32ee08","Type":"ContainerStarted","Data":"a157f15f652ce5ccffd6ba75def9f6db4aaeb819cb5331a1e123589a4b15c6d7"}
Jul 28 18:33:33.316457 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:33.316104    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:33:33.335381 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:33.335333    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv" podStartSLOduration=3.854697035 podStartE2EDuration="4.335318038s" totalImagesPullingTime="480.621003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:33:33.333663682 +0000 UTC m=+1948.649165807" watchObservedRunningTime="2026-07-28 18:33:33.335318038 +0000 UTC m=+1948.650820151"
Jul 28 18:33:39.326395 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:39.326363    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:33:42.312713 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:42.312683    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"
Jul 28 18:33:42.313216 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:33:42.312847    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"
Jul 28 18:34:12.363571 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.363489    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"]
Jul 28 18:34:12.364113 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.363725    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv" podUID="7397e292-6db3-4298-bfdd-ff025e32ee08" containerName="maas-api" containerID="cri-o://a157f15f652ce5ccffd6ba75def9f6db4aaeb819cb5331a1e123589a4b15c6d7" gracePeriod=30
Jul 28 18:34:12.406665 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.406632    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"]
Jul 28 18:34:12.406941 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.406911    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k" podUID="6d051f7b-c58a-47f9-a98b-8ebf2b7039a0" containerName="payload-processing" containerID="cri-o://6fc06a9e29031b65f1babaee0e296c310bc1aebc51c58a672337ca565ac23755" gracePeriod=30
Jul 28 18:34:12.412390 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.412361    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"]
Jul 28 18:34:12.412793 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.412746    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz" podUID="0cb70a42-8539-414a-ab83-d9aad0eb4003" containerName="payload-pre-processing" containerID="cri-o://d0e5468ae00d05b67fcfa567bbe5b65277ce00f11a9e40f329c371f71db6d266" gracePeriod=30
Jul 28 18:34:12.485322 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.485293    2564 generic.go:356] "Generic (PLEG): container finished" podID="7397e292-6db3-4298-bfdd-ff025e32ee08" containerID="a157f15f652ce5ccffd6ba75def9f6db4aaeb819cb5331a1e123589a4b15c6d7" exitCode=0
Jul 28 18:34:12.485437 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.485354    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv" event={"ID":"7397e292-6db3-4298-bfdd-ff025e32ee08","Type":"ContainerDied","Data":"a157f15f652ce5ccffd6ba75def9f6db4aaeb819cb5331a1e123589a4b15c6d7"}
Jul 28 18:34:12.682762 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.682738    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:34:12.691609 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.691586    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"
Jul 28 18:34:12.711198 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.711170    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"
Jul 28 18:34:12.778716 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.778684    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7397e292-6db3-4298-bfdd-ff025e32ee08-openshift-service-ca\") pod \"7397e292-6db3-4298-bfdd-ff025e32ee08\" (UID: \"7397e292-6db3-4298-bfdd-ff025e32ee08\") "
Jul 28 18:34:12.778716 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.778728    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d051f7b-c58a-47f9-a98b-8ebf2b7039a0-plugins-config-volume\") pod \"6d051f7b-c58a-47f9-a98b-8ebf2b7039a0\" (UID: \"6d051f7b-c58a-47f9-a98b-8ebf2b7039a0\") "
Jul 28 18:34:12.779007 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.778762    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7397e292-6db3-4298-bfdd-ff025e32ee08-maas-api-tls\") pod \"7397e292-6db3-4298-bfdd-ff025e32ee08\" (UID: \"7397e292-6db3-4298-bfdd-ff025e32ee08\") "
Jul 28 18:34:12.779007 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.778803    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jdmr\" (UniqueName: \"kubernetes.io/projected/6d051f7b-c58a-47f9-a98b-8ebf2b7039a0-kube-api-access-4jdmr\") pod \"6d051f7b-c58a-47f9-a98b-8ebf2b7039a0\" (UID: \"6d051f7b-c58a-47f9-a98b-8ebf2b7039a0\") "
Jul 28 18:34:12.779007 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.778869    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2vh4\" (UniqueName: \"kubernetes.io/projected/7397e292-6db3-4298-bfdd-ff025e32ee08-kube-api-access-m2vh4\") pod \"7397e292-6db3-4298-bfdd-ff025e32ee08\" (UID: \"7397e292-6db3-4298-bfdd-ff025e32ee08\") "
Jul 28 18:34:12.779227 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.779127    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7397e292-6db3-4298-bfdd-ff025e32ee08-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7397e292-6db3-4298-bfdd-ff025e32ee08" (UID: "7397e292-6db3-4298-bfdd-ff025e32ee08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:34:12.779227 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.779149    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d051f7b-c58a-47f9-a98b-8ebf2b7039a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d051f7b-c58a-47f9-a98b-8ebf2b7039a0" (UID: "6d051f7b-c58a-47f9-a98b-8ebf2b7039a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:34:12.780971 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.780936    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d051f7b-c58a-47f9-a98b-8ebf2b7039a0-kube-api-access-4jdmr" (OuterVolumeSpecName: "kube-api-access-4jdmr") pod "6d051f7b-c58a-47f9-a98b-8ebf2b7039a0" (UID: "6d051f7b-c58a-47f9-a98b-8ebf2b7039a0"). InnerVolumeSpecName "kube-api-access-4jdmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:34:12.780971 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.780948    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7397e292-6db3-4298-bfdd-ff025e32ee08-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7397e292-6db3-4298-bfdd-ff025e32ee08" (UID: "7397e292-6db3-4298-bfdd-ff025e32ee08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:34:12.781139 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.781116    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7397e292-6db3-4298-bfdd-ff025e32ee08-kube-api-access-m2vh4" (OuterVolumeSpecName: "kube-api-access-m2vh4") pod "7397e292-6db3-4298-bfdd-ff025e32ee08" (UID: "7397e292-6db3-4298-bfdd-ff025e32ee08"). InnerVolumeSpecName "kube-api-access-m2vh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:34:12.880114 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.880080    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qb8f\" (UniqueName: \"kubernetes.io/projected/0cb70a42-8539-414a-ab83-d9aad0eb4003-kube-api-access-5qb8f\") pod \"0cb70a42-8539-414a-ab83-d9aad0eb4003\" (UID: \"0cb70a42-8539-414a-ab83-d9aad0eb4003\") "
Jul 28 18:34:12.880282 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.880183    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb70a42-8539-414a-ab83-d9aad0eb4003-plugins-config-volume\") pod \"0cb70a42-8539-414a-ab83-d9aad0eb4003\" (UID: \"0cb70a42-8539-414a-ab83-d9aad0eb4003\") "
Jul 28 18:34:12.880351 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.880289    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jdmr\" (UniqueName: \"kubernetes.io/projected/6d051f7b-c58a-47f9-a98b-8ebf2b7039a0-kube-api-access-4jdmr\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:34:12.880351 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.880301    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2vh4\" (UniqueName: \"kubernetes.io/projected/7397e292-6db3-4298-bfdd-ff025e32ee08-kube-api-access-m2vh4\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:34:12.880351 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.880310    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7397e292-6db3-4298-bfdd-ff025e32ee08-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:34:12.880351 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.880318    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d051f7b-c58a-47f9-a98b-8ebf2b7039a0-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:34:12.880351 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.880328    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7397e292-6db3-4298-bfdd-ff025e32ee08-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:34:12.880630 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.880599    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb70a42-8539-414a-ab83-d9aad0eb4003-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0cb70a42-8539-414a-ab83-d9aad0eb4003" (UID: "0cb70a42-8539-414a-ab83-d9aad0eb4003"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:34:12.882038 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.882018    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb70a42-8539-414a-ab83-d9aad0eb4003-kube-api-access-5qb8f" (OuterVolumeSpecName: "kube-api-access-5qb8f") pod "0cb70a42-8539-414a-ab83-d9aad0eb4003" (UID: "0cb70a42-8539-414a-ab83-d9aad0eb4003"). InnerVolumeSpecName "kube-api-access-5qb8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:34:12.981732 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.981645    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb70a42-8539-414a-ab83-d9aad0eb4003-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:34:12.981732 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:12.981676    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qb8f\" (UniqueName: \"kubernetes.io/projected/0cb70a42-8539-414a-ab83-d9aad0eb4003-kube-api-access-5qb8f\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:34:13.490893 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.490856    2564 generic.go:356] "Generic (PLEG): container finished" podID="0cb70a42-8539-414a-ab83-d9aad0eb4003" containerID="d0e5468ae00d05b67fcfa567bbe5b65277ce00f11a9e40f329c371f71db6d266" exitCode=0
Jul 28 18:34:13.491370 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.490905    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"
Jul 28 18:34:13.491370 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.490932    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz" event={"ID":"0cb70a42-8539-414a-ab83-d9aad0eb4003","Type":"ContainerDied","Data":"d0e5468ae00d05b67fcfa567bbe5b65277ce00f11a9e40f329c371f71db6d266"}
Jul 28 18:34:13.491370 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.490990    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz" event={"ID":"0cb70a42-8539-414a-ab83-d9aad0eb4003","Type":"ContainerDied","Data":"b564929e59443215bf58705135eda9d13db9ee3dbe489ac74061a97439cb49ae"}
Jul 28 18:34:13.491370 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.491023    2564 scope.go:117] "RemoveContainer" containerID="d0e5468ae00d05b67fcfa567bbe5b65277ce00f11a9e40f329c371f71db6d266"
Jul 28 18:34:13.492404 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.492387    2564 generic.go:356] "Generic (PLEG): container finished" podID="6d051f7b-c58a-47f9-a98b-8ebf2b7039a0" containerID="6fc06a9e29031b65f1babaee0e296c310bc1aebc51c58a672337ca565ac23755" exitCode=0
Jul 28 18:34:13.492511 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.492417    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k" event={"ID":"6d051f7b-c58a-47f9-a98b-8ebf2b7039a0","Type":"ContainerDied","Data":"6fc06a9e29031b65f1babaee0e296c310bc1aebc51c58a672337ca565ac23755"}
Jul 28 18:34:13.492511 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.492442    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"
Jul 28 18:34:13.492511 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.492466    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k" event={"ID":"6d051f7b-c58a-47f9-a98b-8ebf2b7039a0","Type":"ContainerDied","Data":"e2526bf6797528f784a2edd6b76ec493034eb37b626c1a066e89dee6faf9e96e"}
Jul 28 18:34:13.494102 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.494079    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv" event={"ID":"7397e292-6db3-4298-bfdd-ff025e32ee08","Type":"ContainerDied","Data":"a7c2b0c94954cf1e0c885356c5a630f2e4b0cca7b238483cabba22ba187bb830"}
Jul 28 18:34:13.494212 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.494107    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"
Jul 28 18:34:13.500291 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.500275    2564 scope.go:117] "RemoveContainer" containerID="d0e5468ae00d05b67fcfa567bbe5b65277ce00f11a9e40f329c371f71db6d266"
Jul 28 18:34:13.500546 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:34:13.500521    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e5468ae00d05b67fcfa567bbe5b65277ce00f11a9e40f329c371f71db6d266\": container with ID starting with d0e5468ae00d05b67fcfa567bbe5b65277ce00f11a9e40f329c371f71db6d266 not found: ID does not exist" containerID="d0e5468ae00d05b67fcfa567bbe5b65277ce00f11a9e40f329c371f71db6d266"
Jul 28 18:34:13.500645 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.500554    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e5468ae00d05b67fcfa567bbe5b65277ce00f11a9e40f329c371f71db6d266"} err="failed to get container status \"d0e5468ae00d05b67fcfa567bbe5b65277ce00f11a9e40f329c371f71db6d266\": rpc error: code = NotFound desc = could not find container \"d0e5468ae00d05b67fcfa567bbe5b65277ce00f11a9e40f329c371f71db6d266\": container with ID starting with d0e5468ae00d05b67fcfa567bbe5b65277ce00f11a9e40f329c371f71db6d266 not found: ID does not exist"
Jul 28 18:34:13.500645 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.500574    2564 scope.go:117] "RemoveContainer" containerID="6fc06a9e29031b65f1babaee0e296c310bc1aebc51c58a672337ca565ac23755"
Jul 28 18:34:13.507887 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.507871    2564 scope.go:117] "RemoveContainer" containerID="6fc06a9e29031b65f1babaee0e296c310bc1aebc51c58a672337ca565ac23755"
Jul 28 18:34:13.508118 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:34:13.508101    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc06a9e29031b65f1babaee0e296c310bc1aebc51c58a672337ca565ac23755\": container with ID starting with 6fc06a9e29031b65f1babaee0e296c310bc1aebc51c58a672337ca565ac23755 not found: ID does not exist" containerID="6fc06a9e29031b65f1babaee0e296c310bc1aebc51c58a672337ca565ac23755"
Jul 28 18:34:13.508162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.508123    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc06a9e29031b65f1babaee0e296c310bc1aebc51c58a672337ca565ac23755"} err="failed to get container status \"6fc06a9e29031b65f1babaee0e296c310bc1aebc51c58a672337ca565ac23755\": rpc error: code = NotFound desc = could not find container \"6fc06a9e29031b65f1babaee0e296c310bc1aebc51c58a672337ca565ac23755\": container with ID starting with 6fc06a9e29031b65f1babaee0e296c310bc1aebc51c58a672337ca565ac23755 not found: ID does not exist"
Jul 28 18:34:13.508162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.508138    2564 scope.go:117] "RemoveContainer" containerID="a157f15f652ce5ccffd6ba75def9f6db4aaeb819cb5331a1e123589a4b15c6d7"
Jul 28 18:34:13.514297 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.514276    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"]
Jul 28 18:34:13.518652 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.518633    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fec3f09a-745cc58648-9qpgz"]
Jul 28 18:34:13.536408 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.536389    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"]
Jul 28 18:34:13.542446 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.542425    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fec3f09a-5b68787c9f-snsgv"]
Jul 28 18:34:13.553146 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.553118    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"]
Jul 28 18:34:13.559211 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:13.559193    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fec3f09a-59dfc659fd-phh4k"]
Jul 28 18:34:15.215661 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:15.215622    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cb70a42-8539-414a-ab83-d9aad0eb4003" path="/var/lib/kubelet/pods/0cb70a42-8539-414a-ab83-d9aad0eb4003/volumes"
Jul 28 18:34:15.216182 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:15.216128    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d051f7b-c58a-47f9-a98b-8ebf2b7039a0" path="/var/lib/kubelet/pods/6d051f7b-c58a-47f9-a98b-8ebf2b7039a0/volumes"
Jul 28 18:34:15.216541 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:15.216521    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7397e292-6db3-4298-bfdd-ff025e32ee08" path="/var/lib/kubelet/pods/7397e292-6db3-4298-bfdd-ff025e32ee08/volumes"
Jul 28 18:34:42.178980 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.178947    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"]
Jul 28 18:34:42.179442 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.179223    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7397e292-6db3-4298-bfdd-ff025e32ee08" containerName="maas-api"
Jul 28 18:34:42.179442 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.179234    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7397e292-6db3-4298-bfdd-ff025e32ee08" containerName="maas-api"
Jul 28 18:34:42.179442 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.179246    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cb70a42-8539-414a-ab83-d9aad0eb4003" containerName="payload-pre-processing"
Jul 28 18:34:42.179442 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.179251    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cb70a42-8539-414a-ab83-d9aad0eb4003" containerName="payload-pre-processing"
Jul 28 18:34:42.179442 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.179258    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d051f7b-c58a-47f9-a98b-8ebf2b7039a0" containerName="payload-processing"
Jul 28 18:34:42.179442 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.179263    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d051f7b-c58a-47f9-a98b-8ebf2b7039a0" containerName="payload-processing"
Jul 28 18:34:42.179442 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.179320    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d051f7b-c58a-47f9-a98b-8ebf2b7039a0" containerName="payload-processing"
Jul 28 18:34:42.179442 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.179329    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cb70a42-8539-414a-ab83-d9aad0eb4003" containerName="payload-pre-processing"
Jul 28 18:34:42.179442 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.179334    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7397e292-6db3-4298-bfdd-ff025e32ee08" containerName="maas-api"
Jul 28 18:34:42.182276 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.182256    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"
Jul 28 18:34:42.185513 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.185477    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4cfa56f3-serving-cert\""
Jul 28 18:34:42.185628 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.185553    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 18:34:42.186648 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.186629    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gffgz\""
Jul 28 18:34:42.186752 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.186674    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 18:34:42.192702 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.192679    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"]
Jul 28 18:34:42.213340 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.213307    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bb2688-0dd6-40e7-bbce-dba58a239202-openshift-service-ca\") pod \"maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8\" (UID: \"e4bb2688-0dd6-40e7-bbce-dba58a239202\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"
Jul 28 18:34:42.213497 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.213363    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4bb2688-0dd6-40e7-bbce-dba58a239202-maas-api-tls\") pod \"maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8\" (UID: \"e4bb2688-0dd6-40e7-bbce-dba58a239202\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"
Jul 28 18:34:42.213497 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.213391    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8sp\" (UniqueName: \"kubernetes.io/projected/e4bb2688-0dd6-40e7-bbce-dba58a239202-kube-api-access-nj8sp\") pod \"maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8\" (UID: \"e4bb2688-0dd6-40e7-bbce-dba58a239202\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"
Jul 28 18:34:42.314839 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.314775    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4bb2688-0dd6-40e7-bbce-dba58a239202-maas-api-tls\") pod \"maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8\" (UID: \"e4bb2688-0dd6-40e7-bbce-dba58a239202\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"
Jul 28 18:34:42.315019 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:34:42.314939    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-serving-cert: secret "maas-api-e2e-mt-4cfa56f3-serving-cert" not found
Jul 28 18:34:42.315019 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.314941    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8sp\" (UniqueName: \"kubernetes.io/projected/e4bb2688-0dd6-40e7-bbce-dba58a239202-kube-api-access-nj8sp\") pod \"maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8\" (UID: \"e4bb2688-0dd6-40e7-bbce-dba58a239202\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"
Jul 28 18:34:42.315019 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:34:42.315001    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4bb2688-0dd6-40e7-bbce-dba58a239202-maas-api-tls podName:e4bb2688-0dd6-40e7-bbce-dba58a239202 nodeName:}" failed. No retries permitted until 2026-07-28 18:34:42.814983627 +0000 UTC m=+2018.130485718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4bb2688-0dd6-40e7-bbce-dba58a239202-maas-api-tls") pod "maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8" (UID: "e4bb2688-0dd6-40e7-bbce-dba58a239202") : secret "maas-api-e2e-mt-4cfa56f3-serving-cert" not found
Jul 28 18:34:42.315135 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.315058    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bb2688-0dd6-40e7-bbce-dba58a239202-openshift-service-ca\") pod \"maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8\" (UID: \"e4bb2688-0dd6-40e7-bbce-dba58a239202\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"
Jul 28 18:34:42.315686 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.315668    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bb2688-0dd6-40e7-bbce-dba58a239202-openshift-service-ca\") pod \"maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8\" (UID: \"e4bb2688-0dd6-40e7-bbce-dba58a239202\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"
Jul 28 18:34:42.325910 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.325887    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8sp\" (UniqueName: \"kubernetes.io/projected/e4bb2688-0dd6-40e7-bbce-dba58a239202-kube-api-access-nj8sp\") pod \"maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8\" (UID: \"e4bb2688-0dd6-40e7-bbce-dba58a239202\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"
Jul 28 18:34:42.823451 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.823408    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4bb2688-0dd6-40e7-bbce-dba58a239202-maas-api-tls\") pod \"maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8\" (UID: \"e4bb2688-0dd6-40e7-bbce-dba58a239202\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"
Jul 28 18:34:42.825932 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.825907    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4bb2688-0dd6-40e7-bbce-dba58a239202-maas-api-tls\") pod \"maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8\" (UID: \"e4bb2688-0dd6-40e7-bbce-dba58a239202\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"
Jul 28 18:34:42.858742 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.858708    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"]
Jul 28 18:34:42.862130 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.862115    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"
Jul 28 18:34:42.864702 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.864679    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4cfa56f3\""
Jul 28 18:34:42.870696 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.870669    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"]
Jul 28 18:34:42.912251 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.912210    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"]
Jul 28 18:34:42.915580 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.915557    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"
Jul 28 18:34:42.918160 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.918139    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4cfa56f3-dockercfg-fv97r\""
Jul 28 18:34:42.924353 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.924316    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgtkx\" (UniqueName: \"kubernetes.io/projected/da964642-e241-4e1e-b427-52984764ba67-kube-api-access-zgtkx\") pod \"payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5\" (UID: \"da964642-e241-4e1e-b427-52984764ba67\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"
Jul 28 18:34:42.924486 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.924455    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da964642-e241-4e1e-b427-52984764ba67-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5\" (UID: \"da964642-e241-4e1e-b427-52984764ba67\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"
Jul 28 18:34:42.925238 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:42.925212    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"]
Jul 28 18:34:43.025705 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.025657    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgtkx\" (UniqueName: \"kubernetes.io/projected/da964642-e241-4e1e-b427-52984764ba67-kube-api-access-zgtkx\") pod \"payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5\" (UID: \"da964642-e241-4e1e-b427-52984764ba67\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"
Jul 28 18:34:43.025948 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.025715    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94449a0-1c72-412c-80c0-07436bc6ab47-plugins-config-volume\") pod \"payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5\" (UID: \"b94449a0-1c72-412c-80c0-07436bc6ab47\") " pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"
Jul 28 18:34:43.025948 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.025763    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da964642-e241-4e1e-b427-52984764ba67-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5\" (UID: \"da964642-e241-4e1e-b427-52984764ba67\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"
Jul 28 18:34:43.025948 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.025785    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv4g6\" (UniqueName: \"kubernetes.io/projected/b94449a0-1c72-412c-80c0-07436bc6ab47-kube-api-access-cv4g6\") pod \"payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5\" (UID: \"b94449a0-1c72-412c-80c0-07436bc6ab47\") " pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"
Jul 28 18:34:43.026501 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.026482    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da964642-e241-4e1e-b427-52984764ba67-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5\" (UID: \"da964642-e241-4e1e-b427-52984764ba67\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"
Jul 28 18:34:43.034150 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.034114    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgtkx\" (UniqueName: \"kubernetes.io/projected/da964642-e241-4e1e-b427-52984764ba67-kube-api-access-zgtkx\") pod \"payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5\" (UID: \"da964642-e241-4e1e-b427-52984764ba67\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"
Jul 28 18:34:43.094001 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.093912    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"
Jul 28 18:34:43.126268 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.126239    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv4g6\" (UniqueName: \"kubernetes.io/projected/b94449a0-1c72-412c-80c0-07436bc6ab47-kube-api-access-cv4g6\") pod \"payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5\" (UID: \"b94449a0-1c72-412c-80c0-07436bc6ab47\") " pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"
Jul 28 18:34:43.126424 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.126313    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94449a0-1c72-412c-80c0-07436bc6ab47-plugins-config-volume\") pod \"payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5\" (UID: \"b94449a0-1c72-412c-80c0-07436bc6ab47\") " pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"
Jul 28 18:34:43.127039 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.127017    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94449a0-1c72-412c-80c0-07436bc6ab47-plugins-config-volume\") pod \"payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5\" (UID: \"b94449a0-1c72-412c-80c0-07436bc6ab47\") " pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"
Jul 28 18:34:43.134850 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.134802    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv4g6\" (UniqueName: \"kubernetes.io/projected/b94449a0-1c72-412c-80c0-07436bc6ab47-kube-api-access-cv4g6\") pod \"payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5\" (UID: \"b94449a0-1c72-412c-80c0-07436bc6ab47\") " pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"
Jul 28 18:34:43.173225 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.173199    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"
Jul 28 18:34:43.225767 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.225736    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"
Jul 28 18:34:43.238522 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.238462    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"]
Jul 28 18:34:43.244304 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:34:43.244276    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4bb2688_0dd6_40e7_bbce_dba58a239202.slice/crio-ebb4c4bd69a6032ecaae1f504cca3dcb430119c788f07a66853c41eb2df65746 WatchSource:0}: Error finding container ebb4c4bd69a6032ecaae1f504cca3dcb430119c788f07a66853c41eb2df65746: Status 404 returned error can't find the container with id ebb4c4bd69a6032ecaae1f504cca3dcb430119c788f07a66853c41eb2df65746
Jul 28 18:34:43.245607 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.245587    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:34:43.317270 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.317240    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"]
Jul 28 18:34:43.318677 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:34:43.318649    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda964642_e241_4e1e_b427_52984764ba67.slice/crio-5199b823b598aef900574dd23b116b6120bf0c88e281455fe753cd2ae1c612c2 WatchSource:0}: Error finding container 5199b823b598aef900574dd23b116b6120bf0c88e281455fe753cd2ae1c612c2: Status 404 returned error can't find the container with id 5199b823b598aef900574dd23b116b6120bf0c88e281455fe753cd2ae1c612c2
Jul 28 18:34:43.383703 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.383678    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"]
Jul 28 18:34:43.386215 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:34:43.386052    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94449a0_1c72_412c_80c0_07436bc6ab47.slice/crio-c9a6e21e5a101460330c4c3095b2f2fe1d2b55ab78825e1d922114fd6373792d WatchSource:0}: Error finding container c9a6e21e5a101460330c4c3095b2f2fe1d2b55ab78825e1d922114fd6373792d: Status 404 returned error can't find the container with id c9a6e21e5a101460330c4c3095b2f2fe1d2b55ab78825e1d922114fd6373792d
Jul 28 18:34:43.629575 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.629490    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8" event={"ID":"e4bb2688-0dd6-40e7-bbce-dba58a239202","Type":"ContainerStarted","Data":"ebb4c4bd69a6032ecaae1f504cca3dcb430119c788f07a66853c41eb2df65746"}
Jul 28 18:34:43.631093 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.631066    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5" event={"ID":"b94449a0-1c72-412c-80c0-07436bc6ab47","Type":"ContainerStarted","Data":"69d5902d4841809484752c1ffc4f64bebc04b1d52202cb7c74c8954a2b32d218"}
Jul 28 18:34:43.631235 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.631111    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5" event={"ID":"b94449a0-1c72-412c-80c0-07436bc6ab47","Type":"ContainerStarted","Data":"c9a6e21e5a101460330c4c3095b2f2fe1d2b55ab78825e1d922114fd6373792d"}
Jul 28 18:34:43.631235 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.631174    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"
Jul 28 18:34:43.632843 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.632802    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5" event={"ID":"da964642-e241-4e1e-b427-52984764ba67","Type":"ContainerStarted","Data":"940ee9462a22466e5e10fc68a426e73d8a0c8e9cc410cb73766f65fa65689521"}
Jul 28 18:34:43.632967 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.632861    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5" event={"ID":"da964642-e241-4e1e-b427-52984764ba67","Type":"ContainerStarted","Data":"5199b823b598aef900574dd23b116b6120bf0c88e281455fe753cd2ae1c612c2"}
Jul 28 18:34:43.632967 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.632901    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"
Jul 28 18:34:43.648662 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.648614    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5" podStartSLOduration=1.648597562 podStartE2EDuration="1.648597562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:34:43.646439469 +0000 UTC m=+2018.961941611" watchObservedRunningTime="2026-07-28 18:34:43.648597562 +0000 UTC m=+2018.964099675"
Jul 28 18:34:43.661093 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:43.661038    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5" podStartSLOduration=1.6610184650000002 podStartE2EDuration="1.661018465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:34:43.660458779 +0000 UTC m=+2018.975960948" watchObservedRunningTime="2026-07-28 18:34:43.661018465 +0000 UTC m=+2018.976520578"
Jul 28 18:34:44.639159 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:44.639118    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8" event={"ID":"e4bb2688-0dd6-40e7-bbce-dba58a239202","Type":"ContainerStarted","Data":"afd50fecf50b5500d35934b276d4ef1b1cfb547375b58322063b41fe88070fb4"}
Jul 28 18:34:44.639632 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:44.639176    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"
Jul 28 18:34:44.657292 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:44.657239    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8" podStartSLOduration=2.079168268 podStartE2EDuration="2.657220571s" totalImagesPullingTime="578.052303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:34:44.655724814 +0000 UTC m=+2019.971226973" watchObservedRunningTime="2026-07-28 18:34:44.657220571 +0000 UTC m=+2019.972722684"
Jul 28 18:34:50.649146 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:50.649114    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"
Jul 28 18:34:54.640925 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:54.640894    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"
Jul 28 18:34:54.641376 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:34:54.640965    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"
Jul 28 18:36:02.128921 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.128890    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"]
Jul 28 18:36:02.129433 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.129096    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8" podUID="e4bb2688-0dd6-40e7-bbce-dba58a239202" containerName="maas-api" containerID="cri-o://afd50fecf50b5500d35934b276d4ef1b1cfb547375b58322063b41fe88070fb4" gracePeriod=30
Jul 28 18:36:02.180416 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.180382    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"]
Jul 28 18:36:02.180790 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.180595    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5" podUID="b94449a0-1c72-412c-80c0-07436bc6ab47" containerName="payload-processing" containerID="cri-o://69d5902d4841809484752c1ffc4f64bebc04b1d52202cb7c74c8954a2b32d218" gracePeriod=30
Jul 28 18:36:02.187272 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.187245    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"]
Jul 28 18:36:02.187509 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.187472    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5" podUID="da964642-e241-4e1e-b427-52984764ba67" containerName="payload-pre-processing" containerID="cri-o://940ee9462a22466e5e10fc68a426e73d8a0c8e9cc410cb73766f65fa65689521" gracePeriod=30
Jul 28 18:36:02.442114 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.442080    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"
Jul 28 18:36:02.459293 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.459263    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"
Jul 28 18:36:02.469775 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.469752    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"
Jul 28 18:36:02.571403 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.571371    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgtkx\" (UniqueName: \"kubernetes.io/projected/da964642-e241-4e1e-b427-52984764ba67-kube-api-access-zgtkx\") pod \"da964642-e241-4e1e-b427-52984764ba67\" (UID: \"da964642-e241-4e1e-b427-52984764ba67\") "
Jul 28 18:36:02.571403 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.571413    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94449a0-1c72-412c-80c0-07436bc6ab47-plugins-config-volume\") pod \"b94449a0-1c72-412c-80c0-07436bc6ab47\" (UID: \"b94449a0-1c72-412c-80c0-07436bc6ab47\") "
Jul 28 18:36:02.571657 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.571445    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da964642-e241-4e1e-b427-52984764ba67-plugins-config-volume\") pod \"da964642-e241-4e1e-b427-52984764ba67\" (UID: \"da964642-e241-4e1e-b427-52984764ba67\") "
Jul 28 18:36:02.571657 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.571468    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bb2688-0dd6-40e7-bbce-dba58a239202-openshift-service-ca\") pod \"e4bb2688-0dd6-40e7-bbce-dba58a239202\" (UID: \"e4bb2688-0dd6-40e7-bbce-dba58a239202\") "
Jul 28 18:36:02.571657 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.571486    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv4g6\" (UniqueName: \"kubernetes.io/projected/b94449a0-1c72-412c-80c0-07436bc6ab47-kube-api-access-cv4g6\") pod \"b94449a0-1c72-412c-80c0-07436bc6ab47\" (UID: \"b94449a0-1c72-412c-80c0-07436bc6ab47\") "
Jul 28 18:36:02.571657 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.571506    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj8sp\" (UniqueName: \"kubernetes.io/projected/e4bb2688-0dd6-40e7-bbce-dba58a239202-kube-api-access-nj8sp\") pod \"e4bb2688-0dd6-40e7-bbce-dba58a239202\" (UID: \"e4bb2688-0dd6-40e7-bbce-dba58a239202\") "
Jul 28 18:36:02.571657 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.571561    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4bb2688-0dd6-40e7-bbce-dba58a239202-maas-api-tls\") pod \"e4bb2688-0dd6-40e7-bbce-dba58a239202\" (UID: \"e4bb2688-0dd6-40e7-bbce-dba58a239202\") "
Jul 28 18:36:02.571958 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.571870    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94449a0-1c72-412c-80c0-07436bc6ab47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b94449a0-1c72-412c-80c0-07436bc6ab47" (UID: "b94449a0-1c72-412c-80c0-07436bc6ab47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:36:02.571958 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.571881    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4bb2688-0dd6-40e7-bbce-dba58a239202-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4bb2688-0dd6-40e7-bbce-dba58a239202" (UID: "e4bb2688-0dd6-40e7-bbce-dba58a239202"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:36:02.571958 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.571932    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da964642-e241-4e1e-b427-52984764ba67-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da964642-e241-4e1e-b427-52984764ba67" (UID: "da964642-e241-4e1e-b427-52984764ba67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:36:02.573563 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.573539    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da964642-e241-4e1e-b427-52984764ba67-kube-api-access-zgtkx" (OuterVolumeSpecName: "kube-api-access-zgtkx") pod "da964642-e241-4e1e-b427-52984764ba67" (UID: "da964642-e241-4e1e-b427-52984764ba67"). InnerVolumeSpecName "kube-api-access-zgtkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:36:02.573640 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.573589    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4bb2688-0dd6-40e7-bbce-dba58a239202-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4bb2688-0dd6-40e7-bbce-dba58a239202" (UID: "e4bb2688-0dd6-40e7-bbce-dba58a239202"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:36:02.574149 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.574125    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bb2688-0dd6-40e7-bbce-dba58a239202-kube-api-access-nj8sp" (OuterVolumeSpecName: "kube-api-access-nj8sp") pod "e4bb2688-0dd6-40e7-bbce-dba58a239202" (UID: "e4bb2688-0dd6-40e7-bbce-dba58a239202"). InnerVolumeSpecName "kube-api-access-nj8sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:36:02.574149 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.574137    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94449a0-1c72-412c-80c0-07436bc6ab47-kube-api-access-cv4g6" (OuterVolumeSpecName: "kube-api-access-cv4g6") pod "b94449a0-1c72-412c-80c0-07436bc6ab47" (UID: "b94449a0-1c72-412c-80c0-07436bc6ab47"). InnerVolumeSpecName "kube-api-access-cv4g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:36:02.672411 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.672326    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4bb2688-0dd6-40e7-bbce-dba58a239202-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:36:02.672411 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.672357    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgtkx\" (UniqueName: \"kubernetes.io/projected/da964642-e241-4e1e-b427-52984764ba67-kube-api-access-zgtkx\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:36:02.672411 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.672368    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94449a0-1c72-412c-80c0-07436bc6ab47-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:36:02.672411 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.672376    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da964642-e241-4e1e-b427-52984764ba67-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:36:02.672411 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.672384    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bb2688-0dd6-40e7-bbce-dba58a239202-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:36:02.672411 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.672394    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv4g6\" (UniqueName: \"kubernetes.io/projected/b94449a0-1c72-412c-80c0-07436bc6ab47-kube-api-access-cv4g6\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:36:02.672411 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.672401    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj8sp\" (UniqueName: \"kubernetes.io/projected/e4bb2688-0dd6-40e7-bbce-dba58a239202-kube-api-access-nj8sp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:36:02.980441 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.980348    2564 generic.go:356] "Generic (PLEG): container finished" podID="da964642-e241-4e1e-b427-52984764ba67" containerID="940ee9462a22466e5e10fc68a426e73d8a0c8e9cc410cb73766f65fa65689521" exitCode=0
Jul 28 18:36:02.980441 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.980393    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"
Jul 28 18:36:02.980441 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.980430    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5" event={"ID":"da964642-e241-4e1e-b427-52984764ba67","Type":"ContainerDied","Data":"940ee9462a22466e5e10fc68a426e73d8a0c8e9cc410cb73766f65fa65689521"}
Jul 28 18:36:02.980704 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.980478    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5" event={"ID":"da964642-e241-4e1e-b427-52984764ba67","Type":"ContainerDied","Data":"5199b823b598aef900574dd23b116b6120bf0c88e281455fe753cd2ae1c612c2"}
Jul 28 18:36:02.980704 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.980502    2564 scope.go:117] "RemoveContainer" containerID="940ee9462a22466e5e10fc68a426e73d8a0c8e9cc410cb73766f65fa65689521"
Jul 28 18:36:02.981822 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.981783    2564 generic.go:356] "Generic (PLEG): container finished" podID="e4bb2688-0dd6-40e7-bbce-dba58a239202" containerID="afd50fecf50b5500d35934b276d4ef1b1cfb547375b58322063b41fe88070fb4" exitCode=0
Jul 28 18:36:02.981923 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.981844    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"
Jul 28 18:36:02.981923 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.981844    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8" event={"ID":"e4bb2688-0dd6-40e7-bbce-dba58a239202","Type":"ContainerDied","Data":"afd50fecf50b5500d35934b276d4ef1b1cfb547375b58322063b41fe88070fb4"}
Jul 28 18:36:02.981923 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.981892    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8" event={"ID":"e4bb2688-0dd6-40e7-bbce-dba58a239202","Type":"ContainerDied","Data":"ebb4c4bd69a6032ecaae1f504cca3dcb430119c788f07a66853c41eb2df65746"}
Jul 28 18:36:02.983414 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.983391    2564 generic.go:356] "Generic (PLEG): container finished" podID="b94449a0-1c72-412c-80c0-07436bc6ab47" containerID="69d5902d4841809484752c1ffc4f64bebc04b1d52202cb7c74c8954a2b32d218" exitCode=0
Jul 28 18:36:02.983530 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.983443    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5" event={"ID":"b94449a0-1c72-412c-80c0-07436bc6ab47","Type":"ContainerDied","Data":"69d5902d4841809484752c1ffc4f64bebc04b1d52202cb7c74c8954a2b32d218"}
Jul 28 18:36:02.983530 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.983477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5" event={"ID":"b94449a0-1c72-412c-80c0-07436bc6ab47","Type":"ContainerDied","Data":"c9a6e21e5a101460330c4c3095b2f2fe1d2b55ab78825e1d922114fd6373792d"}
Jul 28 18:36:02.983530 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.983491    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"
Jul 28 18:36:02.990169 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.990125    2564 scope.go:117] "RemoveContainer" containerID="940ee9462a22466e5e10fc68a426e73d8a0c8e9cc410cb73766f65fa65689521"
Jul 28 18:36:02.990567 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:36:02.990529    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940ee9462a22466e5e10fc68a426e73d8a0c8e9cc410cb73766f65fa65689521\": container with ID starting with 940ee9462a22466e5e10fc68a426e73d8a0c8e9cc410cb73766f65fa65689521 not found: ID does not exist" containerID="940ee9462a22466e5e10fc68a426e73d8a0c8e9cc410cb73766f65fa65689521"
Jul 28 18:36:02.990675 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.990575    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940ee9462a22466e5e10fc68a426e73d8a0c8e9cc410cb73766f65fa65689521"} err="failed to get container status \"940ee9462a22466e5e10fc68a426e73d8a0c8e9cc410cb73766f65fa65689521\": rpc error: code = NotFound desc = could not find container \"940ee9462a22466e5e10fc68a426e73d8a0c8e9cc410cb73766f65fa65689521\": container with ID starting with 940ee9462a22466e5e10fc68a426e73d8a0c8e9cc410cb73766f65fa65689521 not found: ID does not exist"
Jul 28 18:36:02.990675 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.990594    2564 scope.go:117] "RemoveContainer" containerID="afd50fecf50b5500d35934b276d4ef1b1cfb547375b58322063b41fe88070fb4"
Jul 28 18:36:02.999314 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.999299    2564 scope.go:117] "RemoveContainer" containerID="afd50fecf50b5500d35934b276d4ef1b1cfb547375b58322063b41fe88070fb4"
Jul 28 18:36:02.999543 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:36:02.999520    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd50fecf50b5500d35934b276d4ef1b1cfb547375b58322063b41fe88070fb4\": container with ID starting with afd50fecf50b5500d35934b276d4ef1b1cfb547375b58322063b41fe88070fb4 not found: ID does not exist" containerID="afd50fecf50b5500d35934b276d4ef1b1cfb547375b58322063b41fe88070fb4"
Jul 28 18:36:02.999604 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.999548    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd50fecf50b5500d35934b276d4ef1b1cfb547375b58322063b41fe88070fb4"} err="failed to get container status \"afd50fecf50b5500d35934b276d4ef1b1cfb547375b58322063b41fe88070fb4\": rpc error: code = NotFound desc = could not find container \"afd50fecf50b5500d35934b276d4ef1b1cfb547375b58322063b41fe88070fb4\": container with ID starting with afd50fecf50b5500d35934b276d4ef1b1cfb547375b58322063b41fe88070fb4 not found: ID does not exist"
Jul 28 18:36:02.999604 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:02.999564    2564 scope.go:117] "RemoveContainer" containerID="69d5902d4841809484752c1ffc4f64bebc04b1d52202cb7c74c8954a2b32d218"
Jul 28 18:36:03.004942 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:03.004922    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"]
Jul 28 18:36:03.006687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:03.006659    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cfa56f3-86f764c464-j96k5"]
Jul 28 18:36:03.008693 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:03.008677    2564 scope.go:117] "RemoveContainer" containerID="69d5902d4841809484752c1ffc4f64bebc04b1d52202cb7c74c8954a2b32d218"
Jul 28 18:36:03.008967 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:36:03.008950    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d5902d4841809484752c1ffc4f64bebc04b1d52202cb7c74c8954a2b32d218\": container with ID starting with 69d5902d4841809484752c1ffc4f64bebc04b1d52202cb7c74c8954a2b32d218 not found: ID does not exist" containerID="69d5902d4841809484752c1ffc4f64bebc04b1d52202cb7c74c8954a2b32d218"
Jul 28 18:36:03.009022 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:03.008976    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d5902d4841809484752c1ffc4f64bebc04b1d52202cb7c74c8954a2b32d218"} err="failed to get container status \"69d5902d4841809484752c1ffc4f64bebc04b1d52202cb7c74c8954a2b32d218\": rpc error: code = NotFound desc = could not find container \"69d5902d4841809484752c1ffc4f64bebc04b1d52202cb7c74c8954a2b32d218\": container with ID starting with 69d5902d4841809484752c1ffc4f64bebc04b1d52202cb7c74c8954a2b32d218 not found: ID does not exist"
Jul 28 18:36:03.016227 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:03.016208    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"]
Jul 28 18:36:03.020025 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:03.020006    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cfa56f3-64d694f99d-kbrl5"]
Jul 28 18:36:03.029636 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:03.029613    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"]
Jul 28 18:36:03.033624 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:03.033604    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cfa56f3-c9bf4855b-5r9s8"]
Jul 28 18:36:03.216705 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:03.216677    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b94449a0-1c72-412c-80c0-07436bc6ab47" path="/var/lib/kubelet/pods/b94449a0-1c72-412c-80c0-07436bc6ab47/volumes"
Jul 28 18:36:03.217078 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:03.217028    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da964642-e241-4e1e-b427-52984764ba67" path="/var/lib/kubelet/pods/da964642-e241-4e1e-b427-52984764ba67/volumes"
Jul 28 18:36:03.217358 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:03.217346    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4bb2688-0dd6-40e7-bbce-dba58a239202" path="/var/lib/kubelet/pods/e4bb2688-0dd6-40e7-bbce-dba58a239202/volumes"
Jul 28 18:36:05.660518 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:36:05.660492    2564 scope.go:117] "RemoveContainer" containerID="a105724971a55dc7a5c4efe1ca1385a1e55a45372a4936a377428a17a3e6d3a6"
Jul 28 18:37:52.166471 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.166439    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"]
Jul 28 18:37:52.166896 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.166733    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da964642-e241-4e1e-b427-52984764ba67" containerName="payload-pre-processing"
Jul 28 18:37:52.166896 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.166751    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="da964642-e241-4e1e-b427-52984764ba67" containerName="payload-pre-processing"
Jul 28 18:37:52.166896 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.166772    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b94449a0-1c72-412c-80c0-07436bc6ab47" containerName="payload-processing"
Jul 28 18:37:52.166896 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.166780    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b94449a0-1c72-412c-80c0-07436bc6ab47" containerName="payload-processing"
Jul 28 18:37:52.167054 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.166900    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4bb2688-0dd6-40e7-bbce-dba58a239202" containerName="maas-api"
Jul 28 18:37:52.167054 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.166913    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4bb2688-0dd6-40e7-bbce-dba58a239202" containerName="maas-api"
Jul 28 18:37:52.167054 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.166991    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="da964642-e241-4e1e-b427-52984764ba67" containerName="payload-pre-processing"
Jul 28 18:37:52.167054 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.167000    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b94449a0-1c72-412c-80c0-07436bc6ab47" containerName="payload-processing"
Jul 28 18:37:52.167054 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.167007    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4bb2688-0dd6-40e7-bbce-dba58a239202" containerName="maas-api"
Jul 28 18:37:52.169392 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.169371    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"
Jul 28 18:37:52.172290 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.172271    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 18:37:52.172397 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.172295    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-61d29d-serving-cert\""
Jul 28 18:37:52.173460 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.173439    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gffgz\""
Jul 28 18:37:52.173460 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.173450    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 18:37:52.180567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.180544    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"]
Jul 28 18:37:52.272912 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.272881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsh7q\" (UniqueName: \"kubernetes.io/projected/fa409841-fea7-4f21-86bf-d5ec1a966356-kube-api-access-lsh7q\") pod \"maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp\" (UID: \"fa409841-fea7-4f21-86bf-d5ec1a966356\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"
Jul 28 18:37:52.273101 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.272933    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa409841-fea7-4f21-86bf-d5ec1a966356-openshift-service-ca\") pod \"maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp\" (UID: \"fa409841-fea7-4f21-86bf-d5ec1a966356\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"
Jul 28 18:37:52.273101 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.272965    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa409841-fea7-4f21-86bf-d5ec1a966356-maas-api-tls\") pod \"maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp\" (UID: \"fa409841-fea7-4f21-86bf-d5ec1a966356\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"
Jul 28 18:37:52.373471 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.373447    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsh7q\" (UniqueName: \"kubernetes.io/projected/fa409841-fea7-4f21-86bf-d5ec1a966356-kube-api-access-lsh7q\") pod \"maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp\" (UID: \"fa409841-fea7-4f21-86bf-d5ec1a966356\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"
Jul 28 18:37:52.373645 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.373482    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa409841-fea7-4f21-86bf-d5ec1a966356-openshift-service-ca\") pod \"maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp\" (UID: \"fa409841-fea7-4f21-86bf-d5ec1a966356\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"
Jul 28 18:37:52.373645 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.373510    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa409841-fea7-4f21-86bf-d5ec1a966356-maas-api-tls\") pod \"maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp\" (UID: \"fa409841-fea7-4f21-86bf-d5ec1a966356\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"
Jul 28 18:37:52.374201 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.374172    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa409841-fea7-4f21-86bf-d5ec1a966356-openshift-service-ca\") pod \"maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp\" (UID: \"fa409841-fea7-4f21-86bf-d5ec1a966356\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"
Jul 28 18:37:52.376014 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.375990    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa409841-fea7-4f21-86bf-d5ec1a966356-maas-api-tls\") pod \"maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp\" (UID: \"fa409841-fea7-4f21-86bf-d5ec1a966356\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"
Jul 28 18:37:52.381390 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.381362    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsh7q\" (UniqueName: \"kubernetes.io/projected/fa409841-fea7-4f21-86bf-d5ec1a966356-kube-api-access-lsh7q\") pod \"maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp\" (UID: \"fa409841-fea7-4f21-86bf-d5ec1a966356\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"
Jul 28 18:37:52.480520 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.480444    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"
Jul 28 18:37:52.605913 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.605880    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"]
Jul 28 18:37:52.609630 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:37:52.609603    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa409841_fea7_4f21_86bf_d5ec1a966356.slice/crio-abd4906e682007f85b6224a2c4b3795e6226307b6e5e65007af8acdfcef0b693 WatchSource:0}: Error finding container abd4906e682007f85b6224a2c4b3795e6226307b6e5e65007af8acdfcef0b693: Status 404 returned error can't find the container with id abd4906e682007f85b6224a2c4b3795e6226307b6e5e65007af8acdfcef0b693
Jul 28 18:37:52.871387 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.871352    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"]
Jul 28 18:37:52.875563 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.875545    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"
Jul 28 18:37:52.878140 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.878119    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-61d29d-dockercfg-2h5qt\""
Jul 28 18:37:52.878256 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.878219    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-61d29d\""
Jul 28 18:37:52.885165 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.885145    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"]
Jul 28 18:37:52.906360 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.906338    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"]
Jul 28 18:37:52.914886 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.914852    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"
Jul 28 18:37:52.920065 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.920040    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"]
Jul 28 18:37:52.979298 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.979271    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7fzl\" (UniqueName: \"kubernetes.io/projected/bbb11b94-5fab-4d45-80e1-2b8329f90f99-kube-api-access-n7fzl\") pod \"payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc\" (UID: \"bbb11b94-5fab-4d45-80e1-2b8329f90f99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"
Jul 28 18:37:52.979423 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.979314    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca32e8cc-7250-494a-9303-40427eea96de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl\" (UID: \"ca32e8cc-7250-494a-9303-40427eea96de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"
Jul 28 18:37:52.979423 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.979352    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6wnb\" (UniqueName: \"kubernetes.io/projected/ca32e8cc-7250-494a-9303-40427eea96de-kube-api-access-z6wnb\") pod \"payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl\" (UID: \"ca32e8cc-7250-494a-9303-40427eea96de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"
Jul 28 18:37:52.979423 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:52.979377    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb11b94-5fab-4d45-80e1-2b8329f90f99-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc\" (UID: \"bbb11b94-5fab-4d45-80e1-2b8329f90f99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"
Jul 28 18:37:53.080087 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.080028    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca32e8cc-7250-494a-9303-40427eea96de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl\" (UID: \"ca32e8cc-7250-494a-9303-40427eea96de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"
Jul 28 18:37:53.080087 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.080074    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wnb\" (UniqueName: \"kubernetes.io/projected/ca32e8cc-7250-494a-9303-40427eea96de-kube-api-access-z6wnb\") pod \"payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl\" (UID: \"ca32e8cc-7250-494a-9303-40427eea96de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"
Jul 28 18:37:53.080223 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.080109    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb11b94-5fab-4d45-80e1-2b8329f90f99-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc\" (UID: \"bbb11b94-5fab-4d45-80e1-2b8329f90f99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"
Jul 28 18:37:53.080223 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.080196    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7fzl\" (UniqueName: \"kubernetes.io/projected/bbb11b94-5fab-4d45-80e1-2b8329f90f99-kube-api-access-n7fzl\") pod \"payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc\" (UID: \"bbb11b94-5fab-4d45-80e1-2b8329f90f99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"
Jul 28 18:37:53.080791 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.080763    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca32e8cc-7250-494a-9303-40427eea96de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl\" (UID: \"ca32e8cc-7250-494a-9303-40427eea96de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"
Jul 28 18:37:53.080896 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.080867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb11b94-5fab-4d45-80e1-2b8329f90f99-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc\" (UID: \"bbb11b94-5fab-4d45-80e1-2b8329f90f99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"
Jul 28 18:37:53.092761 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.092739    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7fzl\" (UniqueName: \"kubernetes.io/projected/bbb11b94-5fab-4d45-80e1-2b8329f90f99-kube-api-access-n7fzl\") pod \"payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc\" (UID: \"bbb11b94-5fab-4d45-80e1-2b8329f90f99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"
Jul 28 18:37:53.093328 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.093308    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wnb\" (UniqueName: \"kubernetes.io/projected/ca32e8cc-7250-494a-9303-40427eea96de-kube-api-access-z6wnb\") pod \"payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl\" (UID: \"ca32e8cc-7250-494a-9303-40427eea96de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"
Jul 28 18:37:53.187105 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.187038    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"
Jul 28 18:37:53.225839 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.225796    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"
Jul 28 18:37:53.328621 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.328584    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"]
Jul 28 18:37:53.332879 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:37:53.332838    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca32e8cc_7250_494a_9303_40427eea96de.slice/crio-d01ebfae36de36615fd0463fd0bb066e08902768e16cef66e27208f72330b137 WatchSource:0}: Error finding container d01ebfae36de36615fd0463fd0bb066e08902768e16cef66e27208f72330b137: Status 404 returned error can't find the container with id d01ebfae36de36615fd0463fd0bb066e08902768e16cef66e27208f72330b137
Jul 28 18:37:53.363985 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.363959    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"]
Jul 28 18:37:53.366730 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:37:53.366701    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb11b94_5fab_4d45_80e1_2b8329f90f99.slice/crio-dec73ad121ad9a3ed39052c9c1de30da91d357f04431386970d6274b50c83525 WatchSource:0}: Error finding container dec73ad121ad9a3ed39052c9c1de30da91d357f04431386970d6274b50c83525: Status 404 returned error can't find the container with id dec73ad121ad9a3ed39052c9c1de30da91d357f04431386970d6274b50c83525
Jul 28 18:37:53.444005 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.443929    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl" event={"ID":"ca32e8cc-7250-494a-9303-40427eea96de","Type":"ContainerStarted","Data":"efc7d4d326e822eaabcc4beadccfd69bbc8a194f8539813e6fd2e04848999ab1"}
Jul 28 18:37:53.444005 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.443985    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl" event={"ID":"ca32e8cc-7250-494a-9303-40427eea96de","Type":"ContainerStarted","Data":"d01ebfae36de36615fd0463fd0bb066e08902768e16cef66e27208f72330b137"}
Jul 28 18:37:53.444167 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.444017    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"
Jul 28 18:37:53.445627 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.445601    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc" event={"ID":"bbb11b94-5fab-4d45-80e1-2b8329f90f99","Type":"ContainerStarted","Data":"edfcf31bc2467d681de1be3da8a781c8ac23692719a9a0c112c99777fa98d13d"}
Jul 28 18:37:53.445745 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.445656    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc" event={"ID":"bbb11b94-5fab-4d45-80e1-2b8329f90f99","Type":"ContainerStarted","Data":"dec73ad121ad9a3ed39052c9c1de30da91d357f04431386970d6274b50c83525"}
Jul 28 18:37:53.445745 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.445685    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"
Jul 28 18:37:53.447050 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.447026    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp" event={"ID":"fa409841-fea7-4f21-86bf-d5ec1a966356","Type":"ContainerStarted","Data":"e01025ef4c5e2a2f43dc00087706303d79d86d8caccaeb7caf27810bd1124071"}
Jul 28 18:37:53.447138 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.447068    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp" event={"ID":"fa409841-fea7-4f21-86bf-d5ec1a966356","Type":"ContainerStarted","Data":"abd4906e682007f85b6224a2c4b3795e6226307b6e5e65007af8acdfcef0b693"}
Jul 28 18:37:53.447201 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.447163    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"
Jul 28 18:37:53.460631 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.460584    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl" podStartSLOduration=1.460569978 podStartE2EDuration="1.460569978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:37:53.459651382 +0000 UTC m=+2208.775153522" watchObservedRunningTime="2026-07-28 18:37:53.460569978 +0000 UTC m=+2208.776072091"
Jul 28 18:37:53.478482 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.478442    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp" podStartSLOduration=1.06278575 podStartE2EDuration="1.478427846s" totalImagesPullingTime="415.642096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:37:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:37:53.476879591 +0000 UTC m=+2208.792381720" watchObservedRunningTime="2026-07-28 18:37:53.478427846 +0000 UTC m=+2208.793929959"
Jul 28 18:37:53.495101 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:53.495063    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc" podStartSLOduration=1.495048175 podStartE2EDuration="1.495048175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:37:53.492703883 +0000 UTC m=+2208.808205997" watchObservedRunningTime="2026-07-28 18:37:53.495048175 +0000 UTC m=+2208.810550288"
Jul 28 18:37:59.456661 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:37:59.456630    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"
Jul 28 18:38:04.452949 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:04.452921    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"
Jul 28 18:38:04.453348 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:04.452995    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"
Jul 28 18:38:20.293873 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.293838    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"]
Jul 28 18:38:20.297479 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.297461    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:38:20.300064 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.300042    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-db5d01-serving-cert\""
Jul 28 18:38:20.303773 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.303743    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jrhv\" (UniqueName: \"kubernetes.io/projected/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-kube-api-access-9jrhv\") pod \"maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q\" (UID: \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:38:20.303929 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.303791    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-maas-api-tls\") pod \"maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q\" (UID: \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:38:20.303929 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.303914    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-openshift-service-ca\") pod \"maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q\" (UID: \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:38:20.305097 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.304990    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"]
Jul 28 18:38:20.404933 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.404903    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-openshift-service-ca\") pod \"maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q\" (UID: \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:38:20.405073 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.404955    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrhv\" (UniqueName: \"kubernetes.io/projected/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-kube-api-access-9jrhv\") pod \"maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q\" (UID: \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:38:20.405073 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.404985    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-maas-api-tls\") pod \"maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q\" (UID: \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:38:20.405148 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:38:20.405067    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-serving-cert: secret "maas-api-e2e-inf-b-db5d01-serving-cert" not found
Jul 28 18:38:20.405183 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:38:20.405148    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-maas-api-tls podName:12ee4b3d-8249-4b81-a0c5-fca06ecb38ce nodeName:}" failed. No retries permitted until 2026-07-28 18:38:20.905126855 +0000 UTC m=+2236.220628954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-maas-api-tls") pod "maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q" (UID: "12ee4b3d-8249-4b81-a0c5-fca06ecb38ce") : secret "maas-api-e2e-inf-b-db5d01-serving-cert" not found
Jul 28 18:38:20.405517 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.405500    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-openshift-service-ca\") pod \"maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q\" (UID: \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:38:20.413335 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.413307    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jrhv\" (UniqueName: \"kubernetes.io/projected/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-kube-api-access-9jrhv\") pod \"maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q\" (UID: \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:38:20.908508 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.908475    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-maas-api-tls\") pod \"maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q\" (UID: \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:38:20.908663 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:38:20.908599    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-serving-cert: secret "maas-api-e2e-inf-b-db5d01-serving-cert" not found
Jul 28 18:38:20.908663 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:38:20.908655    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-maas-api-tls podName:12ee4b3d-8249-4b81-a0c5-fca06ecb38ce nodeName:}" failed. No retries permitted until 2026-07-28 18:38:21.90863922 +0000 UTC m=+2237.224141313 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-maas-api-tls") pod "maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q" (UID: "12ee4b3d-8249-4b81-a0c5-fca06ecb38ce") : secret "maas-api-e2e-inf-b-db5d01-serving-cert" not found
Jul 28 18:38:20.979695 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.979659    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"]
Jul 28 18:38:20.983104 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.983082    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"
Jul 28 18:38:20.985758 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.985739    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-db5d01\""
Jul 28 18:38:20.985889 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.985784    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-db5d01-dockercfg-2n9w8\""
Jul 28 18:38:20.992966 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:20.992939    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"]
Jul 28 18:38:21.009294 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.009268    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hpw2\" (UniqueName: \"kubernetes.io/projected/050d2aed-043b-4fb7-970c-dd82ff5a2bd8-kube-api-access-5hpw2\") pod \"payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v\" (UID: \"050d2aed-043b-4fb7-970c-dd82ff5a2bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"
Jul 28 18:38:21.009417 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.009326    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050d2aed-043b-4fb7-970c-dd82ff5a2bd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v\" (UID: \"050d2aed-043b-4fb7-970c-dd82ff5a2bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"
Jul 28 18:38:21.107153 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.107118    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"]
Jul 28 18:38:21.110468 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.110446    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050d2aed-043b-4fb7-970c-dd82ff5a2bd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v\" (UID: \"050d2aed-043b-4fb7-970c-dd82ff5a2bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"
Jul 28 18:38:21.110598 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.110503    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"
Jul 28 18:38:21.110598 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.110524    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hpw2\" (UniqueName: \"kubernetes.io/projected/050d2aed-043b-4fb7-970c-dd82ff5a2bd8-kube-api-access-5hpw2\") pod \"payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v\" (UID: \"050d2aed-043b-4fb7-970c-dd82ff5a2bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"
Jul 28 18:38:21.111286 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.111257    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050d2aed-043b-4fb7-970c-dd82ff5a2bd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v\" (UID: \"050d2aed-043b-4fb7-970c-dd82ff5a2bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"
Jul 28 18:38:21.121255 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.121233    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"]
Jul 28 18:38:21.123555 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.123527    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hpw2\" (UniqueName: \"kubernetes.io/projected/050d2aed-043b-4fb7-970c-dd82ff5a2bd8-kube-api-access-5hpw2\") pod \"payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v\" (UID: \"050d2aed-043b-4fb7-970c-dd82ff5a2bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"
Jul 28 18:38:21.211518 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.211431    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a53e457-9640-40e3-8157-b32d6bd89980-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2\" (UID: \"0a53e457-9640-40e3-8157-b32d6bd89980\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"
Jul 28 18:38:21.211518 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.211500    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8vt5\" (UniqueName: \"kubernetes.io/projected/0a53e457-9640-40e3-8157-b32d6bd89980-kube-api-access-j8vt5\") pod \"payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2\" (UID: \"0a53e457-9640-40e3-8157-b32d6bd89980\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"
Jul 28 18:38:21.294228 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.294195    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"
Jul 28 18:38:21.312961 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.312934    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a53e457-9640-40e3-8157-b32d6bd89980-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2\" (UID: \"0a53e457-9640-40e3-8157-b32d6bd89980\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"
Jul 28 18:38:21.313232 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.313208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8vt5\" (UniqueName: \"kubernetes.io/projected/0a53e457-9640-40e3-8157-b32d6bd89980-kube-api-access-j8vt5\") pod \"payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2\" (UID: \"0a53e457-9640-40e3-8157-b32d6bd89980\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"
Jul 28 18:38:21.313722 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.313701    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a53e457-9640-40e3-8157-b32d6bd89980-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2\" (UID: \"0a53e457-9640-40e3-8157-b32d6bd89980\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"
Jul 28 18:38:21.321263 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.321242    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8vt5\" (UniqueName: \"kubernetes.io/projected/0a53e457-9640-40e3-8157-b32d6bd89980-kube-api-access-j8vt5\") pod \"payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2\" (UID: \"0a53e457-9640-40e3-8157-b32d6bd89980\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"
Jul 28 18:38:21.421147 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.421119    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"
Jul 28 18:38:21.425887 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.425860    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"]
Jul 28 18:38:21.430692 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:38:21.430664    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050d2aed_043b_4fb7_970c_dd82ff5a2bd8.slice/crio-f88d840b5748c64bdf6268637358e7e5809701e64e8d1551c51ed1dbfe84a2ef WatchSource:0}: Error finding container f88d840b5748c64bdf6268637358e7e5809701e64e8d1551c51ed1dbfe84a2ef: Status 404 returned error can't find the container with id f88d840b5748c64bdf6268637358e7e5809701e64e8d1551c51ed1dbfe84a2ef
Jul 28 18:38:21.567067 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.564008    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"]
Jul 28 18:38:21.575732 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.575703    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v" event={"ID":"050d2aed-043b-4fb7-970c-dd82ff5a2bd8","Type":"ContainerStarted","Data":"cc36811a41a531757fdf490594d276f62c0617bdd00c4edb322deee32e167a9f"}
Jul 28 18:38:21.575843 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.575762    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"
Jul 28 18:38:21.575843 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.575779    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v" event={"ID":"050d2aed-043b-4fb7-970c-dd82ff5a2bd8","Type":"ContainerStarted","Data":"f88d840b5748c64bdf6268637358e7e5809701e64e8d1551c51ed1dbfe84a2ef"}
Jul 28 18:38:21.594097 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.594056    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v" podStartSLOduration=1.5940356 podStartE2EDuration="1.5940356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:38:21.592079104 +0000 UTC m=+2236.907581247" watchObservedRunningTime="2026-07-28 18:38:21.5940356 +0000 UTC m=+2236.909537718"
Jul 28 18:38:21.918089 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.918054    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-maas-api-tls\") pod \"maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q\" (UID: \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:38:21.920416 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:21.920396    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-maas-api-tls\") pod \"maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q\" (UID: \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:38:22.109274 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:22.109244    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:38:22.242746 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:22.242654    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"]
Jul 28 18:38:22.245008 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:38:22.244978    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ee4b3d_8249_4b81_a0c5_fca06ecb38ce.slice/crio-0df052b56a01fd095ec1841e0d9392659e8897472ca2811abd31e69d92affcda WatchSource:0}: Error finding container 0df052b56a01fd095ec1841e0d9392659e8897472ca2811abd31e69d92affcda: Status 404 returned error can't find the container with id 0df052b56a01fd095ec1841e0d9392659e8897472ca2811abd31e69d92affcda
Jul 28 18:38:22.581498 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:22.581461    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q" event={"ID":"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce","Type":"ContainerStarted","Data":"0df052b56a01fd095ec1841e0d9392659e8897472ca2811abd31e69d92affcda"}
Jul 28 18:38:22.583056 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:22.583030    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2" event={"ID":"0a53e457-9640-40e3-8157-b32d6bd89980","Type":"ContainerStarted","Data":"706bfce940cd0208acc96931d060da8925edfd1d530963a0486c4af4de373996"}
Jul 28 18:38:22.583196 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:22.583082    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2" event={"ID":"0a53e457-9640-40e3-8157-b32d6bd89980","Type":"ContainerStarted","Data":"18b1402c798006e3205898f0c4f5f758d7602c9486b05c63cd18552cc5918dbd"}
Jul 28 18:38:22.583196 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:22.583183    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"
Jul 28 18:38:22.601996 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:22.601955    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2" podStartSLOduration=1.601940046 podStartE2EDuration="1.601940046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:38:22.600702966 +0000 UTC m=+2237.916205126" watchObservedRunningTime="2026-07-28 18:38:22.601940046 +0000 UTC m=+2237.917442161"
Jul 28 18:38:23.589043 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:23.589000    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q" event={"ID":"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce","Type":"ContainerStarted","Data":"73fd1fc02d6e4e3a6666f199c393e873e71ff3290c96ef13cbf84337de6446aa"}
Jul 28 18:38:23.607496 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:23.607441    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q" podStartSLOduration=3.23011094 podStartE2EDuration="3.607424007s" totalImagesPullingTime="377.313067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:38:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:38:23.605531338 +0000 UTC m=+2238.921033493" watchObservedRunningTime="2026-07-28 18:38:23.607424007 +0000 UTC m=+2238.922926121"
Jul 28 18:38:24.593506 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:24.593477    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:38:30.603844 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:30.603750    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:38:32.585382 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:32.585352    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"
Jul 28 18:38:33.591518 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:38:33.591488    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"
Jul 28 18:40:21.441401 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.441354    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"]
Jul 28 18:40:21.442144 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.441625    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp" podUID="fa409841-fea7-4f21-86bf-d5ec1a966356" containerName="maas-api" containerID="cri-o://e01025ef4c5e2a2f43dc00087706303d79d86d8caccaeb7caf27810bd1124071" gracePeriod=30
Jul 28 18:40:21.490947 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.490920    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"]
Jul 28 18:40:21.491150 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.491122    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc" podUID="bbb11b94-5fab-4d45-80e1-2b8329f90f99" containerName="payload-processing" containerID="cri-o://edfcf31bc2467d681de1be3da8a781c8ac23692719a9a0c112c99777fa98d13d" gracePeriod=30
Jul 28 18:40:21.496015 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.495939    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"]
Jul 28 18:40:21.496216 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.496190    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl" podUID="ca32e8cc-7250-494a-9303-40427eea96de" containerName="payload-pre-processing" containerID="cri-o://efc7d4d326e822eaabcc4beadccfd69bbc8a194f8539813e6fd2e04848999ab1" gracePeriod=30
Jul 28 18:40:21.735678 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.735653    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"
Jul 28 18:40:21.770567 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.770543    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"
Jul 28 18:40:21.779745 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.779717    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"
Jul 28 18:40:21.847514 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.847480    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsh7q\" (UniqueName: \"kubernetes.io/projected/fa409841-fea7-4f21-86bf-d5ec1a966356-kube-api-access-lsh7q\") pod \"fa409841-fea7-4f21-86bf-d5ec1a966356\" (UID: \"fa409841-fea7-4f21-86bf-d5ec1a966356\") "
Jul 28 18:40:21.847686 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.847541    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca32e8cc-7250-494a-9303-40427eea96de-plugins-config-volume\") pod \"ca32e8cc-7250-494a-9303-40427eea96de\" (UID: \"ca32e8cc-7250-494a-9303-40427eea96de\") "
Jul 28 18:40:21.847686 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.847577    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa409841-fea7-4f21-86bf-d5ec1a966356-maas-api-tls\") pod \"fa409841-fea7-4f21-86bf-d5ec1a966356\" (UID: \"fa409841-fea7-4f21-86bf-d5ec1a966356\") "
Jul 28 18:40:21.847786 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.847690    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6wnb\" (UniqueName: \"kubernetes.io/projected/ca32e8cc-7250-494a-9303-40427eea96de-kube-api-access-z6wnb\") pod \"ca32e8cc-7250-494a-9303-40427eea96de\" (UID: \"ca32e8cc-7250-494a-9303-40427eea96de\") "
Jul 28 18:40:21.847786 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.847777    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7fzl\" (UniqueName: \"kubernetes.io/projected/bbb11b94-5fab-4d45-80e1-2b8329f90f99-kube-api-access-n7fzl\") pod \"bbb11b94-5fab-4d45-80e1-2b8329f90f99\" (UID: \"bbb11b94-5fab-4d45-80e1-2b8329f90f99\") "
Jul 28 18:40:21.847929 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.847831    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa409841-fea7-4f21-86bf-d5ec1a966356-openshift-service-ca\") pod \"fa409841-fea7-4f21-86bf-d5ec1a966356\" (UID: \"fa409841-fea7-4f21-86bf-d5ec1a966356\") "
Jul 28 18:40:21.847929 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.847857    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb11b94-5fab-4d45-80e1-2b8329f90f99-plugins-config-volume\") pod \"bbb11b94-5fab-4d45-80e1-2b8329f90f99\" (UID: \"bbb11b94-5fab-4d45-80e1-2b8329f90f99\") "
Jul 28 18:40:21.848037 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.848010    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca32e8cc-7250-494a-9303-40427eea96de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ca32e8cc-7250-494a-9303-40427eea96de" (UID: "ca32e8cc-7250-494a-9303-40427eea96de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:40:21.848292 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.848154    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca32e8cc-7250-494a-9303-40427eea96de-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:40:21.848451 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.848423    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb11b94-5fab-4d45-80e1-2b8329f90f99-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bbb11b94-5fab-4d45-80e1-2b8329f90f99" (UID: "bbb11b94-5fab-4d45-80e1-2b8329f90f99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:40:21.848542 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.848467    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa409841-fea7-4f21-86bf-d5ec1a966356-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fa409841-fea7-4f21-86bf-d5ec1a966356" (UID: "fa409841-fea7-4f21-86bf-d5ec1a966356"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:40:21.849917 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.849886    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa409841-fea7-4f21-86bf-d5ec1a966356-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fa409841-fea7-4f21-86bf-d5ec1a966356" (UID: "fa409841-fea7-4f21-86bf-d5ec1a966356"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:40:21.849917 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.849888    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa409841-fea7-4f21-86bf-d5ec1a966356-kube-api-access-lsh7q" (OuterVolumeSpecName: "kube-api-access-lsh7q") pod "fa409841-fea7-4f21-86bf-d5ec1a966356" (UID: "fa409841-fea7-4f21-86bf-d5ec1a966356"). InnerVolumeSpecName "kube-api-access-lsh7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:40:21.850042 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.849936    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca32e8cc-7250-494a-9303-40427eea96de-kube-api-access-z6wnb" (OuterVolumeSpecName: "kube-api-access-z6wnb") pod "ca32e8cc-7250-494a-9303-40427eea96de" (UID: "ca32e8cc-7250-494a-9303-40427eea96de"). InnerVolumeSpecName "kube-api-access-z6wnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:40:21.850042 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.849945    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb11b94-5fab-4d45-80e1-2b8329f90f99-kube-api-access-n7fzl" (OuterVolumeSpecName: "kube-api-access-n7fzl") pod "bbb11b94-5fab-4d45-80e1-2b8329f90f99" (UID: "bbb11b94-5fab-4d45-80e1-2b8329f90f99"). InnerVolumeSpecName "kube-api-access-n7fzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:40:21.949516 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.949457    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsh7q\" (UniqueName: \"kubernetes.io/projected/fa409841-fea7-4f21-86bf-d5ec1a966356-kube-api-access-lsh7q\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:40:21.949516 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.949478    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa409841-fea7-4f21-86bf-d5ec1a966356-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:40:21.949516 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.949487    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6wnb\" (UniqueName: \"kubernetes.io/projected/ca32e8cc-7250-494a-9303-40427eea96de-kube-api-access-z6wnb\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:40:21.949516 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.949496    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7fzl\" (UniqueName: \"kubernetes.io/projected/bbb11b94-5fab-4d45-80e1-2b8329f90f99-kube-api-access-n7fzl\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:40:21.949516 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.949504    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa409841-fea7-4f21-86bf-d5ec1a966356-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:40:21.949516 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:21.949512    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb11b94-5fab-4d45-80e1-2b8329f90f99-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:40:22.119240 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.119208    2564 generic.go:356] "Generic (PLEG): container finished" podID="bbb11b94-5fab-4d45-80e1-2b8329f90f99" containerID="edfcf31bc2467d681de1be3da8a781c8ac23692719a9a0c112c99777fa98d13d" exitCode=0
Jul 28 18:40:22.119401 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.119252    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"
Jul 28 18:40:22.119401 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.119287    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc" event={"ID":"bbb11b94-5fab-4d45-80e1-2b8329f90f99","Type":"ContainerDied","Data":"edfcf31bc2467d681de1be3da8a781c8ac23692719a9a0c112c99777fa98d13d"}
Jul 28 18:40:22.119401 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.119334    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc" event={"ID":"bbb11b94-5fab-4d45-80e1-2b8329f90f99","Type":"ContainerDied","Data":"dec73ad121ad9a3ed39052c9c1de30da91d357f04431386970d6274b50c83525"}
Jul 28 18:40:22.119401 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.119366    2564 scope.go:117] "RemoveContainer" containerID="edfcf31bc2467d681de1be3da8a781c8ac23692719a9a0c112c99777fa98d13d"
Jul 28 18:40:22.120662 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.120634    2564 generic.go:356] "Generic (PLEG): container finished" podID="fa409841-fea7-4f21-86bf-d5ec1a966356" containerID="e01025ef4c5e2a2f43dc00087706303d79d86d8caccaeb7caf27810bd1124071" exitCode=0
Jul 28 18:40:22.120794 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.120675    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"
Jul 28 18:40:22.120794 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.120713    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp" event={"ID":"fa409841-fea7-4f21-86bf-d5ec1a966356","Type":"ContainerDied","Data":"e01025ef4c5e2a2f43dc00087706303d79d86d8caccaeb7caf27810bd1124071"}
Jul 28 18:40:22.120794 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.120758    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp" event={"ID":"fa409841-fea7-4f21-86bf-d5ec1a966356","Type":"ContainerDied","Data":"abd4906e682007f85b6224a2c4b3795e6226307b6e5e65007af8acdfcef0b693"}
Jul 28 18:40:22.122128 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.122103    2564 generic.go:356] "Generic (PLEG): container finished" podID="ca32e8cc-7250-494a-9303-40427eea96de" containerID="efc7d4d326e822eaabcc4beadccfd69bbc8a194f8539813e6fd2e04848999ab1" exitCode=0
Jul 28 18:40:22.122208 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.122141    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"
Jul 28 18:40:22.122208 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.122150    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl" event={"ID":"ca32e8cc-7250-494a-9303-40427eea96de","Type":"ContainerDied","Data":"efc7d4d326e822eaabcc4beadccfd69bbc8a194f8539813e6fd2e04848999ab1"}
Jul 28 18:40:22.122208 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.122187    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl" event={"ID":"ca32e8cc-7250-494a-9303-40427eea96de","Type":"ContainerDied","Data":"d01ebfae36de36615fd0463fd0bb066e08902768e16cef66e27208f72330b137"}
Jul 28 18:40:22.129229 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.129211    2564 scope.go:117] "RemoveContainer" containerID="edfcf31bc2467d681de1be3da8a781c8ac23692719a9a0c112c99777fa98d13d"
Jul 28 18:40:22.129589 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:40:22.129478    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edfcf31bc2467d681de1be3da8a781c8ac23692719a9a0c112c99777fa98d13d\": container with ID starting with edfcf31bc2467d681de1be3da8a781c8ac23692719a9a0c112c99777fa98d13d not found: ID does not exist" containerID="edfcf31bc2467d681de1be3da8a781c8ac23692719a9a0c112c99777fa98d13d"
Jul 28 18:40:22.129589 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.129500    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edfcf31bc2467d681de1be3da8a781c8ac23692719a9a0c112c99777fa98d13d"} err="failed to get container status \"edfcf31bc2467d681de1be3da8a781c8ac23692719a9a0c112c99777fa98d13d\": rpc error: code = NotFound desc = could not find container \"edfcf31bc2467d681de1be3da8a781c8ac23692719a9a0c112c99777fa98d13d\": container with ID starting with edfcf31bc2467d681de1be3da8a781c8ac23692719a9a0c112c99777fa98d13d not found: ID does not exist"
Jul 28 18:40:22.129589 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.129516    2564 scope.go:117] "RemoveContainer" containerID="e01025ef4c5e2a2f43dc00087706303d79d86d8caccaeb7caf27810bd1124071"
Jul 28 18:40:22.137985 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.137969    2564 scope.go:117] "RemoveContainer" containerID="e01025ef4c5e2a2f43dc00087706303d79d86d8caccaeb7caf27810bd1124071"
Jul 28 18:40:22.138222 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:40:22.138206    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e01025ef4c5e2a2f43dc00087706303d79d86d8caccaeb7caf27810bd1124071\": container with ID starting with e01025ef4c5e2a2f43dc00087706303d79d86d8caccaeb7caf27810bd1124071 not found: ID does not exist" containerID="e01025ef4c5e2a2f43dc00087706303d79d86d8caccaeb7caf27810bd1124071"
Jul 28 18:40:22.138300 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.138232    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01025ef4c5e2a2f43dc00087706303d79d86d8caccaeb7caf27810bd1124071"} err="failed to get container status \"e01025ef4c5e2a2f43dc00087706303d79d86d8caccaeb7caf27810bd1124071\": rpc error: code = NotFound desc = could not find container \"e01025ef4c5e2a2f43dc00087706303d79d86d8caccaeb7caf27810bd1124071\": container with ID starting with e01025ef4c5e2a2f43dc00087706303d79d86d8caccaeb7caf27810bd1124071 not found: ID does not exist"
Jul 28 18:40:22.138300 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.138252    2564 scope.go:117] "RemoveContainer" containerID="efc7d4d326e822eaabcc4beadccfd69bbc8a194f8539813e6fd2e04848999ab1"
Jul 28 18:40:22.143523 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.143497    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"]
Jul 28 18:40:22.146187 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.146167    2564 scope.go:117] "RemoveContainer" containerID="efc7d4d326e822eaabcc4beadccfd69bbc8a194f8539813e6fd2e04848999ab1"
Jul 28 18:40:22.146424 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:40:22.146405    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc7d4d326e822eaabcc4beadccfd69bbc8a194f8539813e6fd2e04848999ab1\": container with ID starting with efc7d4d326e822eaabcc4beadccfd69bbc8a194f8539813e6fd2e04848999ab1 not found: ID does not exist" containerID="efc7d4d326e822eaabcc4beadccfd69bbc8a194f8539813e6fd2e04848999ab1"
Jul 28 18:40:22.146490 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.146430    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc7d4d326e822eaabcc4beadccfd69bbc8a194f8539813e6fd2e04848999ab1"} err="failed to get container status \"efc7d4d326e822eaabcc4beadccfd69bbc8a194f8539813e6fd2e04848999ab1\": rpc error: code = NotFound desc = could not find container \"efc7d4d326e822eaabcc4beadccfd69bbc8a194f8539813e6fd2e04848999ab1\": container with ID starting with efc7d4d326e822eaabcc4beadccfd69bbc8a194f8539813e6fd2e04848999ab1 not found: ID does not exist"
Jul 28 18:40:22.147400 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.147383    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-61d29d-67f68954-xqpjc"]
Jul 28 18:40:22.160722 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.160702    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"]
Jul 28 18:40:22.162782 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.162765    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-61d29d-56695cdff6-6rrgl"]
Jul 28 18:40:22.173838 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.173817    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"]
Jul 28 18:40:22.177707 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:22.177688    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61d29d-58f9944d9d-pm4dp"]
Jul 28 18:40:23.217337 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:23.217302    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbb11b94-5fab-4d45-80e1-2b8329f90f99" path="/var/lib/kubelet/pods/bbb11b94-5fab-4d45-80e1-2b8329f90f99/volumes"
Jul 28 18:40:23.217713 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:23.217649    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca32e8cc-7250-494a-9303-40427eea96de" path="/var/lib/kubelet/pods/ca32e8cc-7250-494a-9303-40427eea96de/volumes"
Jul 28 18:40:23.217996 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:23.217982    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa409841-fea7-4f21-86bf-d5ec1a966356" path="/var/lib/kubelet/pods/fa409841-fea7-4f21-86bf-d5ec1a966356/volumes"
Jul 28 18:40:39.321310 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.321271    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"]
Jul 28 18:40:39.321857 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.321536    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q" podUID="12ee4b3d-8249-4b81-a0c5-fca06ecb38ce" containerName="maas-api" containerID="cri-o://73fd1fc02d6e4e3a6666f199c393e873e71ff3290c96ef13cbf84337de6446aa" gracePeriod=30
Jul 28 18:40:39.373850 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.373796    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"]
Jul 28 18:40:39.374127 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.374075    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2" podUID="0a53e457-9640-40e3-8157-b32d6bd89980" containerName="payload-processing" containerID="cri-o://706bfce940cd0208acc96931d060da8925edfd1d530963a0486c4af4de373996" gracePeriod=30
Jul 28 18:40:39.378691 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.378451    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"]
Jul 28 18:40:39.378691 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.378675    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v" podUID="050d2aed-043b-4fb7-970c-dd82ff5a2bd8" containerName="payload-pre-processing" containerID="cri-o://cc36811a41a531757fdf490594d276f62c0617bdd00c4edb322deee32e167a9f" gracePeriod=30
Jul 28 18:40:39.637854 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.637827    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:40:39.640426 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.640406    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"
Jul 28 18:40:39.664751 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.664727    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"
Jul 28 18:40:39.800773 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.800735    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jrhv\" (UniqueName: \"kubernetes.io/projected/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-kube-api-access-9jrhv\") pod \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\" (UID: \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\") "
Jul 28 18:40:39.801002 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.800840    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050d2aed-043b-4fb7-970c-dd82ff5a2bd8-plugins-config-volume\") pod \"050d2aed-043b-4fb7-970c-dd82ff5a2bd8\" (UID: \"050d2aed-043b-4fb7-970c-dd82ff5a2bd8\") "
Jul 28 18:40:39.801002 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.800868    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-openshift-service-ca\") pod \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\" (UID: \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\") "
Jul 28 18:40:39.801002 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.800894    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8vt5\" (UniqueName: \"kubernetes.io/projected/0a53e457-9640-40e3-8157-b32d6bd89980-kube-api-access-j8vt5\") pod \"0a53e457-9640-40e3-8157-b32d6bd89980\" (UID: \"0a53e457-9640-40e3-8157-b32d6bd89980\") "
Jul 28 18:40:39.801002 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.800915    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hpw2\" (UniqueName: \"kubernetes.io/projected/050d2aed-043b-4fb7-970c-dd82ff5a2bd8-kube-api-access-5hpw2\") pod \"050d2aed-043b-4fb7-970c-dd82ff5a2bd8\" (UID: \"050d2aed-043b-4fb7-970c-dd82ff5a2bd8\") "
Jul 28 18:40:39.801002 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.800943    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-maas-api-tls\") pod \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\" (UID: \"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce\") "
Jul 28 18:40:39.801002 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.800977    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a53e457-9640-40e3-8157-b32d6bd89980-plugins-config-volume\") pod \"0a53e457-9640-40e3-8157-b32d6bd89980\" (UID: \"0a53e457-9640-40e3-8157-b32d6bd89980\") "
Jul 28 18:40:39.801388 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.801275    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "12ee4b3d-8249-4b81-a0c5-fca06ecb38ce" (UID: "12ee4b3d-8249-4b81-a0c5-fca06ecb38ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:40:39.801517 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.801485    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050d2aed-043b-4fb7-970c-dd82ff5a2bd8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "050d2aed-043b-4fb7-970c-dd82ff5a2bd8" (UID: "050d2aed-043b-4fb7-970c-dd82ff5a2bd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:40:39.801945 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.801919    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a53e457-9640-40e3-8157-b32d6bd89980-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a53e457-9640-40e3-8157-b32d6bd89980" (UID: "0a53e457-9640-40e3-8157-b32d6bd89980"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:40:39.803073 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.803040    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "12ee4b3d-8249-4b81-a0c5-fca06ecb38ce" (UID: "12ee4b3d-8249-4b81-a0c5-fca06ecb38ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:40:39.803172 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.803104    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a53e457-9640-40e3-8157-b32d6bd89980-kube-api-access-j8vt5" (OuterVolumeSpecName: "kube-api-access-j8vt5") pod "0a53e457-9640-40e3-8157-b32d6bd89980" (UID: "0a53e457-9640-40e3-8157-b32d6bd89980"). InnerVolumeSpecName "kube-api-access-j8vt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:40:39.803172 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.803116    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-kube-api-access-9jrhv" (OuterVolumeSpecName: "kube-api-access-9jrhv") pod "12ee4b3d-8249-4b81-a0c5-fca06ecb38ce" (UID: "12ee4b3d-8249-4b81-a0c5-fca06ecb38ce"). InnerVolumeSpecName "kube-api-access-9jrhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:40:39.803172 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.803136    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050d2aed-043b-4fb7-970c-dd82ff5a2bd8-kube-api-access-5hpw2" (OuterVolumeSpecName: "kube-api-access-5hpw2") pod "050d2aed-043b-4fb7-970c-dd82ff5a2bd8" (UID: "050d2aed-043b-4fb7-970c-dd82ff5a2bd8"). InnerVolumeSpecName "kube-api-access-5hpw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:40:39.902266 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.902166    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:40:39.902266 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.902200    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8vt5\" (UniqueName: \"kubernetes.io/projected/0a53e457-9640-40e3-8157-b32d6bd89980-kube-api-access-j8vt5\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:40:39.902266 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.902212    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hpw2\" (UniqueName: \"kubernetes.io/projected/050d2aed-043b-4fb7-970c-dd82ff5a2bd8-kube-api-access-5hpw2\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:40:39.902266 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.902221    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:40:39.902266 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.902229    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a53e457-9640-40e3-8157-b32d6bd89980-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:40:39.902266 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.902237    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jrhv\" (UniqueName: \"kubernetes.io/projected/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce-kube-api-access-9jrhv\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:40:39.902266 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:39.902246    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050d2aed-043b-4fb7-970c-dd82ff5a2bd8-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:40:40.207117 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.207035    2564 generic.go:356] "Generic (PLEG): container finished" podID="12ee4b3d-8249-4b81-a0c5-fca06ecb38ce" containerID="73fd1fc02d6e4e3a6666f199c393e873e71ff3290c96ef13cbf84337de6446aa" exitCode=0
Jul 28 18:40:40.207117 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.207081    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"
Jul 28 18:40:40.207342 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.207113    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q" event={"ID":"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce","Type":"ContainerDied","Data":"73fd1fc02d6e4e3a6666f199c393e873e71ff3290c96ef13cbf84337de6446aa"}
Jul 28 18:40:40.207342 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.207178    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q" event={"ID":"12ee4b3d-8249-4b81-a0c5-fca06ecb38ce","Type":"ContainerDied","Data":"0df052b56a01fd095ec1841e0d9392659e8897472ca2811abd31e69d92affcda"}
Jul 28 18:40:40.207342 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.207211    2564 scope.go:117] "RemoveContainer" containerID="73fd1fc02d6e4e3a6666f199c393e873e71ff3290c96ef13cbf84337de6446aa"
Jul 28 18:40:40.208487 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.208466    2564 generic.go:356] "Generic (PLEG): container finished" podID="0a53e457-9640-40e3-8157-b32d6bd89980" containerID="706bfce940cd0208acc96931d060da8925edfd1d530963a0486c4af4de373996" exitCode=0
Jul 28 18:40:40.208576 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.208511    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"
Jul 28 18:40:40.208576 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.208554    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2" event={"ID":"0a53e457-9640-40e3-8157-b32d6bd89980","Type":"ContainerDied","Data":"706bfce940cd0208acc96931d060da8925edfd1d530963a0486c4af4de373996"}
Jul 28 18:40:40.208687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.208607    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2" event={"ID":"0a53e457-9640-40e3-8157-b32d6bd89980","Type":"ContainerDied","Data":"18b1402c798006e3205898f0c4f5f758d7602c9486b05c63cd18552cc5918dbd"}
Jul 28 18:40:40.209965 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.209942    2564 generic.go:356] "Generic (PLEG): container finished" podID="050d2aed-043b-4fb7-970c-dd82ff5a2bd8" containerID="cc36811a41a531757fdf490594d276f62c0617bdd00c4edb322deee32e167a9f" exitCode=0
Jul 28 18:40:40.210120 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.209998    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v" event={"ID":"050d2aed-043b-4fb7-970c-dd82ff5a2bd8","Type":"ContainerDied","Data":"cc36811a41a531757fdf490594d276f62c0617bdd00c4edb322deee32e167a9f"}
Jul 28 18:40:40.210120 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.210002    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"
Jul 28 18:40:40.210120 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.210033    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v" event={"ID":"050d2aed-043b-4fb7-970c-dd82ff5a2bd8","Type":"ContainerDied","Data":"f88d840b5748c64bdf6268637358e7e5809701e64e8d1551c51ed1dbfe84a2ef"}
Jul 28 18:40:40.216332 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.216307    2564 scope.go:117] "RemoveContainer" containerID="73fd1fc02d6e4e3a6666f199c393e873e71ff3290c96ef13cbf84337de6446aa"
Jul 28 18:40:40.216586 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:40:40.216558    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73fd1fc02d6e4e3a6666f199c393e873e71ff3290c96ef13cbf84337de6446aa\": container with ID starting with 73fd1fc02d6e4e3a6666f199c393e873e71ff3290c96ef13cbf84337de6446aa not found: ID does not exist" containerID="73fd1fc02d6e4e3a6666f199c393e873e71ff3290c96ef13cbf84337de6446aa"
Jul 28 18:40:40.216692 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.216594    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fd1fc02d6e4e3a6666f199c393e873e71ff3290c96ef13cbf84337de6446aa"} err="failed to get container status \"73fd1fc02d6e4e3a6666f199c393e873e71ff3290c96ef13cbf84337de6446aa\": rpc error: code = NotFound desc = could not find container \"73fd1fc02d6e4e3a6666f199c393e873e71ff3290c96ef13cbf84337de6446aa\": container with ID starting with 73fd1fc02d6e4e3a6666f199c393e873e71ff3290c96ef13cbf84337de6446aa not found: ID does not exist"
Jul 28 18:40:40.216692 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.216617    2564 scope.go:117] "RemoveContainer" containerID="706bfce940cd0208acc96931d060da8925edfd1d530963a0486c4af4de373996"
Jul 28 18:40:40.224966 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.224951    2564 scope.go:117] "RemoveContainer" containerID="706bfce940cd0208acc96931d060da8925edfd1d530963a0486c4af4de373996"
Jul 28 18:40:40.225206 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:40:40.225185    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"706bfce940cd0208acc96931d060da8925edfd1d530963a0486c4af4de373996\": container with ID starting with 706bfce940cd0208acc96931d060da8925edfd1d530963a0486c4af4de373996 not found: ID does not exist" containerID="706bfce940cd0208acc96931d060da8925edfd1d530963a0486c4af4de373996"
Jul 28 18:40:40.225317 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.225213    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706bfce940cd0208acc96931d060da8925edfd1d530963a0486c4af4de373996"} err="failed to get container status \"706bfce940cd0208acc96931d060da8925edfd1d530963a0486c4af4de373996\": rpc error: code = NotFound desc = could not find container \"706bfce940cd0208acc96931d060da8925edfd1d530963a0486c4af4de373996\": container with ID starting with 706bfce940cd0208acc96931d060da8925edfd1d530963a0486c4af4de373996 not found: ID does not exist"
Jul 28 18:40:40.225317 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.225234    2564 scope.go:117] "RemoveContainer" containerID="cc36811a41a531757fdf490594d276f62c0617bdd00c4edb322deee32e167a9f"
Jul 28 18:40:40.232575 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.232554    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"]
Jul 28 18:40:40.234526 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.234502    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db5d01-7f664c4647-zz89q"]
Jul 28 18:40:40.238754 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.238738    2564 scope.go:117] "RemoveContainer" containerID="cc36811a41a531757fdf490594d276f62c0617bdd00c4edb322deee32e167a9f"
Jul 28 18:40:40.239028 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:40:40.239010    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc36811a41a531757fdf490594d276f62c0617bdd00c4edb322deee32e167a9f\": container with ID starting with cc36811a41a531757fdf490594d276f62c0617bdd00c4edb322deee32e167a9f not found: ID does not exist" containerID="cc36811a41a531757fdf490594d276f62c0617bdd00c4edb322deee32e167a9f"
Jul 28 18:40:40.239099 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.239035    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc36811a41a531757fdf490594d276f62c0617bdd00c4edb322deee32e167a9f"} err="failed to get container status \"cc36811a41a531757fdf490594d276f62c0617bdd00c4edb322deee32e167a9f\": rpc error: code = NotFound desc = could not find container \"cc36811a41a531757fdf490594d276f62c0617bdd00c4edb322deee32e167a9f\": container with ID starting with cc36811a41a531757fdf490594d276f62c0617bdd00c4edb322deee32e167a9f not found: ID does not exist"
Jul 28 18:40:40.243515 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.243496    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"]
Jul 28 18:40:40.248246 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.248226    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db5d01-6c7f84fc56-9wdf2"]
Jul 28 18:40:40.258165 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.258145    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"]
Jul 28 18:40:40.261583 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:40.261565    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db5d01-6949554d4d-zrr5v"]
Jul 28 18:40:41.215902 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:41.215872    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="050d2aed-043b-4fb7-970c-dd82ff5a2bd8" path="/var/lib/kubelet/pods/050d2aed-043b-4fb7-970c-dd82ff5a2bd8/volumes"
Jul 28 18:40:41.216333 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:41.216299    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a53e457-9640-40e3-8157-b32d6bd89980" path="/var/lib/kubelet/pods/0a53e457-9640-40e3-8157-b32d6bd89980/volumes"
Jul 28 18:40:41.216767 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:41.216742    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12ee4b3d-8249-4b81-a0c5-fca06ecb38ce" path="/var/lib/kubelet/pods/12ee4b3d-8249-4b81-a0c5-fca06ecb38ce/volumes"
Jul 28 18:40:50.083032 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083003    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"]
Jul 28 18:40:50.083399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083284    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa409841-fea7-4f21-86bf-d5ec1a966356" containerName="maas-api"
Jul 28 18:40:50.083399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083295    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa409841-fea7-4f21-86bf-d5ec1a966356" containerName="maas-api"
Jul 28 18:40:50.083399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083304    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050d2aed-043b-4fb7-970c-dd82ff5a2bd8" containerName="payload-pre-processing"
Jul 28 18:40:50.083399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083310    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="050d2aed-043b-4fb7-970c-dd82ff5a2bd8" containerName="payload-pre-processing"
Jul 28 18:40:50.083399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083319    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a53e457-9640-40e3-8157-b32d6bd89980" containerName="payload-processing"
Jul 28 18:40:50.083399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083325    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a53e457-9640-40e3-8157-b32d6bd89980" containerName="payload-processing"
Jul 28 18:40:50.083399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083333    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12ee4b3d-8249-4b81-a0c5-fca06ecb38ce" containerName="maas-api"
Jul 28 18:40:50.083399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083340    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="12ee4b3d-8249-4b81-a0c5-fca06ecb38ce" containerName="maas-api"
Jul 28 18:40:50.083399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083348    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbb11b94-5fab-4d45-80e1-2b8329f90f99" containerName="payload-processing"
Jul 28 18:40:50.083399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083353    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbb11b94-5fab-4d45-80e1-2b8329f90f99" containerName="payload-processing"
Jul 28 18:40:50.083399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083360    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca32e8cc-7250-494a-9303-40427eea96de" containerName="payload-pre-processing"
Jul 28 18:40:50.083399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083366    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca32e8cc-7250-494a-9303-40427eea96de" containerName="payload-pre-processing"
Jul 28 18:40:50.083774 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083414    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa409841-fea7-4f21-86bf-d5ec1a966356" containerName="maas-api"
Jul 28 18:40:50.083774 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083421    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbb11b94-5fab-4d45-80e1-2b8329f90f99" containerName="payload-processing"
Jul 28 18:40:50.083774 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083427    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a53e457-9640-40e3-8157-b32d6bd89980" containerName="payload-processing"
Jul 28 18:40:50.083774 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083434    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca32e8cc-7250-494a-9303-40427eea96de" containerName="payload-pre-processing"
Jul 28 18:40:50.083774 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083440    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="050d2aed-043b-4fb7-970c-dd82ff5a2bd8" containerName="payload-pre-processing"
Jul 28 18:40:50.083774 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.083445    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="12ee4b3d-8249-4b81-a0c5-fca06ecb38ce" containerName="maas-api"
Jul 28 18:40:50.087839 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.087818    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"
Jul 28 18:40:50.090326 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.090302    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gffgz\""
Jul 28 18:40:50.090472 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.090397    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 18:40:50.091455 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.091434    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ac88b3-serving-cert\""
Jul 28 18:40:50.091455 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.091446    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 18:40:50.097857 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.097833    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"]
Jul 28 18:40:50.186592 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.186557    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6rcz\" (UniqueName: \"kubernetes.io/projected/0e97caab-2dfa-4a67-9325-585f1732da7d-kube-api-access-p6rcz\") pod \"maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb\" (UID: \"0e97caab-2dfa-4a67-9325-585f1732da7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"
Jul 28 18:40:50.186748 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.186679    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e97caab-2dfa-4a67-9325-585f1732da7d-openshift-service-ca\") pod \"maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb\" (UID: \"0e97caab-2dfa-4a67-9325-585f1732da7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"
Jul 28 18:40:50.186748 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.186732    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e97caab-2dfa-4a67-9325-585f1732da7d-maas-api-tls\") pod \"maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb\" (UID: \"0e97caab-2dfa-4a67-9325-585f1732da7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"
Jul 28 18:40:50.288051 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.288013    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e97caab-2dfa-4a67-9325-585f1732da7d-maas-api-tls\") pod \"maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb\" (UID: \"0e97caab-2dfa-4a67-9325-585f1732da7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"
Jul 28 18:40:50.288252 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.288066    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6rcz\" (UniqueName: \"kubernetes.io/projected/0e97caab-2dfa-4a67-9325-585f1732da7d-kube-api-access-p6rcz\") pod \"maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb\" (UID: \"0e97caab-2dfa-4a67-9325-585f1732da7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"
Jul 28 18:40:50.288252 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.288108    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e97caab-2dfa-4a67-9325-585f1732da7d-openshift-service-ca\") pod \"maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb\" (UID: \"0e97caab-2dfa-4a67-9325-585f1732da7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"
Jul 28 18:40:50.288252 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:40:50.288168    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-serving-cert: secret "maas-api-e2e-api-a-ac88b3-serving-cert" not found
Jul 28 18:40:50.288252 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:40:50.288242    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e97caab-2dfa-4a67-9325-585f1732da7d-maas-api-tls podName:0e97caab-2dfa-4a67-9325-585f1732da7d nodeName:}" failed. No retries permitted until 2026-07-28 18:40:50.788223193 +0000 UTC m=+2386.103725284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e97caab-2dfa-4a67-9325-585f1732da7d-maas-api-tls") pod "maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb" (UID: "0e97caab-2dfa-4a67-9325-585f1732da7d") : secret "maas-api-e2e-api-a-ac88b3-serving-cert" not found
Jul 28 18:40:50.288703 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.288685    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e97caab-2dfa-4a67-9325-585f1732da7d-openshift-service-ca\") pod \"maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb\" (UID: \"0e97caab-2dfa-4a67-9325-585f1732da7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"
Jul 28 18:40:50.299045 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.299016    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6rcz\" (UniqueName: \"kubernetes.io/projected/0e97caab-2dfa-4a67-9325-585f1732da7d-kube-api-access-p6rcz\") pod \"maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb\" (UID: \"0e97caab-2dfa-4a67-9325-585f1732da7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"
Jul 28 18:40:50.675860 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.675822    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"]
Jul 28 18:40:50.678994 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.678971    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"
Jul 28 18:40:50.681520 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.681498    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ac88b3-dockercfg-zl57g\""
Jul 28 18:40:50.681629 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.681522    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ac88b3\""
Jul 28 18:40:50.688535 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.688507    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"]
Jul 28 18:40:50.691938 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.691917    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"]
Jul 28 18:40:50.692023 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.692001    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"
Jul 28 18:40:50.692428 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.692406    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bcc7\" (UniqueName: \"kubernetes.io/projected/17139879-f19c-4b2c-abbc-31d59f003fb9-kube-api-access-8bcc7\") pod \"payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg\" (UID: \"17139879-f19c-4b2c-abbc-31d59f003fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"
Jul 28 18:40:50.692519 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.692458    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17139879-f19c-4b2c-abbc-31d59f003fb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg\" (UID: \"17139879-f19c-4b2c-abbc-31d59f003fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"
Jul 28 18:40:50.692603 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.692588    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf78a7bf-821a-425b-85d6-d812cdb23058-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5\" (UID: \"bf78a7bf-821a-425b-85d6-d812cdb23058\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"
Jul 28 18:40:50.692659 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.692622    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x66bp\" (UniqueName: \"kubernetes.io/projected/bf78a7bf-821a-425b-85d6-d812cdb23058-kube-api-access-x66bp\") pod \"payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5\" (UID: \"bf78a7bf-821a-425b-85d6-d812cdb23058\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"
Jul 28 18:40:50.702046 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.702008    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"]
Jul 28 18:40:50.793427 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.793390    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x66bp\" (UniqueName: \"kubernetes.io/projected/bf78a7bf-821a-425b-85d6-d812cdb23058-kube-api-access-x66bp\") pod \"payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5\" (UID: \"bf78a7bf-821a-425b-85d6-d812cdb23058\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"
Jul 28 18:40:50.793427 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.793430    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bcc7\" (UniqueName: \"kubernetes.io/projected/17139879-f19c-4b2c-abbc-31d59f003fb9-kube-api-access-8bcc7\") pod \"payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg\" (UID: \"17139879-f19c-4b2c-abbc-31d59f003fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"
Jul 28 18:40:50.793685 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.793462    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17139879-f19c-4b2c-abbc-31d59f003fb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg\" (UID: \"17139879-f19c-4b2c-abbc-31d59f003fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"
Jul 28 18:40:50.793685 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.793507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e97caab-2dfa-4a67-9325-585f1732da7d-maas-api-tls\") pod \"maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb\" (UID: \"0e97caab-2dfa-4a67-9325-585f1732da7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"
Jul 28 18:40:50.793685 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.793542    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf78a7bf-821a-425b-85d6-d812cdb23058-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5\" (UID: \"bf78a7bf-821a-425b-85d6-d812cdb23058\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"
Jul 28 18:40:50.794248 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.794223    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17139879-f19c-4b2c-abbc-31d59f003fb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg\" (UID: \"17139879-f19c-4b2c-abbc-31d59f003fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"
Jul 28 18:40:50.794344 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.794247    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf78a7bf-821a-425b-85d6-d812cdb23058-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5\" (UID: \"bf78a7bf-821a-425b-85d6-d812cdb23058\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"
Jul 28 18:40:50.795975 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.795951    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e97caab-2dfa-4a67-9325-585f1732da7d-maas-api-tls\") pod \"maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb\" (UID: \"0e97caab-2dfa-4a67-9325-585f1732da7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"
Jul 28 18:40:50.811259 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.811233    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bcc7\" (UniqueName: \"kubernetes.io/projected/17139879-f19c-4b2c-abbc-31d59f003fb9-kube-api-access-8bcc7\") pod \"payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg\" (UID: \"17139879-f19c-4b2c-abbc-31d59f003fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"
Jul 28 18:40:50.811402 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.811378    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x66bp\" (UniqueName: \"kubernetes.io/projected/bf78a7bf-821a-425b-85d6-d812cdb23058-kube-api-access-x66bp\") pod \"payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5\" (UID: \"bf78a7bf-821a-425b-85d6-d812cdb23058\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"
Jul 28 18:40:50.991086 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:50.990990    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"
Jul 28 18:40:51.001088 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:51.001059    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"
Jul 28 18:40:51.002174 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:51.002148    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"
Jul 28 18:40:51.151033 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:51.150999    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"]
Jul 28 18:40:51.155431 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:40:51.155401    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17139879_f19c_4b2c_abbc_31d59f003fb9.slice/crio-a485cedcb6bdc342344b14fbf77733d7504e20e2a3dce5f45178cb37a4efab60 WatchSource:0}: Error finding container a485cedcb6bdc342344b14fbf77733d7504e20e2a3dce5f45178cb37a4efab60: Status 404 returned error can't find the container with id a485cedcb6bdc342344b14fbf77733d7504e20e2a3dce5f45178cb37a4efab60
Jul 28 18:40:51.260546 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:51.260518    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg" event={"ID":"17139879-f19c-4b2c-abbc-31d59f003fb9","Type":"ContainerStarted","Data":"c1948e2b4ea381381f8a09006d8958e3fd0bfd0e646de342d3dc55a37cb1cf32"}
Jul 28 18:40:51.260712 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:51.260568    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg" event={"ID":"17139879-f19c-4b2c-abbc-31d59f003fb9","Type":"ContainerStarted","Data":"a485cedcb6bdc342344b14fbf77733d7504e20e2a3dce5f45178cb37a4efab60"}
Jul 28 18:40:51.260712 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:51.260625    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"
Jul 28 18:40:51.281328 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:51.281283    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg" podStartSLOduration=1.2812662879999999 podStartE2EDuration="1.281266288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:40:51.280365104 +0000 UTC m=+2386.595867240" watchObservedRunningTime="2026-07-28 18:40:51.281266288 +0000 UTC m=+2386.596768400"
Jul 28 18:40:51.405338 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:51.405312    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"]
Jul 28 18:40:51.407481 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:51.407240    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"]
Jul 28 18:40:51.407481 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:40:51.407450    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e97caab_2dfa_4a67_9325_585f1732da7d.slice/crio-1adc712461b58af2e36968796e43080e09fffa9f15500bccd6e4cdc0497ce2e0 WatchSource:0}: Error finding container 1adc712461b58af2e36968796e43080e09fffa9f15500bccd6e4cdc0497ce2e0: Status 404 returned error can't find the container with id 1adc712461b58af2e36968796e43080e09fffa9f15500bccd6e4cdc0497ce2e0
Jul 28 18:40:51.408434 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:40:51.408309    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf78a7bf_821a_425b_85d6_d812cdb23058.slice/crio-20a279f46f1ca1e35c6f0eb7a20251d3ac265e2f5b389986be9e335a2601fe65 WatchSource:0}: Error finding container 20a279f46f1ca1e35c6f0eb7a20251d3ac265e2f5b389986be9e335a2601fe65: Status 404 returned error can't find the container with id 20a279f46f1ca1e35c6f0eb7a20251d3ac265e2f5b389986be9e335a2601fe65
Jul 28 18:40:51.409237 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:51.409212    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:40:52.266289 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:52.266245    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5" event={"ID":"bf78a7bf-821a-425b-85d6-d812cdb23058","Type":"ContainerStarted","Data":"bbd721f0dc14d7111d1c2924bda1b1a40f84f9db5fd62bbab6e1dc4ee127318f"}
Jul 28 18:40:52.266718 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:52.266313    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5" event={"ID":"bf78a7bf-821a-425b-85d6-d812cdb23058","Type":"ContainerStarted","Data":"20a279f46f1ca1e35c6f0eb7a20251d3ac265e2f5b389986be9e335a2601fe65"}
Jul 28 18:40:52.266718 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:52.266344    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"
Jul 28 18:40:52.267739 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:52.267716    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb" event={"ID":"0e97caab-2dfa-4a67-9325-585f1732da7d","Type":"ContainerStarted","Data":"35a21bec3bba27c74a90d3d9eae7b6cafe107ab9778fafe0604365421653abee"}
Jul 28 18:40:52.267848 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:52.267759    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb" event={"ID":"0e97caab-2dfa-4a67-9325-585f1732da7d","Type":"ContainerStarted","Data":"1adc712461b58af2e36968796e43080e09fffa9f15500bccd6e4cdc0497ce2e0"}
Jul 28 18:40:52.285381 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:52.285334    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5" podStartSLOduration=2.2853174640000002 podStartE2EDuration="2.285317464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:40:52.283645082 +0000 UTC m=+2387.599147194" watchObservedRunningTime="2026-07-28 18:40:52.285317464 +0000 UTC m=+2387.600819576"
Jul 28 18:40:52.300575 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:52.300517    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb" podStartSLOduration=1.766301406 podStartE2EDuration="2.300498489s" totalImagesPullingTime="534.197083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:40:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:40:52.300442691 +0000 UTC m=+2387.615944840" watchObservedRunningTime="2026-07-28 18:40:52.300498489 +0000 UTC m=+2387.616000605"
Jul 28 18:40:53.272179 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:53.272149    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"
Jul 28 18:40:59.282070 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:40:59.282037    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"
Jul 28 18:41:02.269941 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:02.269912    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"
Jul 28 18:41:03.274853 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:03.274802    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"
Jul 28 18:41:19.362752 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.362718    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"]
Jul 28 18:41:19.365989 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.365957    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"
Jul 28 18:41:19.368843 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.368792    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f7f010-serving-cert\""
Jul 28 18:41:19.378181 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.378156    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"]
Jul 28 18:41:19.411959 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.411921    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8936dc-b093-48b1-8dad-6ee4e3b0f980-openshift-service-ca\") pod \"maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf\" (UID: \"da8936dc-b093-48b1-8dad-6ee4e3b0f980\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"
Jul 28 18:41:19.412131 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.412021    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvtfj\" (UniqueName: \"kubernetes.io/projected/da8936dc-b093-48b1-8dad-6ee4e3b0f980-kube-api-access-pvtfj\") pod \"maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf\" (UID: \"da8936dc-b093-48b1-8dad-6ee4e3b0f980\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"
Jul 28 18:41:19.412131 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.412067    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8936dc-b093-48b1-8dad-6ee4e3b0f980-maas-api-tls\") pod \"maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf\" (UID: \"da8936dc-b093-48b1-8dad-6ee4e3b0f980\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"
Jul 28 18:41:19.513009 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.512970    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvtfj\" (UniqueName: \"kubernetes.io/projected/da8936dc-b093-48b1-8dad-6ee4e3b0f980-kube-api-access-pvtfj\") pod \"maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf\" (UID: \"da8936dc-b093-48b1-8dad-6ee4e3b0f980\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"
Jul 28 18:41:19.513009 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.513013    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8936dc-b093-48b1-8dad-6ee4e3b0f980-maas-api-tls\") pod \"maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf\" (UID: \"da8936dc-b093-48b1-8dad-6ee4e3b0f980\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"
Jul 28 18:41:19.513223 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:41:19.513112    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-serving-cert: secret "maas-api-e2e-api-b-f7f010-serving-cert" not found
Jul 28 18:41:19.513223 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.513135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8936dc-b093-48b1-8dad-6ee4e3b0f980-openshift-service-ca\") pod \"maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf\" (UID: \"da8936dc-b093-48b1-8dad-6ee4e3b0f980\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"
Jul 28 18:41:19.513223 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:41:19.513162    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8936dc-b093-48b1-8dad-6ee4e3b0f980-maas-api-tls podName:da8936dc-b093-48b1-8dad-6ee4e3b0f980 nodeName:}" failed. No retries permitted until 2026-07-28 18:41:20.013144973 +0000 UTC m=+2415.328647064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da8936dc-b093-48b1-8dad-6ee4e3b0f980-maas-api-tls") pod "maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf" (UID: "da8936dc-b093-48b1-8dad-6ee4e3b0f980") : secret "maas-api-e2e-api-b-f7f010-serving-cert" not found
Jul 28 18:41:19.513691 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.513674    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8936dc-b093-48b1-8dad-6ee4e3b0f980-openshift-service-ca\") pod \"maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf\" (UID: \"da8936dc-b093-48b1-8dad-6ee4e3b0f980\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"
Jul 28 18:41:19.521745 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.521718    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvtfj\" (UniqueName: \"kubernetes.io/projected/da8936dc-b093-48b1-8dad-6ee4e3b0f980-kube-api-access-pvtfj\") pod \"maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf\" (UID: \"da8936dc-b093-48b1-8dad-6ee4e3b0f980\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"
Jul 28 18:41:19.950719 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.950682    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"]
Jul 28 18:41:19.953995 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.953979    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"
Jul 28 18:41:19.956788 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.956764    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f7f010-dockercfg-h9bkq\""
Jul 28 18:41:19.956911 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.956831    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f7f010\""
Jul 28 18:41:19.963702 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.963672    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"]
Jul 28 18:41:19.988193 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.988159    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"]
Jul 28 18:41:19.991322 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:19.991304    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"
Jul 28 18:41:20.001551 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.001508    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"]
Jul 28 18:41:20.016551 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.016519    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8936dc-b093-48b1-8dad-6ee4e3b0f980-maas-api-tls\") pod \"maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf\" (UID: \"da8936dc-b093-48b1-8dad-6ee4e3b0f980\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"
Jul 28 18:41:20.016551 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.016554    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzwxm\" (UniqueName: \"kubernetes.io/projected/5258dc4d-087f-4815-9135-6a32c5a0d243-kube-api-access-jzwxm\") pod \"payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch\" (UID: \"5258dc4d-087f-4815-9135-6a32c5a0d243\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"
Jul 28 18:41:20.016891 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.016658    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5258dc4d-087f-4815-9135-6a32c5a0d243-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch\" (UID: \"5258dc4d-087f-4815-9135-6a32c5a0d243\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"
Jul 28 18:41:20.016891 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.016696    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cefa225-2ae8-4df6-8839-2a50e37ab2ca-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5\" (UID: \"1cefa225-2ae8-4df6-8839-2a50e37ab2ca\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"
Jul 28 18:41:20.016891 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.016743    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gts96\" (UniqueName: \"kubernetes.io/projected/1cefa225-2ae8-4df6-8839-2a50e37ab2ca-kube-api-access-gts96\") pod \"payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5\" (UID: \"1cefa225-2ae8-4df6-8839-2a50e37ab2ca\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"
Jul 28 18:41:20.019632 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.019601    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8936dc-b093-48b1-8dad-6ee4e3b0f980-maas-api-tls\") pod \"maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf\" (UID: \"da8936dc-b093-48b1-8dad-6ee4e3b0f980\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"
Jul 28 18:41:20.118113 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.118075    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzwxm\" (UniqueName: \"kubernetes.io/projected/5258dc4d-087f-4815-9135-6a32c5a0d243-kube-api-access-jzwxm\") pod \"payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch\" (UID: \"5258dc4d-087f-4815-9135-6a32c5a0d243\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"
Jul 28 18:41:20.118315 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.118122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5258dc4d-087f-4815-9135-6a32c5a0d243-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch\" (UID: \"5258dc4d-087f-4815-9135-6a32c5a0d243\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"
Jul 28 18:41:20.118315 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.118154    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cefa225-2ae8-4df6-8839-2a50e37ab2ca-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5\" (UID: \"1cefa225-2ae8-4df6-8839-2a50e37ab2ca\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"
Jul 28 18:41:20.118315 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.118193    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gts96\" (UniqueName: \"kubernetes.io/projected/1cefa225-2ae8-4df6-8839-2a50e37ab2ca-kube-api-access-gts96\") pod \"payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5\" (UID: \"1cefa225-2ae8-4df6-8839-2a50e37ab2ca\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"
Jul 28 18:41:20.119070 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.119046    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5258dc4d-087f-4815-9135-6a32c5a0d243-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch\" (UID: \"5258dc4d-087f-4815-9135-6a32c5a0d243\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"
Jul 28 18:41:20.119124 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.119046    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cefa225-2ae8-4df6-8839-2a50e37ab2ca-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5\" (UID: \"1cefa225-2ae8-4df6-8839-2a50e37ab2ca\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"
Jul 28 18:41:20.128550 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.128526    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzwxm\" (UniqueName: \"kubernetes.io/projected/5258dc4d-087f-4815-9135-6a32c5a0d243-kube-api-access-jzwxm\") pod \"payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch\" (UID: \"5258dc4d-087f-4815-9135-6a32c5a0d243\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"
Jul 28 18:41:20.130255 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.130231    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gts96\" (UniqueName: \"kubernetes.io/projected/1cefa225-2ae8-4df6-8839-2a50e37ab2ca-kube-api-access-gts96\") pod \"payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5\" (UID: \"1cefa225-2ae8-4df6-8839-2a50e37ab2ca\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"
Jul 28 18:41:20.266088 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.266041    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"
Jul 28 18:41:20.279161 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.279136    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"
Jul 28 18:41:20.303461 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.303431    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"
Jul 28 18:41:20.419373 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.418799    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"]
Jul 28 18:41:20.423524 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:41:20.423469    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5258dc4d_087f_4815_9135_6a32c5a0d243.slice/crio-f8e2c40f773c2d517a7c74a35b66556f8bde27f9f71a2902da558968f889392d WatchSource:0}: Error finding container f8e2c40f773c2d517a7c74a35b66556f8bde27f9f71a2902da558968f889392d: Status 404 returned error can't find the container with id f8e2c40f773c2d517a7c74a35b66556f8bde27f9f71a2902da558968f889392d
Jul 28 18:41:20.451727 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.451489    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"]
Jul 28 18:41:20.460024 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:41:20.459995    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8936dc_b093_48b1_8dad_6ee4e3b0f980.slice/crio-1e2cc609de163b38580d6b59035c1cb65af47091da9cc3cef0e38d4edf88a04a WatchSource:0}: Error finding container 1e2cc609de163b38580d6b59035c1cb65af47091da9cc3cef0e38d4edf88a04a: Status 404 returned error can't find the container with id 1e2cc609de163b38580d6b59035c1cb65af47091da9cc3cef0e38d4edf88a04a
Jul 28 18:41:20.476191 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:20.476167    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"]
Jul 28 18:41:20.478645 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:41:20.478616    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cefa225_2ae8_4df6_8839_2a50e37ab2ca.slice/crio-31bed28913c21d784b7110a89284191b553e5fe372e93caeba4f5922630038bb WatchSource:0}: Error finding container 31bed28913c21d784b7110a89284191b553e5fe372e93caeba4f5922630038bb: Status 404 returned error can't find the container with id 31bed28913c21d784b7110a89284191b553e5fe372e93caeba4f5922630038bb
Jul 28 18:41:21.403134 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:21.403096    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch" event={"ID":"5258dc4d-087f-4815-9135-6a32c5a0d243","Type":"ContainerStarted","Data":"8fed3680b825f50a6d671ad5fd7ee7c7f52093dcd850c99e9d6baba57c2437c1"}
Jul 28 18:41:21.403348 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:21.403160    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch" event={"ID":"5258dc4d-087f-4815-9135-6a32c5a0d243","Type":"ContainerStarted","Data":"f8e2c40f773c2d517a7c74a35b66556f8bde27f9f71a2902da558968f889392d"}
Jul 28 18:41:21.403348 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:21.403217    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"
Jul 28 18:41:21.405000 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:21.404972    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5" event={"ID":"1cefa225-2ae8-4df6-8839-2a50e37ab2ca","Type":"ContainerStarted","Data":"c6a299889fa54bc61d0e53dadc83e249b6556151a7c30b220d5bbc2f20b94059"}
Jul 28 18:41:21.405129 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:21.405020    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5" event={"ID":"1cefa225-2ae8-4df6-8839-2a50e37ab2ca","Type":"ContainerStarted","Data":"31bed28913c21d784b7110a89284191b553e5fe372e93caeba4f5922630038bb"}
Jul 28 18:41:21.405129 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:21.405048    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"
Jul 28 18:41:21.406553 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:21.406487    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf" event={"ID":"da8936dc-b093-48b1-8dad-6ee4e3b0f980","Type":"ContainerStarted","Data":"0ffb1d194488b01e4a3ecbdbc26e8e3b7dbefec137726c4e537a058596b96e11"}
Jul 28 18:41:21.406657 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:21.406569    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf" event={"ID":"da8936dc-b093-48b1-8dad-6ee4e3b0f980","Type":"ContainerStarted","Data":"1e2cc609de163b38580d6b59035c1cb65af47091da9cc3cef0e38d4edf88a04a"}
Jul 28 18:41:21.406657 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:21.406627    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"
Jul 28 18:41:21.421102 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:21.421048    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch" podStartSLOduration=2.42103351 podStartE2EDuration="2.42103351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:41:21.419266388 +0000 UTC m=+2416.734768535" watchObservedRunningTime="2026-07-28 18:41:21.42103351 +0000 UTC m=+2416.736535971"
Jul 28 18:41:21.436260 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:21.436219    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5" podStartSLOduration=2.436204713 podStartE2EDuration="2.436204713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:41:21.43474066 +0000 UTC m=+2416.750242773" watchObservedRunningTime="2026-07-28 18:41:21.436204713 +0000 UTC m=+2416.751706825"
Jul 28 18:41:21.451160 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:21.451115    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf" podStartSLOduration=2.134295913 podStartE2EDuration="2.451100645s" totalImagesPullingTime="316.804732ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:41:21.450860506 +0000 UTC m=+2416.766362620" watchObservedRunningTime="2026-07-28 18:41:21.451100645 +0000 UTC m=+2416.766602757"
Jul 28 18:41:27.417704 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:27.417627    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"
Jul 28 18:41:32.414103 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:32.414075    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"
Jul 28 18:41:32.414654 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:41:32.414144    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"
Jul 28 18:42:02.461095 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.461062    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"]
Jul 28 18:42:02.461688 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.461339    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb" podUID="0e97caab-2dfa-4a67-9325-585f1732da7d" containerName="maas-api" containerID="cri-o://35a21bec3bba27c74a90d3d9eae7b6cafe107ab9778fafe0604365421653abee" gracePeriod=30
Jul 28 18:42:02.506690 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.506658    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"]
Jul 28 18:42:02.506970 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.506917    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5" podUID="bf78a7bf-821a-425b-85d6-d812cdb23058" containerName="payload-processing" containerID="cri-o://bbd721f0dc14d7111d1c2924bda1b1a40f84f9db5fd62bbab6e1dc4ee127318f" gracePeriod=30
Jul 28 18:42:02.511654 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.511596    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"]
Jul 28 18:42:02.512024 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.511821    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg" podUID="17139879-f19c-4b2c-abbc-31d59f003fb9" containerName="payload-pre-processing" containerID="cri-o://c1948e2b4ea381381f8a09006d8958e3fd0bfd0e646de342d3dc55a37cb1cf32" gracePeriod=30
Jul 28 18:42:02.596924 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.596899    2564 generic.go:356] "Generic (PLEG): container finished" podID="0e97caab-2dfa-4a67-9325-585f1732da7d" containerID="35a21bec3bba27c74a90d3d9eae7b6cafe107ab9778fafe0604365421653abee" exitCode=0
Jul 28 18:42:02.597028 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.596957    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb" event={"ID":"0e97caab-2dfa-4a67-9325-585f1732da7d","Type":"ContainerDied","Data":"35a21bec3bba27c74a90d3d9eae7b6cafe107ab9778fafe0604365421653abee"}
Jul 28 18:42:02.752096 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.752071    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"
Jul 28 18:42:02.787580 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.787544    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"
Jul 28 18:42:02.787717 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.787584    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6rcz\" (UniqueName: \"kubernetes.io/projected/0e97caab-2dfa-4a67-9325-585f1732da7d-kube-api-access-p6rcz\") pod \"0e97caab-2dfa-4a67-9325-585f1732da7d\" (UID: \"0e97caab-2dfa-4a67-9325-585f1732da7d\") "
Jul 28 18:42:02.787717 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.787652    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e97caab-2dfa-4a67-9325-585f1732da7d-maas-api-tls\") pod \"0e97caab-2dfa-4a67-9325-585f1732da7d\" (UID: \"0e97caab-2dfa-4a67-9325-585f1732da7d\") "
Jul 28 18:42:02.787887 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.787718    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e97caab-2dfa-4a67-9325-585f1732da7d-openshift-service-ca\") pod \"0e97caab-2dfa-4a67-9325-585f1732da7d\" (UID: \"0e97caab-2dfa-4a67-9325-585f1732da7d\") "
Jul 28 18:42:02.788196 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.788159    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e97caab-2dfa-4a67-9325-585f1732da7d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e97caab-2dfa-4a67-9325-585f1732da7d" (UID: "0e97caab-2dfa-4a67-9325-585f1732da7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:42:02.789884 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.789860    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e97caab-2dfa-4a67-9325-585f1732da7d-kube-api-access-p6rcz" (OuterVolumeSpecName: "kube-api-access-p6rcz") pod "0e97caab-2dfa-4a67-9325-585f1732da7d" (UID: "0e97caab-2dfa-4a67-9325-585f1732da7d"). InnerVolumeSpecName "kube-api-access-p6rcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:42:02.789884 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.789865    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e97caab-2dfa-4a67-9325-585f1732da7d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e97caab-2dfa-4a67-9325-585f1732da7d" (UID: "0e97caab-2dfa-4a67-9325-585f1732da7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:42:02.791171 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.791153    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"
Jul 28 18:42:02.888954 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.888920    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17139879-f19c-4b2c-abbc-31d59f003fb9-plugins-config-volume\") pod \"17139879-f19c-4b2c-abbc-31d59f003fb9\" (UID: \"17139879-f19c-4b2c-abbc-31d59f003fb9\") "
Jul 28 18:42:02.888954 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.888954    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bcc7\" (UniqueName: \"kubernetes.io/projected/17139879-f19c-4b2c-abbc-31d59f003fb9-kube-api-access-8bcc7\") pod \"17139879-f19c-4b2c-abbc-31d59f003fb9\" (UID: \"17139879-f19c-4b2c-abbc-31d59f003fb9\") "
Jul 28 18:42:02.889215 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.889006    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x66bp\" (UniqueName: \"kubernetes.io/projected/bf78a7bf-821a-425b-85d6-d812cdb23058-kube-api-access-x66bp\") pod \"bf78a7bf-821a-425b-85d6-d812cdb23058\" (UID: \"bf78a7bf-821a-425b-85d6-d812cdb23058\") "
Jul 28 18:42:02.889215 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.889082    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf78a7bf-821a-425b-85d6-d812cdb23058-plugins-config-volume\") pod \"bf78a7bf-821a-425b-85d6-d812cdb23058\" (UID: \"bf78a7bf-821a-425b-85d6-d812cdb23058\") "
Jul 28 18:42:02.889404 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.889358    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17139879-f19c-4b2c-abbc-31d59f003fb9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17139879-f19c-4b2c-abbc-31d59f003fb9" (UID: "17139879-f19c-4b2c-abbc-31d59f003fb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:42:02.889547 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.889422    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6rcz\" (UniqueName: \"kubernetes.io/projected/0e97caab-2dfa-4a67-9325-585f1732da7d-kube-api-access-p6rcz\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:42:02.889547 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.889430    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf78a7bf-821a-425b-85d6-d812cdb23058-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf78a7bf-821a-425b-85d6-d812cdb23058" (UID: "bf78a7bf-821a-425b-85d6-d812cdb23058"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:42:02.889547 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.889437    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e97caab-2dfa-4a67-9325-585f1732da7d-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:42:02.889547 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.889468    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e97caab-2dfa-4a67-9325-585f1732da7d-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:42:02.891091 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.891071    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf78a7bf-821a-425b-85d6-d812cdb23058-kube-api-access-x66bp" (OuterVolumeSpecName: "kube-api-access-x66bp") pod "bf78a7bf-821a-425b-85d6-d812cdb23058" (UID: "bf78a7bf-821a-425b-85d6-d812cdb23058"). InnerVolumeSpecName "kube-api-access-x66bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:42:02.891247 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.891225    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17139879-f19c-4b2c-abbc-31d59f003fb9-kube-api-access-8bcc7" (OuterVolumeSpecName: "kube-api-access-8bcc7") pod "17139879-f19c-4b2c-abbc-31d59f003fb9" (UID: "17139879-f19c-4b2c-abbc-31d59f003fb9"). InnerVolumeSpecName "kube-api-access-8bcc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:42:02.990371 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.990287    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf78a7bf-821a-425b-85d6-d812cdb23058-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:42:02.990371 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.990318    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17139879-f19c-4b2c-abbc-31d59f003fb9-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:42:02.990371 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.990333    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bcc7\" (UniqueName: \"kubernetes.io/projected/17139879-f19c-4b2c-abbc-31d59f003fb9-kube-api-access-8bcc7\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:42:02.990371 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:02.990342    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x66bp\" (UniqueName: \"kubernetes.io/projected/bf78a7bf-821a-425b-85d6-d812cdb23058-kube-api-access-x66bp\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:42:03.604740 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.604707    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"
Jul 28 18:42:03.605222 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.604707    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb" event={"ID":"0e97caab-2dfa-4a67-9325-585f1732da7d","Type":"ContainerDied","Data":"1adc712461b58af2e36968796e43080e09fffa9f15500bccd6e4cdc0497ce2e0"}
Jul 28 18:42:03.605222 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.604886    2564 scope.go:117] "RemoveContainer" containerID="35a21bec3bba27c74a90d3d9eae7b6cafe107ab9778fafe0604365421653abee"
Jul 28 18:42:03.606446 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.606222    2564 generic.go:356] "Generic (PLEG): container finished" podID="bf78a7bf-821a-425b-85d6-d812cdb23058" containerID="bbd721f0dc14d7111d1c2924bda1b1a40f84f9db5fd62bbab6e1dc4ee127318f" exitCode=0
Jul 28 18:42:03.606446 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.606273    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"
Jul 28 18:42:03.606446 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.606305    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5" event={"ID":"bf78a7bf-821a-425b-85d6-d812cdb23058","Type":"ContainerDied","Data":"bbd721f0dc14d7111d1c2924bda1b1a40f84f9db5fd62bbab6e1dc4ee127318f"}
Jul 28 18:42:03.606446 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.606351    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5" event={"ID":"bf78a7bf-821a-425b-85d6-d812cdb23058","Type":"ContainerDied","Data":"20a279f46f1ca1e35c6f0eb7a20251d3ac265e2f5b389986be9e335a2601fe65"}
Jul 28 18:42:03.608077 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.608045    2564 generic.go:356] "Generic (PLEG): container finished" podID="17139879-f19c-4b2c-abbc-31d59f003fb9" containerID="c1948e2b4ea381381f8a09006d8958e3fd0bfd0e646de342d3dc55a37cb1cf32" exitCode=0
Jul 28 18:42:03.608237 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.608088    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg" event={"ID":"17139879-f19c-4b2c-abbc-31d59f003fb9","Type":"ContainerDied","Data":"c1948e2b4ea381381f8a09006d8958e3fd0bfd0e646de342d3dc55a37cb1cf32"}
Jul 28 18:42:03.608237 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.608093    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"
Jul 28 18:42:03.608237 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.608122    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg" event={"ID":"17139879-f19c-4b2c-abbc-31d59f003fb9","Type":"ContainerDied","Data":"a485cedcb6bdc342344b14fbf77733d7504e20e2a3dce5f45178cb37a4efab60"}
Jul 28 18:42:03.614770 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.614739    2564 scope.go:117] "RemoveContainer" containerID="bbd721f0dc14d7111d1c2924bda1b1a40f84f9db5fd62bbab6e1dc4ee127318f"
Jul 28 18:42:03.622788 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.622767    2564 scope.go:117] "RemoveContainer" containerID="bbd721f0dc14d7111d1c2924bda1b1a40f84f9db5fd62bbab6e1dc4ee127318f"
Jul 28 18:42:03.623163 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:42:03.623133    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd721f0dc14d7111d1c2924bda1b1a40f84f9db5fd62bbab6e1dc4ee127318f\": container with ID starting with bbd721f0dc14d7111d1c2924bda1b1a40f84f9db5fd62bbab6e1dc4ee127318f not found: ID does not exist" containerID="bbd721f0dc14d7111d1c2924bda1b1a40f84f9db5fd62bbab6e1dc4ee127318f"
Jul 28 18:42:03.623273 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.623172    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd721f0dc14d7111d1c2924bda1b1a40f84f9db5fd62bbab6e1dc4ee127318f"} err="failed to get container status \"bbd721f0dc14d7111d1c2924bda1b1a40f84f9db5fd62bbab6e1dc4ee127318f\": rpc error: code = NotFound desc = could not find container \"bbd721f0dc14d7111d1c2924bda1b1a40f84f9db5fd62bbab6e1dc4ee127318f\": container with ID starting with bbd721f0dc14d7111d1c2924bda1b1a40f84f9db5fd62bbab6e1dc4ee127318f not found: ID does not exist"
Jul 28 18:42:03.623273 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.623195    2564 scope.go:117] "RemoveContainer" containerID="c1948e2b4ea381381f8a09006d8958e3fd0bfd0e646de342d3dc55a37cb1cf32"
Jul 28 18:42:03.626053 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.626033    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"]
Jul 28 18:42:03.628759 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.628741    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac88b3-b85d4d758-xqfbb"]
Jul 28 18:42:03.631612 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.631597    2564 scope.go:117] "RemoveContainer" containerID="c1948e2b4ea381381f8a09006d8958e3fd0bfd0e646de342d3dc55a37cb1cf32"
Jul 28 18:42:03.631885 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:42:03.631867    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1948e2b4ea381381f8a09006d8958e3fd0bfd0e646de342d3dc55a37cb1cf32\": container with ID starting with c1948e2b4ea381381f8a09006d8958e3fd0bfd0e646de342d3dc55a37cb1cf32 not found: ID does not exist" containerID="c1948e2b4ea381381f8a09006d8958e3fd0bfd0e646de342d3dc55a37cb1cf32"
Jul 28 18:42:03.631951 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.631893    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1948e2b4ea381381f8a09006d8958e3fd0bfd0e646de342d3dc55a37cb1cf32"} err="failed to get container status \"c1948e2b4ea381381f8a09006d8958e3fd0bfd0e646de342d3dc55a37cb1cf32\": rpc error: code = NotFound desc = could not find container \"c1948e2b4ea381381f8a09006d8958e3fd0bfd0e646de342d3dc55a37cb1cf32\": container with ID starting with c1948e2b4ea381381f8a09006d8958e3fd0bfd0e646de342d3dc55a37cb1cf32 not found: ID does not exist"
Jul 28 18:42:03.637850 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.637828    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"]
Jul 28 18:42:03.640970 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.640951    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac88b3-7456867484-8xmtg"]
Jul 28 18:42:03.650124 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.650103    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"]
Jul 28 18:42:03.657984 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:03.653970    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac88b3-55454f88bb-khhr5"]
Jul 28 18:42:05.215978 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:05.215949    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e97caab-2dfa-4a67-9325-585f1732da7d" path="/var/lib/kubelet/pods/0e97caab-2dfa-4a67-9325-585f1732da7d/volumes"
Jul 28 18:42:05.216338 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:05.216294    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17139879-f19c-4b2c-abbc-31d59f003fb9" path="/var/lib/kubelet/pods/17139879-f19c-4b2c-abbc-31d59f003fb9/volumes"
Jul 28 18:42:05.216607 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:05.216595    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf78a7bf-821a-425b-85d6-d812cdb23058" path="/var/lib/kubelet/pods/bf78a7bf-821a-425b-85d6-d812cdb23058/volumes"
Jul 28 18:42:25.098582 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.098541    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"]
Jul 28 18:42:25.099180 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.098842    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf" podUID="da8936dc-b093-48b1-8dad-6ee4e3b0f980" containerName="maas-api" containerID="cri-o://0ffb1d194488b01e4a3ecbdbc26e8e3b7dbefec137726c4e537a058596b96e11" gracePeriod=30
Jul 28 18:42:25.146506 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.146458    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"]
Jul 28 18:42:25.146786 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.146736    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5" podUID="1cefa225-2ae8-4df6-8839-2a50e37ab2ca" containerName="payload-processing" containerID="cri-o://c6a299889fa54bc61d0e53dadc83e249b6556151a7c30b220d5bbc2f20b94059" gracePeriod=30
Jul 28 18:42:25.152570 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.152542    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"]
Jul 28 18:42:25.152832 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.152760    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch" podUID="5258dc4d-087f-4815-9135-6a32c5a0d243" containerName="payload-pre-processing" containerID="cri-o://8fed3680b825f50a6d671ad5fd7ee7c7f52093dcd850c99e9d6baba57c2437c1" gracePeriod=30
Jul 28 18:42:25.417648 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.417618    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"
Jul 28 18:42:25.422497 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.422478    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"
Jul 28 18:42:25.431230 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.431212    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"
Jul 28 18:42:25.480520 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.480484    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cefa225-2ae8-4df6-8839-2a50e37ab2ca-plugins-config-volume\") pod \"1cefa225-2ae8-4df6-8839-2a50e37ab2ca\" (UID: \"1cefa225-2ae8-4df6-8839-2a50e37ab2ca\") "
Jul 28 18:42:25.480694 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.480534    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8936dc-b093-48b1-8dad-6ee4e3b0f980-openshift-service-ca\") pod \"da8936dc-b093-48b1-8dad-6ee4e3b0f980\" (UID: \"da8936dc-b093-48b1-8dad-6ee4e3b0f980\") "
Jul 28 18:42:25.480694 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.480587    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvtfj\" (UniqueName: \"kubernetes.io/projected/da8936dc-b093-48b1-8dad-6ee4e3b0f980-kube-api-access-pvtfj\") pod \"da8936dc-b093-48b1-8dad-6ee4e3b0f980\" (UID: \"da8936dc-b093-48b1-8dad-6ee4e3b0f980\") "
Jul 28 18:42:25.480694 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.480626    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8936dc-b093-48b1-8dad-6ee4e3b0f980-maas-api-tls\") pod \"da8936dc-b093-48b1-8dad-6ee4e3b0f980\" (UID: \"da8936dc-b093-48b1-8dad-6ee4e3b0f980\") "
Jul 28 18:42:25.480694 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.480662    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gts96\" (UniqueName: \"kubernetes.io/projected/1cefa225-2ae8-4df6-8839-2a50e37ab2ca-kube-api-access-gts96\") pod \"1cefa225-2ae8-4df6-8839-2a50e37ab2ca\" (UID: \"1cefa225-2ae8-4df6-8839-2a50e37ab2ca\") "
Jul 28 18:42:25.481032 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.480993    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8936dc-b093-48b1-8dad-6ee4e3b0f980-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "da8936dc-b093-48b1-8dad-6ee4e3b0f980" (UID: "da8936dc-b093-48b1-8dad-6ee4e3b0f980"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:42:25.481032 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.481009    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cefa225-2ae8-4df6-8839-2a50e37ab2ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1cefa225-2ae8-4df6-8839-2a50e37ab2ca" (UID: "1cefa225-2ae8-4df6-8839-2a50e37ab2ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:42:25.482844 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.482794    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da8936dc-b093-48b1-8dad-6ee4e3b0f980-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "da8936dc-b093-48b1-8dad-6ee4e3b0f980" (UID: "da8936dc-b093-48b1-8dad-6ee4e3b0f980"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:42:25.482967 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.482887    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8936dc-b093-48b1-8dad-6ee4e3b0f980-kube-api-access-pvtfj" (OuterVolumeSpecName: "kube-api-access-pvtfj") pod "da8936dc-b093-48b1-8dad-6ee4e3b0f980" (UID: "da8936dc-b093-48b1-8dad-6ee4e3b0f980"). InnerVolumeSpecName "kube-api-access-pvtfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:42:25.482967 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.482932    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cefa225-2ae8-4df6-8839-2a50e37ab2ca-kube-api-access-gts96" (OuterVolumeSpecName: "kube-api-access-gts96") pod "1cefa225-2ae8-4df6-8839-2a50e37ab2ca" (UID: "1cefa225-2ae8-4df6-8839-2a50e37ab2ca"). InnerVolumeSpecName "kube-api-access-gts96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:42:25.581214 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.581175    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5258dc4d-087f-4815-9135-6a32c5a0d243-plugins-config-volume\") pod \"5258dc4d-087f-4815-9135-6a32c5a0d243\" (UID: \"5258dc4d-087f-4815-9135-6a32c5a0d243\") "
Jul 28 18:42:25.581388 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.581231    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzwxm\" (UniqueName: \"kubernetes.io/projected/5258dc4d-087f-4815-9135-6a32c5a0d243-kube-api-access-jzwxm\") pod \"5258dc4d-087f-4815-9135-6a32c5a0d243\" (UID: \"5258dc4d-087f-4815-9135-6a32c5a0d243\") "
Jul 28 18:42:25.581469 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.581383    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8936dc-b093-48b1-8dad-6ee4e3b0f980-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:42:25.581469 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.581403    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvtfj\" (UniqueName: \"kubernetes.io/projected/da8936dc-b093-48b1-8dad-6ee4e3b0f980-kube-api-access-pvtfj\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:42:25.581469 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.581417    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8936dc-b093-48b1-8dad-6ee4e3b0f980-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:42:25.581469 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.581429    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gts96\" (UniqueName: \"kubernetes.io/projected/1cefa225-2ae8-4df6-8839-2a50e37ab2ca-kube-api-access-gts96\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:42:25.581469 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.581441    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cefa225-2ae8-4df6-8839-2a50e37ab2ca-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:42:25.581683 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.581532    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5258dc4d-087f-4815-9135-6a32c5a0d243-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5258dc4d-087f-4815-9135-6a32c5a0d243" (UID: "5258dc4d-087f-4815-9135-6a32c5a0d243"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:42:25.583310 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.583284    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5258dc4d-087f-4815-9135-6a32c5a0d243-kube-api-access-jzwxm" (OuterVolumeSpecName: "kube-api-access-jzwxm") pod "5258dc4d-087f-4815-9135-6a32c5a0d243" (UID: "5258dc4d-087f-4815-9135-6a32c5a0d243"). InnerVolumeSpecName "kube-api-access-jzwxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:42:25.682734 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.682631    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5258dc4d-087f-4815-9135-6a32c5a0d243-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:42:25.682734 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.682673    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzwxm\" (UniqueName: \"kubernetes.io/projected/5258dc4d-087f-4815-9135-6a32c5a0d243-kube-api-access-jzwxm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:42:25.709046 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.709013    2564 generic.go:356] "Generic (PLEG): container finished" podID="5258dc4d-087f-4815-9135-6a32c5a0d243" containerID="8fed3680b825f50a6d671ad5fd7ee7c7f52093dcd850c99e9d6baba57c2437c1" exitCode=0
Jul 28 18:42:25.709179 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.709067    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"
Jul 28 18:42:25.709179 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.709106    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch" event={"ID":"5258dc4d-087f-4815-9135-6a32c5a0d243","Type":"ContainerDied","Data":"8fed3680b825f50a6d671ad5fd7ee7c7f52093dcd850c99e9d6baba57c2437c1"}
Jul 28 18:42:25.709179 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.709163    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch" event={"ID":"5258dc4d-087f-4815-9135-6a32c5a0d243","Type":"ContainerDied","Data":"f8e2c40f773c2d517a7c74a35b66556f8bde27f9f71a2902da558968f889392d"}
Jul 28 18:42:25.709327 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.709186    2564 scope.go:117] "RemoveContainer" containerID="8fed3680b825f50a6d671ad5fd7ee7c7f52093dcd850c99e9d6baba57c2437c1"
Jul 28 18:42:25.710422 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.710403    2564 generic.go:356] "Generic (PLEG): container finished" podID="1cefa225-2ae8-4df6-8839-2a50e37ab2ca" containerID="c6a299889fa54bc61d0e53dadc83e249b6556151a7c30b220d5bbc2f20b94059" exitCode=0
Jul 28 18:42:25.710511 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.710496    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"
Jul 28 18:42:25.710571 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.710510    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5" event={"ID":"1cefa225-2ae8-4df6-8839-2a50e37ab2ca","Type":"ContainerDied","Data":"c6a299889fa54bc61d0e53dadc83e249b6556151a7c30b220d5bbc2f20b94059"}
Jul 28 18:42:25.710571 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.710548    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5" event={"ID":"1cefa225-2ae8-4df6-8839-2a50e37ab2ca","Type":"ContainerDied","Data":"31bed28913c21d784b7110a89284191b553e5fe372e93caeba4f5922630038bb"}
Jul 28 18:42:25.711961 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.711937    2564 generic.go:356] "Generic (PLEG): container finished" podID="da8936dc-b093-48b1-8dad-6ee4e3b0f980" containerID="0ffb1d194488b01e4a3ecbdbc26e8e3b7dbefec137726c4e537a058596b96e11" exitCode=0
Jul 28 18:42:25.712081 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.711972    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"
Jul 28 18:42:25.712081 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.712014    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf" event={"ID":"da8936dc-b093-48b1-8dad-6ee4e3b0f980","Type":"ContainerDied","Data":"0ffb1d194488b01e4a3ecbdbc26e8e3b7dbefec137726c4e537a058596b96e11"}
Jul 28 18:42:25.712081 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.712073    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf" event={"ID":"da8936dc-b093-48b1-8dad-6ee4e3b0f980","Type":"ContainerDied","Data":"1e2cc609de163b38580d6b59035c1cb65af47091da9cc3cef0e38d4edf88a04a"}
Jul 28 18:42:25.719367 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.719350    2564 scope.go:117] "RemoveContainer" containerID="8fed3680b825f50a6d671ad5fd7ee7c7f52093dcd850c99e9d6baba57c2437c1"
Jul 28 18:42:25.719665 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:42:25.719644    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fed3680b825f50a6d671ad5fd7ee7c7f52093dcd850c99e9d6baba57c2437c1\": container with ID starting with 8fed3680b825f50a6d671ad5fd7ee7c7f52093dcd850c99e9d6baba57c2437c1 not found: ID does not exist" containerID="8fed3680b825f50a6d671ad5fd7ee7c7f52093dcd850c99e9d6baba57c2437c1"
Jul 28 18:42:25.719761 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.719671    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fed3680b825f50a6d671ad5fd7ee7c7f52093dcd850c99e9d6baba57c2437c1"} err="failed to get container status \"8fed3680b825f50a6d671ad5fd7ee7c7f52093dcd850c99e9d6baba57c2437c1\": rpc error: code = NotFound desc = could not find container \"8fed3680b825f50a6d671ad5fd7ee7c7f52093dcd850c99e9d6baba57c2437c1\": container with ID starting with 8fed3680b825f50a6d671ad5fd7ee7c7f52093dcd850c99e9d6baba57c2437c1 not found: ID does not exist"
Jul 28 18:42:25.719761 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.719690    2564 scope.go:117] "RemoveContainer" containerID="c6a299889fa54bc61d0e53dadc83e249b6556151a7c30b220d5bbc2f20b94059"
Jul 28 18:42:25.727900 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.727885    2564 scope.go:117] "RemoveContainer" containerID="c6a299889fa54bc61d0e53dadc83e249b6556151a7c30b220d5bbc2f20b94059"
Jul 28 18:42:25.728124 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:42:25.728107    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a299889fa54bc61d0e53dadc83e249b6556151a7c30b220d5bbc2f20b94059\": container with ID starting with c6a299889fa54bc61d0e53dadc83e249b6556151a7c30b220d5bbc2f20b94059 not found: ID does not exist" containerID="c6a299889fa54bc61d0e53dadc83e249b6556151a7c30b220d5bbc2f20b94059"
Jul 28 18:42:25.728170 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.728130    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a299889fa54bc61d0e53dadc83e249b6556151a7c30b220d5bbc2f20b94059"} err="failed to get container status \"c6a299889fa54bc61d0e53dadc83e249b6556151a7c30b220d5bbc2f20b94059\": rpc error: code = NotFound desc = could not find container \"c6a299889fa54bc61d0e53dadc83e249b6556151a7c30b220d5bbc2f20b94059\": container with ID starting with c6a299889fa54bc61d0e53dadc83e249b6556151a7c30b220d5bbc2f20b94059 not found: ID does not exist"
Jul 28 18:42:25.728170 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.728145    2564 scope.go:117] "RemoveContainer" containerID="0ffb1d194488b01e4a3ecbdbc26e8e3b7dbefec137726c4e537a058596b96e11"
Jul 28 18:42:25.736450 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.736437    2564 scope.go:117] "RemoveContainer" containerID="0ffb1d194488b01e4a3ecbdbc26e8e3b7dbefec137726c4e537a058596b96e11"
Jul 28 18:42:25.736689 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:42:25.736667    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ffb1d194488b01e4a3ecbdbc26e8e3b7dbefec137726c4e537a058596b96e11\": container with ID starting with 0ffb1d194488b01e4a3ecbdbc26e8e3b7dbefec137726c4e537a058596b96e11 not found: ID does not exist" containerID="0ffb1d194488b01e4a3ecbdbc26e8e3b7dbefec137726c4e537a058596b96e11"
Jul 28 18:42:25.736758 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.736696    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ffb1d194488b01e4a3ecbdbc26e8e3b7dbefec137726c4e537a058596b96e11"} err="failed to get container status \"0ffb1d194488b01e4a3ecbdbc26e8e3b7dbefec137726c4e537a058596b96e11\": rpc error: code = NotFound desc = could not find container \"0ffb1d194488b01e4a3ecbdbc26e8e3b7dbefec137726c4e537a058596b96e11\": container with ID starting with 0ffb1d194488b01e4a3ecbdbc26e8e3b7dbefec137726c4e537a058596b96e11 not found: ID does not exist"
Jul 28 18:42:25.752339 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.752309    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"]
Jul 28 18:42:25.765582 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.765555    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7f010-76b7cc89bd-nvnch"]
Jul 28 18:42:25.787640 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.787610    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"]
Jul 28 18:42:25.790550 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.790528    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7f010-585796c4cd-kxfqf"]
Jul 28 18:42:25.815186 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.815157    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"]
Jul 28 18:42:25.821279 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:25.821256    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7f010-b789b6458-k2mj5"]
Jul 28 18:42:27.216720 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:27.216685    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cefa225-2ae8-4df6-8839-2a50e37ab2ca" path="/var/lib/kubelet/pods/1cefa225-2ae8-4df6-8839-2a50e37ab2ca/volumes"
Jul 28 18:42:27.217134 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:27.217066    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5258dc4d-087f-4815-9135-6a32c5a0d243" path="/var/lib/kubelet/pods/5258dc4d-087f-4815-9135-6a32c5a0d243/volumes"
Jul 28 18:42:27.217400 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:27.217388    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da8936dc-b093-48b1-8dad-6ee4e3b0f980" path="/var/lib/kubelet/pods/da8936dc-b093-48b1-8dad-6ee4e3b0f980/volumes"
Jul 28 18:42:35.691559 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691512    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"]
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691850    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cefa225-2ae8-4df6-8839-2a50e37ab2ca" containerName="payload-processing"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691864    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cefa225-2ae8-4df6-8839-2a50e37ab2ca" containerName="payload-processing"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691875    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf78a7bf-821a-425b-85d6-d812cdb23058" containerName="payload-processing"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691880    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf78a7bf-821a-425b-85d6-d812cdb23058" containerName="payload-processing"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691887    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da8936dc-b093-48b1-8dad-6ee4e3b0f980" containerName="maas-api"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691892    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="da8936dc-b093-48b1-8dad-6ee4e3b0f980" containerName="maas-api"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691899    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17139879-f19c-4b2c-abbc-31d59f003fb9" containerName="payload-pre-processing"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691906    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="17139879-f19c-4b2c-abbc-31d59f003fb9" containerName="payload-pre-processing"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691914    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5258dc4d-087f-4815-9135-6a32c5a0d243" containerName="payload-pre-processing"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691920    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5258dc4d-087f-4815-9135-6a32c5a0d243" containerName="payload-pre-processing"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691928    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e97caab-2dfa-4a67-9325-585f1732da7d" containerName="maas-api"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691933    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e97caab-2dfa-4a67-9325-585f1732da7d" containerName="maas-api"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691977    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="17139879-f19c-4b2c-abbc-31d59f003fb9" containerName="payload-pre-processing"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691986    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cefa225-2ae8-4df6-8839-2a50e37ab2ca" containerName="payload-processing"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691993    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="da8936dc-b093-48b1-8dad-6ee4e3b0f980" containerName="maas-api"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.691999    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf78a7bf-821a-425b-85d6-d812cdb23058" containerName="payload-processing"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.692005    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e97caab-2dfa-4a67-9325-585f1732da7d" containerName="maas-api"
Jul 28 18:42:35.692052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.692010    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5258dc4d-087f-4815-9135-6a32c5a0d243" containerName="payload-pre-processing"
Jul 28 18:42:35.696143 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.696124    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"
Jul 28 18:42:35.698842 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.698803    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-37e11d-serving-cert\""
Jul 28 18:42:35.698980 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.698884    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 18:42:35.699049 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.699032    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gffgz\""
Jul 28 18:42:35.699138 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.699120    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 18:42:35.703893 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.703345    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"]
Jul 28 18:42:35.870907 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.870871    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f538fb-a15e-4b81-812e-5091d8475ca9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8\" (UID: \"a5f538fb-a15e-4b81-812e-5091d8475ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"
Jul 28 18:42:35.871093 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.870979    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f538fb-a15e-4b81-812e-5091d8475ca9-maas-api-tls\") pod \"maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8\" (UID: \"a5f538fb-a15e-4b81-812e-5091d8475ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"
Jul 28 18:42:35.871093 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.871013    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcv26\" (UniqueName: \"kubernetes.io/projected/a5f538fb-a15e-4b81-812e-5091d8475ca9-kube-api-access-wcv26\") pod \"maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8\" (UID: \"a5f538fb-a15e-4b81-812e-5091d8475ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"
Jul 28 18:42:35.971901 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.971787    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f538fb-a15e-4b81-812e-5091d8475ca9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8\" (UID: \"a5f538fb-a15e-4b81-812e-5091d8475ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"
Jul 28 18:42:35.972058 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.971906    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f538fb-a15e-4b81-812e-5091d8475ca9-maas-api-tls\") pod \"maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8\" (UID: \"a5f538fb-a15e-4b81-812e-5091d8475ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"
Jul 28 18:42:35.972058 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.971933    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcv26\" (UniqueName: \"kubernetes.io/projected/a5f538fb-a15e-4b81-812e-5091d8475ca9-kube-api-access-wcv26\") pod \"maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8\" (UID: \"a5f538fb-a15e-4b81-812e-5091d8475ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"
Jul 28 18:42:35.972131 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:42:35.972057    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-serving-cert: secret "maas-api-e2e-shared-a-37e11d-serving-cert" not found
Jul 28 18:42:35.972168 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:42:35.972135    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5f538fb-a15e-4b81-812e-5091d8475ca9-maas-api-tls podName:a5f538fb-a15e-4b81-812e-5091d8475ca9 nodeName:}" failed. No retries permitted until 2026-07-28 18:42:36.472114857 +0000 UTC m=+2491.787616950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a5f538fb-a15e-4b81-812e-5091d8475ca9-maas-api-tls") pod "maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8" (UID: "a5f538fb-a15e-4b81-812e-5091d8475ca9") : secret "maas-api-e2e-shared-a-37e11d-serving-cert" not found
Jul 28 18:42:35.972563 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.972546    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f538fb-a15e-4b81-812e-5091d8475ca9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8\" (UID: \"a5f538fb-a15e-4b81-812e-5091d8475ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"
Jul 28 18:42:35.983068 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:35.983037    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcv26\" (UniqueName: \"kubernetes.io/projected/a5f538fb-a15e-4b81-812e-5091d8475ca9-kube-api-access-wcv26\") pod \"maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8\" (UID: \"a5f538fb-a15e-4b81-812e-5091d8475ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"
Jul 28 18:42:36.356414 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.356375    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"]
Jul 28 18:42:36.359766 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.359746    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"
Jul 28 18:42:36.362150 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.362132    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-37e11d\""
Jul 28 18:42:36.362150 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.362144    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-37e11d-dockercfg-4snbk\""
Jul 28 18:42:36.369741 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.369716    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"]
Jul 28 18:42:36.387913 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.387882    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"]
Jul 28 18:42:36.391049 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.391032    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"
Jul 28 18:42:36.400732 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.400701    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"]
Jul 28 18:42:36.476315 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.476209    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f538fb-a15e-4b81-812e-5091d8475ca9-maas-api-tls\") pod \"maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8\" (UID: \"a5f538fb-a15e-4b81-812e-5091d8475ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"
Jul 28 18:42:36.476315 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.476262    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68309e6c-2f60-4fb6-ba40-bfda26857446-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9\" (UID: \"68309e6c-2f60-4fb6-ba40-bfda26857446\") " pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"
Jul 28 18:42:36.476315 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.476289    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wxtw\" (UniqueName: \"kubernetes.io/projected/3d1e757b-a627-4946-9184-7e6633fe0716-kube-api-access-4wxtw\") pod \"payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt\" (UID: \"3d1e757b-a627-4946-9184-7e6633fe0716\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"
Jul 28 18:42:36.485476 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.476385    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1e757b-a627-4946-9184-7e6633fe0716-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt\" (UID: \"3d1e757b-a627-4946-9184-7e6633fe0716\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"
Jul 28 18:42:36.485476 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.476441    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsq88\" (UniqueName: \"kubernetes.io/projected/68309e6c-2f60-4fb6-ba40-bfda26857446-kube-api-access-lsq88\") pod \"payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9\" (UID: \"68309e6c-2f60-4fb6-ba40-bfda26857446\") " pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"
Jul 28 18:42:36.485476 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.478769    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f538fb-a15e-4b81-812e-5091d8475ca9-maas-api-tls\") pod \"maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8\" (UID: \"a5f538fb-a15e-4b81-812e-5091d8475ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"
Jul 28 18:42:36.577103 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.577048    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68309e6c-2f60-4fb6-ba40-bfda26857446-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9\" (UID: \"68309e6c-2f60-4fb6-ba40-bfda26857446\") " pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"
Jul 28 18:42:36.577103 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.577088    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wxtw\" (UniqueName: \"kubernetes.io/projected/3d1e757b-a627-4946-9184-7e6633fe0716-kube-api-access-4wxtw\") pod \"payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt\" (UID: \"3d1e757b-a627-4946-9184-7e6633fe0716\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"
Jul 28 18:42:36.577398 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.577112    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1e757b-a627-4946-9184-7e6633fe0716-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt\" (UID: \"3d1e757b-a627-4946-9184-7e6633fe0716\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"
Jul 28 18:42:36.577398 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.577139    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsq88\" (UniqueName: \"kubernetes.io/projected/68309e6c-2f60-4fb6-ba40-bfda26857446-kube-api-access-lsq88\") pod \"payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9\" (UID: \"68309e6c-2f60-4fb6-ba40-bfda26857446\") " pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"
Jul 28 18:42:36.577911 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.577880    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1e757b-a627-4946-9184-7e6633fe0716-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt\" (UID: \"3d1e757b-a627-4946-9184-7e6633fe0716\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"
Jul 28 18:42:36.578017 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.577878    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68309e6c-2f60-4fb6-ba40-bfda26857446-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9\" (UID: \"68309e6c-2f60-4fb6-ba40-bfda26857446\") " pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"
Jul 28 18:42:36.585609 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.585587    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wxtw\" (UniqueName: \"kubernetes.io/projected/3d1e757b-a627-4946-9184-7e6633fe0716-kube-api-access-4wxtw\") pod \"payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt\" (UID: \"3d1e757b-a627-4946-9184-7e6633fe0716\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"
Jul 28 18:42:36.585851 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.585826    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsq88\" (UniqueName: \"kubernetes.io/projected/68309e6c-2f60-4fb6-ba40-bfda26857446-kube-api-access-lsq88\") pod \"payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9\" (UID: \"68309e6c-2f60-4fb6-ba40-bfda26857446\") " pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"
Jul 28 18:42:36.607743 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.607658    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"
Jul 28 18:42:36.670821 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.670770    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"
Jul 28 18:42:36.701476 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.701413    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"
Jul 28 18:42:36.768204 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.767620    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"]
Jul 28 18:42:36.770718 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:42:36.770686    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f538fb_a15e_4b81_812e_5091d8475ca9.slice/crio-b172e3419605a8505abd62c030eba549b80a8f6153e2161d890571e5678a4eca WatchSource:0}: Error finding container b172e3419605a8505abd62c030eba549b80a8f6153e2161d890571e5678a4eca: Status 404 returned error can't find the container with id b172e3419605a8505abd62c030eba549b80a8f6153e2161d890571e5678a4eca
Jul 28 18:42:36.844000 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.843978    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"]
Jul 28 18:42:36.846062 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:42:36.846036    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1e757b_a627_4946_9184_7e6633fe0716.slice/crio-9c446904a3e9503e5509705ede5498abc39c5f5b4e4c263195c14f97a8e14a9e WatchSource:0}: Error finding container 9c446904a3e9503e5509705ede5498abc39c5f5b4e4c263195c14f97a8e14a9e: Status 404 returned error can't find the container with id 9c446904a3e9503e5509705ede5498abc39c5f5b4e4c263195c14f97a8e14a9e
Jul 28 18:42:36.861983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:36.861897    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"]
Jul 28 18:42:36.864200 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:42:36.864178    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68309e6c_2f60_4fb6_ba40_bfda26857446.slice/crio-45f866802c0753b22c1631aafebded2f9863d1106cb67631ac77d56259582878 WatchSource:0}: Error finding container 45f866802c0753b22c1631aafebded2f9863d1106cb67631ac77d56259582878: Status 404 returned error can't find the container with id 45f866802c0753b22c1631aafebded2f9863d1106cb67631ac77d56259582878
Jul 28 18:42:37.767178 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:37.767142    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8" event={"ID":"a5f538fb-a15e-4b81-812e-5091d8475ca9","Type":"ContainerStarted","Data":"d2c6dc71182f8182f5d0993420f0fbd29073eac19d8bb1a9690abd5bd963cd5d"}
Jul 28 18:42:37.767665 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:37.767198    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8" event={"ID":"a5f538fb-a15e-4b81-812e-5091d8475ca9","Type":"ContainerStarted","Data":"b172e3419605a8505abd62c030eba549b80a8f6153e2161d890571e5678a4eca"}
Jul 28 18:42:37.767665 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:37.767248    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"
Jul 28 18:42:37.768611 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:37.768586    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9" event={"ID":"68309e6c-2f60-4fb6-ba40-bfda26857446","Type":"ContainerStarted","Data":"094cfbfd5d0fd124570b0cd7d089f0eae00c5207daad406aa2786cc33a383174"}
Jul 28 18:42:37.768746 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:37.768635    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"
Jul 28 18:42:37.768746 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:37.768658    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9" event={"ID":"68309e6c-2f60-4fb6-ba40-bfda26857446","Type":"ContainerStarted","Data":"45f866802c0753b22c1631aafebded2f9863d1106cb67631ac77d56259582878"}
Jul 28 18:42:37.775391 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:37.775362    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt" event={"ID":"3d1e757b-a627-4946-9184-7e6633fe0716","Type":"ContainerStarted","Data":"d5ebc8b935951cf1b5ace7e282680b8a18230476df9d8af79625785d3eaf8ed4"}
Jul 28 18:42:37.775498 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:37.775449    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"
Jul 28 18:42:37.775498 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:37.775477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt" event={"ID":"3d1e757b-a627-4946-9184-7e6633fe0716","Type":"ContainerStarted","Data":"9c446904a3e9503e5509705ede5498abc39c5f5b4e4c263195c14f97a8e14a9e"}
Jul 28 18:42:37.790131 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:37.790081    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8" podStartSLOduration=2.289732322 podStartE2EDuration="2.790067074s" totalImagesPullingTime="500.334752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:42:37.789996411 +0000 UTC m=+2493.105498537" watchObservedRunningTime="2026-07-28 18:42:37.790067074 +0000 UTC m=+2493.105569188"
Jul 28 18:42:37.806022 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:37.805944    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt" podStartSLOduration=1.8059234979999998 podStartE2EDuration="1.805923498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:42:37.805105407 +0000 UTC m=+2493.120607552" watchObservedRunningTime="2026-07-28 18:42:37.805923498 +0000 UTC m=+2493.121425613"
Jul 28 18:42:37.823201 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:37.823149    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9" podStartSLOduration=1.823132454 podStartE2EDuration="1.823132454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:42:37.821256164 +0000 UTC m=+2493.136758306" watchObservedRunningTime="2026-07-28 18:42:37.823132454 +0000 UTC m=+2493.138634568"
Jul 28 18:42:43.784910 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:43.784882    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"
Jul 28 18:42:48.781609 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:48.781577    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"
Jul 28 18:42:48.782090 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:42:48.781915    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"
Jul 28 18:43:03.933324 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:03.933234    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"]
Jul 28 18:43:03.936901 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:03.936875    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"
Jul 28 18:43:03.939573 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:03.939540    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-f36101-serving-cert\""
Jul 28 18:43:03.948895 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:03.948871    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"]
Jul 28 18:43:04.002375 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.002340    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb97487-cd38-4561-840c-33f153df2164-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8\" (UID: \"7eb97487-cd38-4561-840c-33f153df2164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"
Jul 28 18:43:04.002557 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.002393    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmlnb\" (UniqueName: \"kubernetes.io/projected/7eb97487-cd38-4561-840c-33f153df2164-kube-api-access-kmlnb\") pod \"maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8\" (UID: \"7eb97487-cd38-4561-840c-33f153df2164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"
Jul 28 18:43:04.002557 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.002499    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eb97487-cd38-4561-840c-33f153df2164-maas-api-tls\") pod \"maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8\" (UID: \"7eb97487-cd38-4561-840c-33f153df2164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"
Jul 28 18:43:04.103105 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.103067    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmlnb\" (UniqueName: \"kubernetes.io/projected/7eb97487-cd38-4561-840c-33f153df2164-kube-api-access-kmlnb\") pod \"maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8\" (UID: \"7eb97487-cd38-4561-840c-33f153df2164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"
Jul 28 18:43:04.103278 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.103129    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eb97487-cd38-4561-840c-33f153df2164-maas-api-tls\") pod \"maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8\" (UID: \"7eb97487-cd38-4561-840c-33f153df2164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"
Jul 28 18:43:04.103278 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.103157    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb97487-cd38-4561-840c-33f153df2164-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8\" (UID: \"7eb97487-cd38-4561-840c-33f153df2164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"
Jul 28 18:43:04.103879 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.103859    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb97487-cd38-4561-840c-33f153df2164-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8\" (UID: \"7eb97487-cd38-4561-840c-33f153df2164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"
Jul 28 18:43:04.105574 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.105554    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eb97487-cd38-4561-840c-33f153df2164-maas-api-tls\") pod \"maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8\" (UID: \"7eb97487-cd38-4561-840c-33f153df2164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"
Jul 28 18:43:04.111209 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.111184    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmlnb\" (UniqueName: \"kubernetes.io/projected/7eb97487-cd38-4561-840c-33f153df2164-kube-api-access-kmlnb\") pod \"maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8\" (UID: \"7eb97487-cd38-4561-840c-33f153df2164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"
Jul 28 18:43:04.247836 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.247732    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"
Jul 28 18:43:04.386274 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.386241    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"]
Jul 28 18:43:04.388569 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:43:04.388541    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb97487_cd38_4561_840c_33f153df2164.slice/crio-f60b7951fd9cb8422417728fc1600a385e6d9ff822eabccb1d609a4d6fbab062 WatchSource:0}: Error finding container f60b7951fd9cb8422417728fc1600a385e6d9ff822eabccb1d609a4d6fbab062: Status 404 returned error can't find the container with id f60b7951fd9cb8422417728fc1600a385e6d9ff822eabccb1d609a4d6fbab062
Jul 28 18:43:04.562974 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.562939    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"]
Jul 28 18:43:04.567438 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.567421    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"
Jul 28 18:43:04.570144 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.570121    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-f36101\""
Jul 28 18:43:04.570271 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.570161    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-f36101-dockercfg-6wzkl\""
Jul 28 18:43:04.575756 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.575732    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"]
Jul 28 18:43:04.608179 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.608145    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4bebc7-ab42-44fc-aed7-b4b455c3ea26-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh\" (UID: \"fa4bebc7-ab42-44fc-aed7-b4b455c3ea26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"
Jul 28 18:43:04.608338 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.608192    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb47x\" (UniqueName: \"kubernetes.io/projected/fa4bebc7-ab42-44fc-aed7-b4b455c3ea26-kube-api-access-hb47x\") pod \"payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh\" (UID: \"fa4bebc7-ab42-44fc-aed7-b4b455c3ea26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"
Jul 28 18:43:04.709035 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.708993    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4bebc7-ab42-44fc-aed7-b4b455c3ea26-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh\" (UID: \"fa4bebc7-ab42-44fc-aed7-b4b455c3ea26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"
Jul 28 18:43:04.709248 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.709062    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb47x\" (UniqueName: \"kubernetes.io/projected/fa4bebc7-ab42-44fc-aed7-b4b455c3ea26-kube-api-access-hb47x\") pod \"payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh\" (UID: \"fa4bebc7-ab42-44fc-aed7-b4b455c3ea26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"
Jul 28 18:43:04.709695 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.709674    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4bebc7-ab42-44fc-aed7-b4b455c3ea26-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh\" (UID: \"fa4bebc7-ab42-44fc-aed7-b4b455c3ea26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"
Jul 28 18:43:04.717508 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.717477    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb47x\" (UniqueName: \"kubernetes.io/projected/fa4bebc7-ab42-44fc-aed7-b4b455c3ea26-kube-api-access-hb47x\") pod \"payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh\" (UID: \"fa4bebc7-ab42-44fc-aed7-b4b455c3ea26\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"
Jul 28 18:43:04.879357 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.879316    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"
Jul 28 18:43:04.902237 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.902193    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8" event={"ID":"7eb97487-cd38-4561-840c-33f153df2164","Type":"ContainerStarted","Data":"84ddf526ae1af11558108472e07156f4858b0fc6043725d0dbfbcfbd91ce8ebc"}
Jul 28 18:43:04.902393 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.902291    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8" event={"ID":"7eb97487-cd38-4561-840c-33f153df2164","Type":"ContainerStarted","Data":"f60b7951fd9cb8422417728fc1600a385e6d9ff822eabccb1d609a4d6fbab062"}
Jul 28 18:43:04.902393 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.902322    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"
Jul 28 18:43:04.923407 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:04.923340    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8" podStartSLOduration=1.5560950230000001 podStartE2EDuration="1.92331546s" totalImagesPullingTime="367.220437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:43:04.91967435 +0000 UTC m=+2520.235176489" watchObservedRunningTime="2026-07-28 18:43:04.92331546 +0000 UTC m=+2520.238817573"
Jul 28 18:43:05.016309 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:05.016272    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"]
Jul 28 18:43:05.018395 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:43:05.018368    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4bebc7_ab42_44fc_aed7_b4b455c3ea26.slice/crio-99da9b37394b6f51a42f362a0c793c197742213ea7fe1326b20aa38bac4ecb0d WatchSource:0}: Error finding container 99da9b37394b6f51a42f362a0c793c197742213ea7fe1326b20aa38bac4ecb0d: Status 404 returned error can't find the container with id 99da9b37394b6f51a42f362a0c793c197742213ea7fe1326b20aa38bac4ecb0d
Jul 28 18:43:05.907995 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:05.907959    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh" event={"ID":"fa4bebc7-ab42-44fc-aed7-b4b455c3ea26","Type":"ContainerStarted","Data":"fd5e402554c7dbff9e0035149b5a31b7a66eddc6a1d2de2146ba6ebdf96ce087"}
Jul 28 18:43:05.908201 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:05.908014    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh" event={"ID":"fa4bebc7-ab42-44fc-aed7-b4b455c3ea26","Type":"ContainerStarted","Data":"99da9b37394b6f51a42f362a0c793c197742213ea7fe1326b20aa38bac4ecb0d"}
Jul 28 18:43:05.908307 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:05.908285    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"
Jul 28 18:43:05.925032 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:05.924950    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh" podStartSLOduration=1.924930658 podStartE2EDuration="1.924930658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:43:05.924677582 +0000 UTC m=+2521.240179695" watchObservedRunningTime="2026-07-28 18:43:05.924930658 +0000 UTC m=+2521.240432771"
Jul 28 18:43:10.913939 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:10.913912    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"
Jul 28 18:43:16.913788 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:16.913752    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"
Jul 28 18:43:34.200790 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.200750    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"]
Jul 28 18:43:34.204059 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.204041    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.207963 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.207939    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-37e11d\"/\"kube-root-ca.crt\""
Jul 28 18:43:34.207963 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.207939    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-37e11d\"/\"openshift-service-ca.crt\""
Jul 28 18:43:34.208157 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.207970    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-37e11d\"/\"auth-test-model-37e11d-kserve-self-signed-certs\""
Jul 28 18:43:34.208157 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.207941    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-37e11d\"/\"default-dockercfg-9tw8g\""
Jul 28 18:43:34.213170 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.213144    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"]
Jul 28 18:43:34.365973 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.365939    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-model-cache\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.366174 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.366000    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-tmp-dir\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.366174 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.366030    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-home\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.366174 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.366051    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35b7fe3c-1d0f-47c4-a922-a012467afe12-tls-certs\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.366174 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.366075    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dmwx\" (UniqueName: \"kubernetes.io/projected/35b7fe3c-1d0f-47c4-a922-a012467afe12-kube-api-access-2dmwx\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.366174 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.366156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-dshm\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.467103 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.467012    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-model-cache\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.467103 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.467079    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-tmp-dir\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.467340 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.467110    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-home\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.467340 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.467136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35b7fe3c-1d0f-47c4-a922-a012467afe12-tls-certs\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.467340 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.467202    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dmwx\" (UniqueName: \"kubernetes.io/projected/35b7fe3c-1d0f-47c4-a922-a012467afe12-kube-api-access-2dmwx\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.467340 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.467284    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-dshm\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.467554 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.467472    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-model-cache\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.467554 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.467499    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-home\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.467687 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.467633    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-tmp-dir\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.469433 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.469413    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-dshm\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.469617 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.469599    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35b7fe3c-1d0f-47c4-a922-a012467afe12-tls-certs\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.475572 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.475547    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dmwx\" (UniqueName: \"kubernetes.io/projected/35b7fe3c-1d0f-47c4-a922-a012467afe12-kube-api-access-2dmwx\") pod \"auth-test-model-37e11d-kserve-5864f8944c-vjmsd\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") " pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.513752 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.513720    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:34.646161 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:34.646131    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"]
Jul 28 18:43:34.648612 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:43:34.648586    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35b7fe3c_1d0f_47c4_a922_a012467afe12.slice/crio-83d9a8cf3d37230c6b97088a6424d16c5c8bb5def9fda4eb376ef41184e4f07c WatchSource:0}: Error finding container 83d9a8cf3d37230c6b97088a6424d16c5c8bb5def9fda4eb376ef41184e4f07c: Status 404 returned error can't find the container with id 83d9a8cf3d37230c6b97088a6424d16c5c8bb5def9fda4eb376ef41184e4f07c
Jul 28 18:43:35.035132 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:35.035093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd" event={"ID":"35b7fe3c-1d0f-47c4-a922-a012467afe12","Type":"ContainerStarted","Data":"45aded795022f773d8e4aae8da204de2aa331b78ea1f0f5375633405c1809c9e"}
Jul 28 18:43:35.035354 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:35.035140    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd" event={"ID":"35b7fe3c-1d0f-47c4-a922-a012467afe12","Type":"ContainerStarted","Data":"83d9a8cf3d37230c6b97088a6424d16c5c8bb5def9fda4eb376ef41184e4f07c"}
Jul 28 18:43:35.054265 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:35.054209    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd" podStartSLOduration=1.054189996 podStartE2EDuration="1.054189996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:43:35.052093737 +0000 UTC m=+2550.367595863" watchObservedRunningTime="2026-07-28 18:43:35.054189996 +0000 UTC m=+2550.369692109"
Jul 28 18:43:44.514627 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:44.514583    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:44.515120 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:44.514712    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:44.527529 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:44.527502    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:43:45.090330 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:43:45.090297    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:48:11.425697 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:11.425661    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"]
Jul 28 18:48:26.444073 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.444028    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd" podUID="35b7fe3c-1d0f-47c4-a922-a012467afe12" containerName="main" containerID="cri-o://45aded795022f773d8e4aae8da204de2aa331b78ea1f0f5375633405c1809c9e" gracePeriod=45
Jul 28 18:48:26.694253 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.694186    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:48:26.771081 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.771052    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-model-cache\") pod \"35b7fe3c-1d0f-47c4-a922-a012467afe12\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") "
Jul 28 18:48:26.771271 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.771122    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-dshm\") pod \"35b7fe3c-1d0f-47c4-a922-a012467afe12\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") "
Jul 28 18:48:26.771271 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.771158    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dmwx\" (UniqueName: \"kubernetes.io/projected/35b7fe3c-1d0f-47c4-a922-a012467afe12-kube-api-access-2dmwx\") pod \"35b7fe3c-1d0f-47c4-a922-a012467afe12\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") "
Jul 28 18:48:26.771271 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.771203    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-tmp-dir\") pod \"35b7fe3c-1d0f-47c4-a922-a012467afe12\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") "
Jul 28 18:48:26.771271 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.771230    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-home\") pod \"35b7fe3c-1d0f-47c4-a922-a012467afe12\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") "
Jul 28 18:48:26.771271 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.771253    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35b7fe3c-1d0f-47c4-a922-a012467afe12-tls-certs\") pod \"35b7fe3c-1d0f-47c4-a922-a012467afe12\" (UID: \"35b7fe3c-1d0f-47c4-a922-a012467afe12\") "
Jul 28 18:48:26.771551 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.771349    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-model-cache" (OuterVolumeSpecName: "model-cache") pod "35b7fe3c-1d0f-47c4-a922-a012467afe12" (UID: "35b7fe3c-1d0f-47c4-a922-a012467afe12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:48:26.771621 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.771547    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35b7fe3c-1d0f-47c4-a922-a012467afe12" (UID: "35b7fe3c-1d0f-47c4-a922-a012467afe12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:48:26.771621 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.771568    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-home" (OuterVolumeSpecName: "home") pod "35b7fe3c-1d0f-47c4-a922-a012467afe12" (UID: "35b7fe3c-1d0f-47c4-a922-a012467afe12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:48:26.771723 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.771639    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:48:26.773477 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.773453    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b7fe3c-1d0f-47c4-a922-a012467afe12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35b7fe3c-1d0f-47c4-a922-a012467afe12" (UID: "35b7fe3c-1d0f-47c4-a922-a012467afe12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:48:26.773477 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.773466    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b7fe3c-1d0f-47c4-a922-a012467afe12-kube-api-access-2dmwx" (OuterVolumeSpecName: "kube-api-access-2dmwx") pod "35b7fe3c-1d0f-47c4-a922-a012467afe12" (UID: "35b7fe3c-1d0f-47c4-a922-a012467afe12"). InnerVolumeSpecName "kube-api-access-2dmwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:48:26.773629 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.773464    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-dshm" (OuterVolumeSpecName: "dshm") pod "35b7fe3c-1d0f-47c4-a922-a012467afe12" (UID: "35b7fe3c-1d0f-47c4-a922-a012467afe12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:48:26.872870 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.872828    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:48:26.872870 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.872866    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dmwx\" (UniqueName: \"kubernetes.io/projected/35b7fe3c-1d0f-47c4-a922-a012467afe12-kube-api-access-2dmwx\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:48:26.872870 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.872882    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:48:26.873091 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.872894    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35b7fe3c-1d0f-47c4-a922-a012467afe12-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:48:26.873091 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:26.872902    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35b7fe3c-1d0f-47c4-a922-a012467afe12-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:48:27.345916 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:27.345884    2564 generic.go:356] "Generic (PLEG): container finished" podID="35b7fe3c-1d0f-47c4-a922-a012467afe12" containerID="45aded795022f773d8e4aae8da204de2aa331b78ea1f0f5375633405c1809c9e" exitCode=0
Jul 28 18:48:27.345916 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:27.345921    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd" event={"ID":"35b7fe3c-1d0f-47c4-a922-a012467afe12","Type":"ContainerDied","Data":"45aded795022f773d8e4aae8da204de2aa331b78ea1f0f5375633405c1809c9e"}
Jul 28 18:48:27.346137 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:27.345928    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"
Jul 28 18:48:27.346137 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:27.345947    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd" event={"ID":"35b7fe3c-1d0f-47c4-a922-a012467afe12","Type":"ContainerDied","Data":"83d9a8cf3d37230c6b97088a6424d16c5c8bb5def9fda4eb376ef41184e4f07c"}
Jul 28 18:48:27.346137 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:27.345964    2564 scope.go:117] "RemoveContainer" containerID="45aded795022f773d8e4aae8da204de2aa331b78ea1f0f5375633405c1809c9e"
Jul 28 18:48:27.354819 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:27.354789    2564 scope.go:117] "RemoveContainer" containerID="45aded795022f773d8e4aae8da204de2aa331b78ea1f0f5375633405c1809c9e"
Jul 28 18:48:27.355070 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:48:27.355051    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45aded795022f773d8e4aae8da204de2aa331b78ea1f0f5375633405c1809c9e\": container with ID starting with 45aded795022f773d8e4aae8da204de2aa331b78ea1f0f5375633405c1809c9e not found: ID does not exist" containerID="45aded795022f773d8e4aae8da204de2aa331b78ea1f0f5375633405c1809c9e"
Jul 28 18:48:27.355150 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:27.355080    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45aded795022f773d8e4aae8da204de2aa331b78ea1f0f5375633405c1809c9e"} err="failed to get container status \"45aded795022f773d8e4aae8da204de2aa331b78ea1f0f5375633405c1809c9e\": rpc error: code = NotFound desc = could not find container \"45aded795022f773d8e4aae8da204de2aa331b78ea1f0f5375633405c1809c9e\": container with ID starting with 45aded795022f773d8e4aae8da204de2aa331b78ea1f0f5375633405c1809c9e not found: ID does not exist"
Jul 28 18:48:27.362345 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:27.362320    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"]
Jul 28 18:48:27.365718 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:27.365697    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-37e11d/auth-test-model-37e11d-kserve-5864f8944c-vjmsd"]
Jul 28 18:48:29.218081 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:29.218047    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35b7fe3c-1d0f-47c4-a922-a012467afe12" path="/var/lib/kubelet/pods/35b7fe3c-1d0f-47c4-a922-a012467afe12/volumes"
Jul 28 18:48:43.005064 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.005027    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"]
Jul 28 18:48:43.005531 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.005400    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b7fe3c-1d0f-47c4-a922-a012467afe12" containerName="main"
Jul 28 18:48:43.005531 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.005417    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b7fe3c-1d0f-47c4-a922-a012467afe12" containerName="main"
Jul 28 18:48:43.005531 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.005493    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="35b7fe3c-1d0f-47c4-a922-a012467afe12" containerName="main"
Jul 28 18:48:43.009600 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.009581    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.012155 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.012137    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f36101\"/\"kube-root-ca.crt\""
Jul 28 18:48:43.013517 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.013488    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f36101\"/\"sub-test-model-f36101-kserve-self-signed-certs\""
Jul 28 18:48:43.013644 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.013518    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f36101\"/\"default-dockercfg-kskpl\""
Jul 28 18:48:43.013644 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.013552    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f36101\"/\"openshift-service-ca.crt\""
Jul 28 18:48:43.018527 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.018506    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"]
Jul 28 18:48:43.104215 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.104186    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-tmp-dir\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.104215 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.104224    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf46w\" (UniqueName: \"kubernetes.io/projected/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-kube-api-access-mf46w\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.104458 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.104252    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-dshm\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.104458 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.104318    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-model-cache\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.104458 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.104355    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-tls-certs\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.104458 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.104378    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-home\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.205480 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.205441    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-tls-certs\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.205635 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.205491    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-home\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.205635 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.205609    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-tmp-dir\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.205765 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.205648    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf46w\" (UniqueName: \"kubernetes.io/projected/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-kube-api-access-mf46w\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.205765 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.205694    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-dshm\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.205765 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.205743    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-model-cache\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.205960 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.205854    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-home\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.206016 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.205981    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-tmp-dir\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.206104 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.206089    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-model-cache\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.207833 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.207802    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-dshm\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.208132 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.208114    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-tls-certs\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.214661 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.214640    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf46w\" (UniqueName: \"kubernetes.io/projected/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-kube-api-access-mf46w\") pod \"sub-test-model-f36101-kserve-6665f4484f-4m6kg\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") " pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.319583 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.319534    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:43.453570 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:43.453546    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"]
Jul 28 18:48:43.455766 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:48:43.455735    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8667a51c_8b4b_4a3b_ad4f_0284f10fa1ed.slice/crio-74b836a0823cb52f08ca3907d1cf4b5e0ec9c59bde1345376d133ff59e601683 WatchSource:0}: Error finding container 74b836a0823cb52f08ca3907d1cf4b5e0ec9c59bde1345376d133ff59e601683: Status 404 returned error can't find the container with id 74b836a0823cb52f08ca3907d1cf4b5e0ec9c59bde1345376d133ff59e601683
Jul 28 18:48:44.432185 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:44.432149    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg" event={"ID":"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed","Type":"ContainerStarted","Data":"3b927faf0945730585726e6f3fde9c46284009662d82024bdbf8f46bcee3863e"}
Jul 28 18:48:44.432185 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:44.432188    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg" event={"ID":"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed","Type":"ContainerStarted","Data":"74b836a0823cb52f08ca3907d1cf4b5e0ec9c59bde1345376d133ff59e601683"}
Jul 28 18:48:44.451230 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:44.451171    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg" podStartSLOduration=2.451150989 podStartE2EDuration="2.451150989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:48:44.449929913 +0000 UTC m=+2859.765432047" watchObservedRunningTime="2026-07-28 18:48:44.451150989 +0000 UTC m=+2859.766653103"
Jul 28 18:48:53.320127 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:53.320094    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:53.320513 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:53.320140    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:53.332440 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:53.332415    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:48:53.482909 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:48:53.482875    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:49:52.076327 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:52.076289    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"]
Jul 28 18:49:56.295339 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.295301    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"]
Jul 28 18:49:56.298533 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.298516    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.302407 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.302383    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-37e11d\"/\"default-dockercfg-9tw8g\""
Jul 28 18:49:56.302554 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.302416    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-37e11d\"/\"kube-root-ca.crt\""
Jul 28 18:49:56.302554 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.302466    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-37e11d\"/\"rate-test-model-37e11d-kserve-self-signed-certs\""
Jul 28 18:49:56.302554 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.302492    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-37e11d\"/\"openshift-service-ca.crt\""
Jul 28 18:49:56.308237 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.308215    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"]
Jul 28 18:49:56.409302 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.409274    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-home\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.409444 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.409308    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-model-cache\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.409444 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.409330    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-dshm\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.409444 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.409411    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-tmp-dir\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.409444 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.409443    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4064c411-cb76-4dc8-b715-c0e5f1082ca3-tls-certs\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.409587 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.409485    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxdz6\" (UniqueName: \"kubernetes.io/projected/4064c411-cb76-4dc8-b715-c0e5f1082ca3-kube-api-access-cxdz6\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.510894 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.510870    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxdz6\" (UniqueName: \"kubernetes.io/projected/4064c411-cb76-4dc8-b715-c0e5f1082ca3-kube-api-access-cxdz6\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.511028 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.510935    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-home\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.511028 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.510955    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-model-cache\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.511028 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.510971    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-dshm\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.511028 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.510992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-tmp-dir\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.511028 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.511010    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4064c411-cb76-4dc8-b715-c0e5f1082ca3-tls-certs\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.511399 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.511374    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-model-cache\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.511500 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.511455    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-home\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.511500 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.511472    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-tmp-dir\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.513249 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.513227    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-dshm\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.513643 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.513623    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4064c411-cb76-4dc8-b715-c0e5f1082ca3-tls-certs\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.520295 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.520274    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxdz6\" (UniqueName: \"kubernetes.io/projected/4064c411-cb76-4dc8-b715-c0e5f1082ca3-kube-api-access-cxdz6\") pod \"rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") " pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.608640 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.608572    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:49:56.738168 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.738142    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"]
Jul 28 18:49:56.740651 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:49:56.740625    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4064c411_cb76_4dc8_b715_c0e5f1082ca3.slice/crio-0ce8d8e017118b00478c10e9834f14ceb9061b48a581d66b8d93738d8b957cc2 WatchSource:0}: Error finding container 0ce8d8e017118b00478c10e9834f14ceb9061b48a581d66b8d93738d8b957cc2: Status 404 returned error can't find the container with id 0ce8d8e017118b00478c10e9834f14ceb9061b48a581d66b8d93738d8b957cc2
Jul 28 18:49:56.749451 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:56.749423    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt" event={"ID":"4064c411-cb76-4dc8-b715-c0e5f1082ca3","Type":"ContainerStarted","Data":"0ce8d8e017118b00478c10e9834f14ceb9061b48a581d66b8d93738d8b957cc2"}
Jul 28 18:49:57.755066 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:57.755023    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt" event={"ID":"4064c411-cb76-4dc8-b715-c0e5f1082ca3","Type":"ContainerStarted","Data":"d1c7f6688830a76eadd27f4a615a9a094be3837200f925232103f4374fff7236"}
Jul 28 18:49:57.773519 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:49:57.773470    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt" podStartSLOduration=1.773454411 podStartE2EDuration="1.773454411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:49:57.771164701 +0000 UTC m=+2933.086666819" watchObservedRunningTime="2026-07-28 18:49:57.773454411 +0000 UTC m=+2933.088956523"
Jul 28 18:50:06.609328 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:06.609296    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:50:06.609739 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:06.609339    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:50:06.622443 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:06.622416    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:50:06.806482 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:06.806450    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:50:07.095158 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.095123    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg" podUID="8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed" containerName="main" containerID="cri-o://3b927faf0945730585726e6f3fde9c46284009662d82024bdbf8f46bcee3863e" gracePeriod=45
Jul 28 18:50:07.347030 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.346965    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:50:07.502111 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.502081    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-model-cache\") pod \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") "
Jul 28 18:50:07.502278 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.502124    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-home\") pod \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") "
Jul 28 18:50:07.502278 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.502191    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-tls-certs\") pod \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") "
Jul 28 18:50:07.502278 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.502265    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mf46w\" (UniqueName: \"kubernetes.io/projected/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-kube-api-access-mf46w\") pod \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") "
Jul 28 18:50:07.502422 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.502303    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-tmp-dir\") pod \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") "
Jul 28 18:50:07.502422 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.502330    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-dshm\") pod \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\" (UID: \"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed\") "
Jul 28 18:50:07.502422 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.502391    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed" (UID: "8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:50:07.502422 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.502415    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-home" (OuterVolumeSpecName: "home") pod "8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed" (UID: "8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:50:07.502631 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.502583    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed" (UID: "8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:50:07.502631 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.502597    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:50:07.502631 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.502611    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:50:07.504512 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.504486    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-kube-api-access-mf46w" (OuterVolumeSpecName: "kube-api-access-mf46w") pod "8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed" (UID: "8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed"). InnerVolumeSpecName "kube-api-access-mf46w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:50:07.504608 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.504509    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-dshm" (OuterVolumeSpecName: "dshm") pod "8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed" (UID: "8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:50:07.504608 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.504542    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed" (UID: "8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:50:07.603529 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.603434    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf46w\" (UniqueName: \"kubernetes.io/projected/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-kube-api-access-mf46w\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:50:07.603529 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.603468    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:50:07.603529 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.603477    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:50:07.603529 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.603491    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:50:07.800698 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.800668    2564 generic.go:356] "Generic (PLEG): container finished" podID="8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed" containerID="3b927faf0945730585726e6f3fde9c46284009662d82024bdbf8f46bcee3863e" exitCode=0
Jul 28 18:50:07.801151 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.800724    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"
Jul 28 18:50:07.801151 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.800751    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg" event={"ID":"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed","Type":"ContainerDied","Data":"3b927faf0945730585726e6f3fde9c46284009662d82024bdbf8f46bcee3863e"}
Jul 28 18:50:07.801151 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.800799    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg" event={"ID":"8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed","Type":"ContainerDied","Data":"74b836a0823cb52f08ca3907d1cf4b5e0ec9c59bde1345376d133ff59e601683"}
Jul 28 18:50:07.801151 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.800829    2564 scope.go:117] "RemoveContainer" containerID="3b927faf0945730585726e6f3fde9c46284009662d82024bdbf8f46bcee3863e"
Jul 28 18:50:07.809609 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.809591    2564 scope.go:117] "RemoveContainer" containerID="3b927faf0945730585726e6f3fde9c46284009662d82024bdbf8f46bcee3863e"
Jul 28 18:50:07.809890 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:50:07.809869    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b927faf0945730585726e6f3fde9c46284009662d82024bdbf8f46bcee3863e\": container with ID starting with 3b927faf0945730585726e6f3fde9c46284009662d82024bdbf8f46bcee3863e not found: ID does not exist" containerID="3b927faf0945730585726e6f3fde9c46284009662d82024bdbf8f46bcee3863e"
Jul 28 18:50:07.809960 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.809904    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b927faf0945730585726e6f3fde9c46284009662d82024bdbf8f46bcee3863e"} err="failed to get container status \"3b927faf0945730585726e6f3fde9c46284009662d82024bdbf8f46bcee3863e\": rpc error: code = NotFound desc = could not find container \"3b927faf0945730585726e6f3fde9c46284009662d82024bdbf8f46bcee3863e\": container with ID starting with 3b927faf0945730585726e6f3fde9c46284009662d82024bdbf8f46bcee3863e not found: ID does not exist"
Jul 28 18:50:07.824914 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.824890    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"]
Jul 28 18:50:07.829118 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:07.829092    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f36101/sub-test-model-f36101-kserve-6665f4484f-4m6kg"]
Jul 28 18:50:09.216154 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:50:09.216119    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed" path="/var/lib/kubelet/pods/8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed/volumes"
Jul 28 18:51:28.430728 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:28.430691    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"]
Jul 28 18:51:33.728659 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:33.728622    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7351-140545011630896"]
Jul 28 18:51:33.729068 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:33.728965    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed" containerName="main"
Jul 28 18:51:33.729068 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:33.728978    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed" containerName="main"
Jul 28 18:51:33.729068 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:33.729039    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8667a51c-8b4b-4a3b-ad4f-0284f10fa1ed" containerName="main"
Jul 28 18:51:33.731851 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:33.731835    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-140545011630896"
Jul 28 18:51:33.734502 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:33.734472    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-swrnk\""
Jul 28 18:51:33.736493 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:33.736470    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7351-140545011630896"]
Jul 28 18:51:33.808154 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:33.808129    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjhj\" (UniqueName: \"kubernetes.io/projected/7c71ea57-dea2-4096-b9ec-68c283e89e24-kube-api-access-5wjhj\") pod \"test-curl-7351-140545011630896\" (UID: \"7c71ea57-dea2-4096-b9ec-68c283e89e24\") " pod="opendatahub/test-curl-7351-140545011630896"
Jul 28 18:51:33.909241 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:33.909214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjhj\" (UniqueName: \"kubernetes.io/projected/7c71ea57-dea2-4096-b9ec-68c283e89e24-kube-api-access-5wjhj\") pod \"test-curl-7351-140545011630896\" (UID: \"7c71ea57-dea2-4096-b9ec-68c283e89e24\") " pod="opendatahub/test-curl-7351-140545011630896"
Jul 28 18:51:33.917008 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:33.916990    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjhj\" (UniqueName: \"kubernetes.io/projected/7c71ea57-dea2-4096-b9ec-68c283e89e24-kube-api-access-5wjhj\") pod \"test-curl-7351-140545011630896\" (UID: \"7c71ea57-dea2-4096-b9ec-68c283e89e24\") " pod="opendatahub/test-curl-7351-140545011630896"
Jul 28 18:51:34.042502 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:34.042474    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-140545011630896"
Jul 28 18:51:34.172843 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:34.172800    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7351-140545011630896"]
Jul 28 18:51:34.175468 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:51:34.175443    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c71ea57_dea2_4096_b9ec_68c283e89e24.slice/crio-516a7b4d356a3432553d4da838f5aa82d56dc5d45cbd7139323a4a62dcaf227d WatchSource:0}: Error finding container 516a7b4d356a3432553d4da838f5aa82d56dc5d45cbd7139323a4a62dcaf227d: Status 404 returned error can't find the container with id 516a7b4d356a3432553d4da838f5aa82d56dc5d45cbd7139323a4a62dcaf227d
Jul 28 18:51:34.177298 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:34.177278    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:51:34.181991 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:34.181963    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7351-140545011630896" event={"ID":"7c71ea57-dea2-4096-b9ec-68c283e89e24","Type":"ContainerStarted","Data":"516a7b4d356a3432553d4da838f5aa82d56dc5d45cbd7139323a4a62dcaf227d"}
Jul 28 18:51:35.188052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:35.187962    2564 generic.go:356] "Generic (PLEG): container finished" podID="7c71ea57-dea2-4096-b9ec-68c283e89e24" containerID="4f8234c7a63be18405a183fdd16644a2829f01f5f6bd268d7c5003aed2694c35" exitCode=0
Jul 28 18:51:35.188052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:35.188038    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7351-140545011630896" event={"ID":"7c71ea57-dea2-4096-b9ec-68c283e89e24","Type":"ContainerDied","Data":"4f8234c7a63be18405a183fdd16644a2829f01f5f6bd268d7c5003aed2694c35"}
Jul 28 18:51:36.327736 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:36.327708    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-140545011630896"
Jul 28 18:51:36.362204 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:36.362173    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7351-140545011630896_7c71ea57-dea2-4096-b9ec-68c283e89e24/test-curl-7351-140545011630896/0.log"
Jul 28 18:51:36.382354 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:36.382326    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7351-140545011630896"]
Jul 28 18:51:36.388151 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:36.388126    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7351-140545011630896"]
Jul 28 18:51:36.430882 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:36.430849    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wjhj\" (UniqueName: \"kubernetes.io/projected/7c71ea57-dea2-4096-b9ec-68c283e89e24-kube-api-access-5wjhj\") pod \"7c71ea57-dea2-4096-b9ec-68c283e89e24\" (UID: \"7c71ea57-dea2-4096-b9ec-68c283e89e24\") "
Jul 28 18:51:36.432973 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:36.432936    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c71ea57-dea2-4096-b9ec-68c283e89e24-kube-api-access-5wjhj" (OuterVolumeSpecName: "kube-api-access-5wjhj") pod "7c71ea57-dea2-4096-b9ec-68c283e89e24" (UID: "7c71ea57-dea2-4096-b9ec-68c283e89e24"). InnerVolumeSpecName "kube-api-access-5wjhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:51:36.532042 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:36.532009    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wjhj\" (UniqueName: \"kubernetes.io/projected/7c71ea57-dea2-4096-b9ec-68c283e89e24-kube-api-access-5wjhj\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:51:37.198735 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:37.198705    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-140545011630896"
Jul 28 18:51:37.198921 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:37.198749    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="516a7b4d356a3432553d4da838f5aa82d56dc5d45cbd7139323a4a62dcaf227d"
Jul 28 18:51:37.208960 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:51:37.208934    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7351-140545011630896\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="7c71ea57-dea2-4096-b9ec-68c283e89e24" pod="opendatahub/test-curl-7351-140545011630896"
Jul 28 18:51:37.216232 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:37.216210    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c71ea57-dea2-4096-b9ec-68c283e89e24" path="/var/lib/kubelet/pods/7c71ea57-dea2-4096-b9ec-68c283e89e24/volumes"
Jul 28 18:51:38.771717 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:38.771681    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7351-140545006178496"]
Jul 28 18:51:38.772111 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:38.772011    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c71ea57-dea2-4096-b9ec-68c283e89e24" containerName="test-curl-7351-140545011630896"
Jul 28 18:51:38.772111 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:38.772022    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c71ea57-dea2-4096-b9ec-68c283e89e24" containerName="test-curl-7351-140545011630896"
Jul 28 18:51:38.772111 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:38.772087    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c71ea57-dea2-4096-b9ec-68c283e89e24" containerName="test-curl-7351-140545011630896"
Jul 28 18:51:38.774986 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:38.774968    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-140545006178496"
Jul 28 18:51:38.777465 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:38.777446    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-swrnk\""
Jul 28 18:51:38.781895 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:38.781869    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7351-140545006178496"]
Jul 28 18:51:38.854458 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:38.854431    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4sbs\" (UniqueName: \"kubernetes.io/projected/441bad3d-981e-47f0-86c3-252bc6a53e90-kube-api-access-g4sbs\") pod \"test-curl-7351-140545006178496\" (UID: \"441bad3d-981e-47f0-86c3-252bc6a53e90\") " pod="opendatahub/test-curl-7351-140545006178496"
Jul 28 18:51:38.955422 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:38.955391    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4sbs\" (UniqueName: \"kubernetes.io/projected/441bad3d-981e-47f0-86c3-252bc6a53e90-kube-api-access-g4sbs\") pod \"test-curl-7351-140545006178496\" (UID: \"441bad3d-981e-47f0-86c3-252bc6a53e90\") " pod="opendatahub/test-curl-7351-140545006178496"
Jul 28 18:51:38.963064 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:38.963032    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4sbs\" (UniqueName: \"kubernetes.io/projected/441bad3d-981e-47f0-86c3-252bc6a53e90-kube-api-access-g4sbs\") pod \"test-curl-7351-140545006178496\" (UID: \"441bad3d-981e-47f0-86c3-252bc6a53e90\") " pod="opendatahub/test-curl-7351-140545006178496"
Jul 28 18:51:39.086492 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:39.086404    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-140545006178496"
Jul 28 18:51:39.216919 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:51:39.216890    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod441bad3d_981e_47f0_86c3_252bc6a53e90.slice/crio-949efeb4e0b5b73fc56ae7a8ab98b3b1d036285fff5bb791a374641f358b4622 WatchSource:0}: Error finding container 949efeb4e0b5b73fc56ae7a8ab98b3b1d036285fff5bb791a374641f358b4622: Status 404 returned error can't find the container with id 949efeb4e0b5b73fc56ae7a8ab98b3b1d036285fff5bb791a374641f358b4622
Jul 28 18:51:39.218195 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:39.218176    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7351-140545006178496"]
Jul 28 18:51:40.212463 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:40.212429    2564 generic.go:356] "Generic (PLEG): container finished" podID="441bad3d-981e-47f0-86c3-252bc6a53e90" containerID="c5c9faf44430f1a2d8ce431c2a71122f5532e1f7fcf5221812e2283baacc0cd2" exitCode=0
Jul 28 18:51:40.212866 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:40.212515    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7351-140545006178496" event={"ID":"441bad3d-981e-47f0-86c3-252bc6a53e90","Type":"ContainerDied","Data":"c5c9faf44430f1a2d8ce431c2a71122f5532e1f7fcf5221812e2283baacc0cd2"}
Jul 28 18:51:40.212866 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:40.212556    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7351-140545006178496" event={"ID":"441bad3d-981e-47f0-86c3-252bc6a53e90","Type":"ContainerStarted","Data":"949efeb4e0b5b73fc56ae7a8ab98b3b1d036285fff5bb791a374641f358b4622"}
Jul 28 18:51:41.346728 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:41.346698    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-140545006178496"
Jul 28 18:51:41.366130 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:41.366092    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7351-140545006178496_441bad3d-981e-47f0-86c3-252bc6a53e90/test-curl-7351-140545006178496/0.log"
Jul 28 18:51:41.388552 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:41.388526    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7351-140545006178496"]
Jul 28 18:51:41.392484 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:41.392459    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7351-140545006178496"]
Jul 28 18:51:41.475892 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:41.475796    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4sbs\" (UniqueName: \"kubernetes.io/projected/441bad3d-981e-47f0-86c3-252bc6a53e90-kube-api-access-g4sbs\") pod \"441bad3d-981e-47f0-86c3-252bc6a53e90\" (UID: \"441bad3d-981e-47f0-86c3-252bc6a53e90\") "
Jul 28 18:51:41.477823 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:41.477787    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441bad3d-981e-47f0-86c3-252bc6a53e90-kube-api-access-g4sbs" (OuterVolumeSpecName: "kube-api-access-g4sbs") pod "441bad3d-981e-47f0-86c3-252bc6a53e90" (UID: "441bad3d-981e-47f0-86c3-252bc6a53e90"). InnerVolumeSpecName "kube-api-access-g4sbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:51:41.577120 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:41.577095    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4sbs\" (UniqueName: \"kubernetes.io/projected/441bad3d-981e-47f0-86c3-252bc6a53e90-kube-api-access-g4sbs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:51:42.223727 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:42.223698    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-140545006178496"
Jul 28 18:51:42.223919 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:42.223737    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="949efeb4e0b5b73fc56ae7a8ab98b3b1d036285fff5bb791a374641f358b4622"
Jul 28 18:51:42.234053 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:51:42.234022    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7351-140545006178496\" is forbidden: User \"system:node:ip-10-0-139-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-99.ec2.internal' and this object" podUID="441bad3d-981e-47f0-86c3-252bc6a53e90" pod="opendatahub/test-curl-7351-140545006178496"
Jul 28 18:51:43.216654 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.216619    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="441bad3d-981e-47f0-86c3-252bc6a53e90" path="/var/lib/kubelet/pods/441bad3d-981e-47f0-86c3-252bc6a53e90/volumes"
Jul 28 18:51:43.451271 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.451232    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt" podUID="4064c411-cb76-4dc8-b715-c0e5f1082ca3" containerName="main" containerID="cri-o://d1c7f6688830a76eadd27f4a615a9a094be3837200f925232103f4374fff7236" gracePeriod=45
Jul 28 18:51:43.706840 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.706818    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:51:43.778103 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.778078    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7351-140545006179504"]
Jul 28 18:51:43.778410 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.778399    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4064c411-cb76-4dc8-b715-c0e5f1082ca3" containerName="main"
Jul 28 18:51:43.778468 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.778411    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4064c411-cb76-4dc8-b715-c0e5f1082ca3" containerName="main"
Jul 28 18:51:43.778468 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.778420    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="441bad3d-981e-47f0-86c3-252bc6a53e90" containerName="test-curl-7351-140545006178496"
Jul 28 18:51:43.778468 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.778427    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="441bad3d-981e-47f0-86c3-252bc6a53e90" containerName="test-curl-7351-140545006178496"
Jul 28 18:51:43.778561 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.778504    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="441bad3d-981e-47f0-86c3-252bc6a53e90" containerName="test-curl-7351-140545006178496"
Jul 28 18:51:43.778561 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.778515    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4064c411-cb76-4dc8-b715-c0e5f1082ca3" containerName="main"
Jul 28 18:51:43.781300 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.781281    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-140545006179504"
Jul 28 18:51:43.783584 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.783562    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-swrnk\""
Jul 28 18:51:43.788928 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.788910    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7351-140545006179504"]
Jul 28 18:51:43.797413 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.797394    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-model-cache\") pod \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") "
Jul 28 18:51:43.797528 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.797429    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4064c411-cb76-4dc8-b715-c0e5f1082ca3-tls-certs\") pod \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") "
Jul 28 18:51:43.797528 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.797479    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-home\") pod \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") "
Jul 28 18:51:43.797528 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.797520    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-dshm\") pod \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") "
Jul 28 18:51:43.797689 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.797556    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-tmp-dir\") pod \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") "
Jul 28 18:51:43.797689 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.797584    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxdz6\" (UniqueName: \"kubernetes.io/projected/4064c411-cb76-4dc8-b715-c0e5f1082ca3-kube-api-access-cxdz6\") pod \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\" (UID: \"4064c411-cb76-4dc8-b715-c0e5f1082ca3\") "
Jul 28 18:51:43.797689 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.797634    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-model-cache" (OuterVolumeSpecName: "model-cache") pod "4064c411-cb76-4dc8-b715-c0e5f1082ca3" (UID: "4064c411-cb76-4dc8-b715-c0e5f1082ca3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:51:43.797870 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.797708    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-home" (OuterVolumeSpecName: "home") pod "4064c411-cb76-4dc8-b715-c0e5f1082ca3" (UID: "4064c411-cb76-4dc8-b715-c0e5f1082ca3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:51:43.797942 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.797870    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4064c411-cb76-4dc8-b715-c0e5f1082ca3" (UID: "4064c411-cb76-4dc8-b715-c0e5f1082ca3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:51:43.797942 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.797886    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:51:43.797942 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.797901    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:51:43.799643 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.799614    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4064c411-cb76-4dc8-b715-c0e5f1082ca3-kube-api-access-cxdz6" (OuterVolumeSpecName: "kube-api-access-cxdz6") pod "4064c411-cb76-4dc8-b715-c0e5f1082ca3" (UID: "4064c411-cb76-4dc8-b715-c0e5f1082ca3"). InnerVolumeSpecName "kube-api-access-cxdz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:51:43.800048 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.800025    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4064c411-cb76-4dc8-b715-c0e5f1082ca3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4064c411-cb76-4dc8-b715-c0e5f1082ca3" (UID: "4064c411-cb76-4dc8-b715-c0e5f1082ca3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:51:43.800129 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.800045    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-dshm" (OuterVolumeSpecName: "dshm") pod "4064c411-cb76-4dc8-b715-c0e5f1082ca3" (UID: "4064c411-cb76-4dc8-b715-c0e5f1082ca3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:51:43.898766 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.898707    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jzmh\" (UniqueName: \"kubernetes.io/projected/0388adf1-87c0-42cd-bfc8-38147fbfc02a-kube-api-access-2jzmh\") pod \"test-curl-7351-140545006179504\" (UID: \"0388adf1-87c0-42cd-bfc8-38147fbfc02a\") " pod="opendatahub/test-curl-7351-140545006179504"
Jul 28 18:51:43.898766 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.898755    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:51:43.898947 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.898771    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4064c411-cb76-4dc8-b715-c0e5f1082ca3-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:51:43.898947 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.898783    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxdz6\" (UniqueName: \"kubernetes.io/projected/4064c411-cb76-4dc8-b715-c0e5f1082ca3-kube-api-access-cxdz6\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:51:43.898947 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.898796    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4064c411-cb76-4dc8-b715-c0e5f1082ca3-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:51:43.999365 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:43.999340    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzmh\" (UniqueName: \"kubernetes.io/projected/0388adf1-87c0-42cd-bfc8-38147fbfc02a-kube-api-access-2jzmh\") pod \"test-curl-7351-140545006179504\" (UID: \"0388adf1-87c0-42cd-bfc8-38147fbfc02a\") " pod="opendatahub/test-curl-7351-140545006179504"
Jul 28 18:51:44.008213 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:44.008197    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jzmh\" (UniqueName: \"kubernetes.io/projected/0388adf1-87c0-42cd-bfc8-38147fbfc02a-kube-api-access-2jzmh\") pod \"test-curl-7351-140545006179504\" (UID: \"0388adf1-87c0-42cd-bfc8-38147fbfc02a\") " pod="opendatahub/test-curl-7351-140545006179504"
Jul 28 18:51:44.091660 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:44.091622    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-140545006179504"
Jul 28 18:51:44.220710 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:44.220688    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7351-140545006179504"]
Jul 28 18:51:44.222865 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:51:44.222843    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0388adf1_87c0_42cd_bfc8_38147fbfc02a.slice/crio-cecb4e8667fd6815b583beeb164fcd40c0c670833aa1236bc42c88a16b4bcbb3 WatchSource:0}: Error finding container cecb4e8667fd6815b583beeb164fcd40c0c670833aa1236bc42c88a16b4bcbb3: Status 404 returned error can't find the container with id cecb4e8667fd6815b583beeb164fcd40c0c670833aa1236bc42c88a16b4bcbb3
Jul 28 18:51:44.233524 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:44.233497    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7351-140545006179504" event={"ID":"0388adf1-87c0-42cd-bfc8-38147fbfc02a","Type":"ContainerStarted","Data":"cecb4e8667fd6815b583beeb164fcd40c0c670833aa1236bc42c88a16b4bcbb3"}
Jul 28 18:51:44.234733 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:44.234710    2564 generic.go:356] "Generic (PLEG): container finished" podID="4064c411-cb76-4dc8-b715-c0e5f1082ca3" containerID="d1c7f6688830a76eadd27f4a615a9a094be3837200f925232103f4374fff7236" exitCode=0
Jul 28 18:51:44.234850 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:44.234751    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"
Jul 28 18:51:44.234850 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:44.234751    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt" event={"ID":"4064c411-cb76-4dc8-b715-c0e5f1082ca3","Type":"ContainerDied","Data":"d1c7f6688830a76eadd27f4a615a9a094be3837200f925232103f4374fff7236"}
Jul 28 18:51:44.234928 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:44.234872    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt" event={"ID":"4064c411-cb76-4dc8-b715-c0e5f1082ca3","Type":"ContainerDied","Data":"0ce8d8e017118b00478c10e9834f14ceb9061b48a581d66b8d93738d8b957cc2"}
Jul 28 18:51:44.234928 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:44.234897    2564 scope.go:117] "RemoveContainer" containerID="d1c7f6688830a76eadd27f4a615a9a094be3837200f925232103f4374fff7236"
Jul 28 18:51:44.244018 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:44.244001    2564 scope.go:117] "RemoveContainer" containerID="d1c7f6688830a76eadd27f4a615a9a094be3837200f925232103f4374fff7236"
Jul 28 18:51:44.244258 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:51:44.244240    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c7f6688830a76eadd27f4a615a9a094be3837200f925232103f4374fff7236\": container with ID starting with d1c7f6688830a76eadd27f4a615a9a094be3837200f925232103f4374fff7236 not found: ID does not exist" containerID="d1c7f6688830a76eadd27f4a615a9a094be3837200f925232103f4374fff7236"
Jul 28 18:51:44.244320 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:44.244269    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c7f6688830a76eadd27f4a615a9a094be3837200f925232103f4374fff7236"} err="failed to get container status \"d1c7f6688830a76eadd27f4a615a9a094be3837200f925232103f4374fff7236\": rpc error: code = NotFound desc = could not find container \"d1c7f6688830a76eadd27f4a615a9a094be3837200f925232103f4374fff7236\": container with ID starting with d1c7f6688830a76eadd27f4a615a9a094be3837200f925232103f4374fff7236 not found: ID does not exist"
Jul 28 18:51:44.256072 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:44.256050    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"]
Jul 28 18:51:44.258182 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:44.258124    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-37e11d/rate-test-model-37e11d-kserve-5dc9d86f5f-lqgqt"]
Jul 28 18:51:45.216959 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:45.216930    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4064c411-cb76-4dc8-b715-c0e5f1082ca3" path="/var/lib/kubelet/pods/4064c411-cb76-4dc8-b715-c0e5f1082ca3/volumes"
Jul 28 18:51:45.240676 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:45.240643    2564 generic.go:356] "Generic (PLEG): container finished" podID="0388adf1-87c0-42cd-bfc8-38147fbfc02a" containerID="6a44293c8c31f7cb27c8bc438b831e0ab3e019ea056a2e0712593e249ff9bf9d" exitCode=0
Jul 28 18:51:45.241097 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:45.240725    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7351-140545006179504" event={"ID":"0388adf1-87c0-42cd-bfc8-38147fbfc02a","Type":"ContainerDied","Data":"6a44293c8c31f7cb27c8bc438b831e0ab3e019ea056a2e0712593e249ff9bf9d"}
Jul 28 18:51:46.376897 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:46.376869    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-140545006179504"
Jul 28 18:51:46.397844 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:46.397797    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7351-140545006179504_0388adf1-87c0-42cd-bfc8-38147fbfc02a/test-curl-7351-140545006179504/0.log"
Jul 28 18:51:46.418500 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:46.418468    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7351-140545006179504"]
Jul 28 18:51:46.423126 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:46.423102    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7351-140545006179504"]
Jul 28 18:51:46.521409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:46.521383    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jzmh\" (UniqueName: \"kubernetes.io/projected/0388adf1-87c0-42cd-bfc8-38147fbfc02a-kube-api-access-2jzmh\") pod \"0388adf1-87c0-42cd-bfc8-38147fbfc02a\" (UID: \"0388adf1-87c0-42cd-bfc8-38147fbfc02a\") "
Jul 28 18:51:46.523431 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:46.523395    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0388adf1-87c0-42cd-bfc8-38147fbfc02a-kube-api-access-2jzmh" (OuterVolumeSpecName: "kube-api-access-2jzmh") pod "0388adf1-87c0-42cd-bfc8-38147fbfc02a" (UID: "0388adf1-87c0-42cd-bfc8-38147fbfc02a"). InnerVolumeSpecName "kube-api-access-2jzmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:51:46.622340 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:46.622307    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jzmh\" (UniqueName: \"kubernetes.io/projected/0388adf1-87c0-42cd-bfc8-38147fbfc02a-kube-api-access-2jzmh\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:51:47.217568 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:47.217531    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0388adf1-87c0-42cd-bfc8-38147fbfc02a" path="/var/lib/kubelet/pods/0388adf1-87c0-42cd-bfc8-38147fbfc02a/volumes"
Jul 28 18:51:47.251128 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:47.251099    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-140545006179504"
Jul 28 18:51:47.251279 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:47.251143    2564 scope.go:117] "RemoveContainer" containerID="6a44293c8c31f7cb27c8bc438b831e0ab3e019ea056a2e0712593e249ff9bf9d"
Jul 28 18:51:51.343240 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.343202    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"]
Jul 28 18:51:51.343613 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.343512    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0388adf1-87c0-42cd-bfc8-38147fbfc02a" containerName="test-curl-7351-140545006179504"
Jul 28 18:51:51.343613 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.343523    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0388adf1-87c0-42cd-bfc8-38147fbfc02a" containerName="test-curl-7351-140545006179504"
Jul 28 18:51:51.343613 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.343599    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0388adf1-87c0-42cd-bfc8-38147fbfc02a" containerName="test-curl-7351-140545006179504"
Jul 28 18:51:51.347931 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.347914    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"
Jul 28 18:51:51.350616 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.350592    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c1237e\""
Jul 28 18:51:51.351012 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.350994    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c1237e-dockercfg-8q6ps\""
Jul 28 18:51:51.356193 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.356170    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"]
Jul 28 18:51:51.383458 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.383431    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"]
Jul 28 18:51:51.386597 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.386575    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"
Jul 28 18:51:51.396581 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.396559    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"]
Jul 28 18:51:51.466907 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.466875    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25c1c67-fd0e-43f3-9364-571fc86526a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp\" (UID: \"d25c1c67-fd0e-43f3-9364-571fc86526a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"
Jul 28 18:51:51.467111 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.466982    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsjv9\" (UniqueName: \"kubernetes.io/projected/d25c1c67-fd0e-43f3-9364-571fc86526a5-kube-api-access-gsjv9\") pod \"payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp\" (UID: \"d25c1c67-fd0e-43f3-9364-571fc86526a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"
Jul 28 18:51:51.567884 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.567846    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mz8c\" (UniqueName: \"kubernetes.io/projected/3b91153a-180a-4825-9b9e-445848e64752-kube-api-access-4mz8c\") pod \"payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592\" (UID: \"3b91153a-180a-4825-9b9e-445848e64752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"
Jul 28 18:51:51.568117 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.567991    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25c1c67-fd0e-43f3-9364-571fc86526a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp\" (UID: \"d25c1c67-fd0e-43f3-9364-571fc86526a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"
Jul 28 18:51:51.568117 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.568101    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b91153a-180a-4825-9b9e-445848e64752-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592\" (UID: \"3b91153a-180a-4825-9b9e-445848e64752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"
Jul 28 18:51:51.568237 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.568128    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsjv9\" (UniqueName: \"kubernetes.io/projected/d25c1c67-fd0e-43f3-9364-571fc86526a5-kube-api-access-gsjv9\") pod \"payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp\" (UID: \"d25c1c67-fd0e-43f3-9364-571fc86526a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"
Jul 28 18:51:51.568692 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.568670    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25c1c67-fd0e-43f3-9364-571fc86526a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp\" (UID: \"d25c1c67-fd0e-43f3-9364-571fc86526a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"
Jul 28 18:51:51.580764 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.580741    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsjv9\" (UniqueName: \"kubernetes.io/projected/d25c1c67-fd0e-43f3-9364-571fc86526a5-kube-api-access-gsjv9\") pod \"payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp\" (UID: \"d25c1c67-fd0e-43f3-9364-571fc86526a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"
Jul 28 18:51:51.659284 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.659199    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"
Jul 28 18:51:51.669495 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.669468    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b91153a-180a-4825-9b9e-445848e64752-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592\" (UID: \"3b91153a-180a-4825-9b9e-445848e64752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"
Jul 28 18:51:51.669599 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.669508    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mz8c\" (UniqueName: \"kubernetes.io/projected/3b91153a-180a-4825-9b9e-445848e64752-kube-api-access-4mz8c\") pod \"payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592\" (UID: \"3b91153a-180a-4825-9b9e-445848e64752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"
Jul 28 18:51:51.670244 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.670225    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b91153a-180a-4825-9b9e-445848e64752-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592\" (UID: \"3b91153a-180a-4825-9b9e-445848e64752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"
Jul 28 18:51:51.677445 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.677416    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mz8c\" (UniqueName: \"kubernetes.io/projected/3b91153a-180a-4825-9b9e-445848e64752-kube-api-access-4mz8c\") pod \"payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592\" (UID: \"3b91153a-180a-4825-9b9e-445848e64752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"
Jul 28 18:51:51.697236 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.697197    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"
Jul 28 18:51:51.805070 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.805036    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"]
Jul 28 18:51:51.808831 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:51:51.808786    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25c1c67_fd0e_43f3_9364_571fc86526a5.slice/crio-155a38c98dfecf588c5944e775afb6c23dbef17df62e9fdad2703809196d08dd WatchSource:0}: Error finding container 155a38c98dfecf588c5944e775afb6c23dbef17df62e9fdad2703809196d08dd: Status 404 returned error can't find the container with id 155a38c98dfecf588c5944e775afb6c23dbef17df62e9fdad2703809196d08dd
Jul 28 18:51:51.853771 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:51.853744    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"]
Jul 28 18:51:51.864987 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:51:51.864962    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b91153a_180a_4825_9b9e_445848e64752.slice/crio-76232eb02f49f37475c9fcf5b5853e6f16f70dc32998fa6a63252840865b44bf WatchSource:0}: Error finding container 76232eb02f49f37475c9fcf5b5853e6f16f70dc32998fa6a63252840865b44bf: Status 404 returned error can't find the container with id 76232eb02f49f37475c9fcf5b5853e6f16f70dc32998fa6a63252840865b44bf
Jul 28 18:51:52.274265 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:52.274225    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592" event={"ID":"3b91153a-180a-4825-9b9e-445848e64752","Type":"ContainerStarted","Data":"6c59433b685ad990e221b537f069c0bc09488a13ecc82cc69f515b3ad9fd6ff5"}
Jul 28 18:51:52.274451 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:52.274297    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592" event={"ID":"3b91153a-180a-4825-9b9e-445848e64752","Type":"ContainerStarted","Data":"76232eb02f49f37475c9fcf5b5853e6f16f70dc32998fa6a63252840865b44bf"}
Jul 28 18:51:52.274451 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:52.274325    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"
Jul 28 18:51:52.275638 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:52.275615    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp" event={"ID":"d25c1c67-fd0e-43f3-9364-571fc86526a5","Type":"ContainerStarted","Data":"fb3d772b3245132ceeb6353c0076d9e978915e51ef208fa75bf119aeb1804154"}
Jul 28 18:51:52.275787 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:52.275658    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp" event={"ID":"d25c1c67-fd0e-43f3-9364-571fc86526a5","Type":"ContainerStarted","Data":"155a38c98dfecf588c5944e775afb6c23dbef17df62e9fdad2703809196d08dd"}
Jul 28 18:51:52.275787 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:52.275730    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"
Jul 28 18:51:52.291162 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:52.291111    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592" podStartSLOduration=1.291092144 podStartE2EDuration="1.291092144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:51:52.290359237 +0000 UTC m=+3047.605861362" watchObservedRunningTime="2026-07-28 18:51:52.291092144 +0000 UTC m=+3047.606594257"
Jul 28 18:51:52.308291 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:51:52.308237    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp" podStartSLOduration=1.30822134 podStartE2EDuration="1.30822134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:51:52.30664895 +0000 UTC m=+3047.622151065" watchObservedRunningTime="2026-07-28 18:51:52.30822134 +0000 UTC m=+3047.623723452"
Jul 28 18:52:03.282273 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:03.282201    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"
Jul 28 18:52:03.282273 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:03.282271    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"
Jul 28 18:52:22.270969 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.270934    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"]
Jul 28 18:52:22.274375 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.274356    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"
Jul 28 18:52:22.277228 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.277208    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0ae953-serving-cert\""
Jul 28 18:52:22.283830 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.283787    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"]
Jul 28 18:52:22.312091 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.312057    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2gwl\" (UniqueName: \"kubernetes.io/projected/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-kube-api-access-g2gwl\") pod \"maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f\" (UID: \"54e294e3-c8ae-4033-8ebf-bac42d3bbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"
Jul 28 18:52:22.312249 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.312104    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f\" (UID: \"54e294e3-c8ae-4033-8ebf-bac42d3bbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"
Jul 28 18:52:22.312249 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.312154    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f\" (UID: \"54e294e3-c8ae-4033-8ebf-bac42d3bbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"
Jul 28 18:52:22.412616 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.412583    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gwl\" (UniqueName: \"kubernetes.io/projected/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-kube-api-access-g2gwl\") pod \"maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f\" (UID: \"54e294e3-c8ae-4033-8ebf-bac42d3bbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"
Jul 28 18:52:22.412801 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.412633    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f\" (UID: \"54e294e3-c8ae-4033-8ebf-bac42d3bbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"
Jul 28 18:52:22.412801 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.412663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f\" (UID: \"54e294e3-c8ae-4033-8ebf-bac42d3bbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"
Jul 28 18:52:22.413334 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.413307    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f\" (UID: \"54e294e3-c8ae-4033-8ebf-bac42d3bbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"
Jul 28 18:52:22.414971 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.414947    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f\" (UID: \"54e294e3-c8ae-4033-8ebf-bac42d3bbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"
Jul 28 18:52:22.421448 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.421423    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2gwl\" (UniqueName: \"kubernetes.io/projected/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-kube-api-access-g2gwl\") pod \"maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f\" (UID: \"54e294e3-c8ae-4033-8ebf-bac42d3bbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"
Jul 28 18:52:22.586800 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.586708    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"
Jul 28 18:52:22.732245 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.732219    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"]
Jul 28 18:52:22.734368 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:52:22.734336    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e294e3_c8ae_4033_8ebf_bac42d3bbd81.slice/crio-fecb9a0c900d3137db613ac7893ee6c854fcfc1c9af3a5f9c1bc211d9126c909 WatchSource:0}: Error finding container fecb9a0c900d3137db613ac7893ee6c854fcfc1c9af3a5f9c1bc211d9126c909: Status 404 returned error can't find the container with id fecb9a0c900d3137db613ac7893ee6c854fcfc1c9af3a5f9c1bc211d9126c909
Jul 28 18:52:22.856998 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.856914    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"]
Jul 28 18:52:22.861352 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.861334    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"
Jul 28 18:52:22.863850 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.863826    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0ae953-dockercfg-vczq6\""
Jul 28 18:52:22.864014 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.863995    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0ae953\""
Jul 28 18:52:22.870637 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.870614    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"]
Jul 28 18:52:22.916658 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.916630    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfjbk\" (UniqueName: \"kubernetes.io/projected/7460543a-f55b-4b52-84a0-40d0ea267d97-kube-api-access-hfjbk\") pod \"payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw\" (UID: \"7460543a-f55b-4b52-84a0-40d0ea267d97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"
Jul 28 18:52:22.916859 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:22.916670    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7460543a-f55b-4b52-84a0-40d0ea267d97-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw\" (UID: \"7460543a-f55b-4b52-84a0-40d0ea267d97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"
Jul 28 18:52:23.017408 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:23.017357    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfjbk\" (UniqueName: \"kubernetes.io/projected/7460543a-f55b-4b52-84a0-40d0ea267d97-kube-api-access-hfjbk\") pod \"payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw\" (UID: \"7460543a-f55b-4b52-84a0-40d0ea267d97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"
Jul 28 18:52:23.017588 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:23.017415    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7460543a-f55b-4b52-84a0-40d0ea267d97-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw\" (UID: \"7460543a-f55b-4b52-84a0-40d0ea267d97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"
Jul 28 18:52:23.018207 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:23.018187    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7460543a-f55b-4b52-84a0-40d0ea267d97-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw\" (UID: \"7460543a-f55b-4b52-84a0-40d0ea267d97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"
Jul 28 18:52:23.026190 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:23.026166    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfjbk\" (UniqueName: \"kubernetes.io/projected/7460543a-f55b-4b52-84a0-40d0ea267d97-kube-api-access-hfjbk\") pod \"payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw\" (UID: \"7460543a-f55b-4b52-84a0-40d0ea267d97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"
Jul 28 18:52:23.173017 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:23.172928    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"
Jul 28 18:52:23.314273 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:23.314247    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"]
Jul 28 18:52:23.316861 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:52:23.316831    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7460543a_f55b_4b52_84a0_40d0ea267d97.slice/crio-1d12cbabc754022b5ed6cabbe9b139f9ccd43131517208a8e4c8315a6d113fce WatchSource:0}: Error finding container 1d12cbabc754022b5ed6cabbe9b139f9ccd43131517208a8e4c8315a6d113fce: Status 404 returned error can't find the container with id 1d12cbabc754022b5ed6cabbe9b139f9ccd43131517208a8e4c8315a6d113fce
Jul 28 18:52:23.422648 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:23.422615    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw" event={"ID":"7460543a-f55b-4b52-84a0-40d0ea267d97","Type":"ContainerStarted","Data":"4f6923958a6850fa731feae71b07569e48cbee9b7e985783e3380ea944e613c9"}
Jul 28 18:52:23.422791 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:23.422672    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw" event={"ID":"7460543a-f55b-4b52-84a0-40d0ea267d97","Type":"ContainerStarted","Data":"1d12cbabc754022b5ed6cabbe9b139f9ccd43131517208a8e4c8315a6d113fce"}
Jul 28 18:52:23.422791 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:23.422700    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"
Jul 28 18:52:23.424391 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:23.424326    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f" event={"ID":"54e294e3-c8ae-4033-8ebf-bac42d3bbd81","Type":"ContainerStarted","Data":"570a1a108ccb6804b3deb3b36ac2c268067e932a767e65e1769328f102328d2c"}
Jul 28 18:52:23.424391 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:23.424379    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f" event={"ID":"54e294e3-c8ae-4033-8ebf-bac42d3bbd81","Type":"ContainerStarted","Data":"fecb9a0c900d3137db613ac7893ee6c854fcfc1c9af3a5f9c1bc211d9126c909"}
Jul 28 18:52:23.424557 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:23.424475    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"
Jul 28 18:52:23.442469 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:23.442430    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw" podStartSLOduration=1.442416272 podStartE2EDuration="1.442416272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:52:23.440424215 +0000 UTC m=+3078.755926354" watchObservedRunningTime="2026-07-28 18:52:23.442416272 +0000 UTC m=+3078.757918384"
Jul 28 18:52:23.458694 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:23.458649    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f" podStartSLOduration=1.152202275 podStartE2EDuration="1.458635283s" totalImagesPullingTime="306.433008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:52:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:52:23.456949304 +0000 UTC m=+3078.772451435" watchObservedRunningTime="2026-07-28 18:52:23.458635283 +0000 UTC m=+3078.774137400"
Jul 28 18:52:29.434574 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:29.434546    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"
Jul 28 18:52:34.430702 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:34.430672    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"
Jul 28 18:52:58.095536 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.095496    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"]
Jul 28 18:52:58.099073 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.099055    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.101688 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.101663    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c1237e\"/\"ipp-route-c1237e-kserve-self-signed-certs\""
Jul 28 18:52:58.103106 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.103085    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c1237e\"/\"openshift-service-ca.crt\""
Jul 28 18:52:58.103202 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.103103    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c1237e\"/\"default-dockercfg-xp95h\""
Jul 28 18:52:58.103202 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.103173    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c1237e\"/\"kube-root-ca.crt\""
Jul 28 18:52:58.110554 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.110531    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"]
Jul 28 18:52:58.226945 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.226903    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-tmp-dir\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.227133 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.226987    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-home\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.227133 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.227029    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x5tk\" (UniqueName: \"kubernetes.io/projected/db3167a8-ea04-4d63-9244-016c165243d7-kube-api-access-5x5tk\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.227133 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.227081    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db3167a8-ea04-4d63-9244-016c165243d7-tls-certs\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.227133 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.227113    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-dshm\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.227312 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.227135    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-model-cache\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.328249 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.328195    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x5tk\" (UniqueName: \"kubernetes.io/projected/db3167a8-ea04-4d63-9244-016c165243d7-kube-api-access-5x5tk\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.328249 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.328255    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db3167a8-ea04-4d63-9244-016c165243d7-tls-certs\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.328509 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.328293    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-dshm\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.328509 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.328332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-model-cache\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.328509 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.328400    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-tmp-dir\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.328509 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.328478    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-home\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.328848 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.328792    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-model-cache\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.328992 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.328874    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-tmp-dir\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.328992 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.328929    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-home\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.330650 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.330627    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-dshm\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.331033 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.331013    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db3167a8-ea04-4d63-9244-016c165243d7-tls-certs\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.336867 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.336846    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x5tk\" (UniqueName: \"kubernetes.io/projected/db3167a8-ea04-4d63-9244-016c165243d7-kube-api-access-5x5tk\") pod \"ipp-route-c1237e-kserve-645bd9898c-h7565\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") " pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.409249 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.409166    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:52:58.545313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.545286    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"]
Jul 28 18:52:58.547843 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:52:58.547774    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb3167a8_ea04_4d63_9244_016c165243d7.slice/crio-1187a0ffd1f6ce7d894725515ac0e05f348ff6d9137817e592949864df0d60fe WatchSource:0}: Error finding container 1187a0ffd1f6ce7d894725515ac0e05f348ff6d9137817e592949864df0d60fe: Status 404 returned error can't find the container with id 1187a0ffd1f6ce7d894725515ac0e05f348ff6d9137817e592949864df0d60fe
Jul 28 18:52:58.591252 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:58.591210    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565" event={"ID":"db3167a8-ea04-4d63-9244-016c165243d7","Type":"ContainerStarted","Data":"1187a0ffd1f6ce7d894725515ac0e05f348ff6d9137817e592949864df0d60fe"}
Jul 28 18:52:59.596885 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:59.596845    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565" event={"ID":"db3167a8-ea04-4d63-9244-016c165243d7","Type":"ContainerStarted","Data":"a6021bb1968ad22bda468d02741b330c3305599d2b630df17e3031c0ecf2dd7c"}
Jul 28 18:52:59.615877 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:52:59.615820    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565" podStartSLOduration=1.61579183 podStartE2EDuration="1.61579183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:52:59.613834546 +0000 UTC m=+3114.929336650" watchObservedRunningTime="2026-07-28 18:52:59.61579183 +0000 UTC m=+3114.931293942"
Jul 28 18:53:08.409261 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:08.409224    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:53:08.409261 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:08.409269    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:53:08.421575 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:08.421552    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:53:08.650239 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:08.650213    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:53:38.760670 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:38.760573    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592_3b91153a-180a-4825-9b9e-445848e64752/payload-processing/0.log"
Jul 28 18:53:42.161959 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.161926    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"]
Jul 28 18:53:42.165410 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.165387    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"
Jul 28 18:53:42.168254 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.168228    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-51b7e8-serving-cert\""
Jul 28 18:53:42.181668 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.181641    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"]
Jul 28 18:53:42.311637 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.311595    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4\" (UID: \"3ec72956-0f6b-46ba-877c-fe8a9c80fb82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"
Jul 28 18:53:42.311898 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.311676    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54msq\" (UniqueName: \"kubernetes.io/projected/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-kube-api-access-54msq\") pod \"maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4\" (UID: \"3ec72956-0f6b-46ba-877c-fe8a9c80fb82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"
Jul 28 18:53:42.311898 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.311829    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4\" (UID: \"3ec72956-0f6b-46ba-877c-fe8a9c80fb82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"
Jul 28 18:53:42.412764 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.412673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4\" (UID: \"3ec72956-0f6b-46ba-877c-fe8a9c80fb82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"
Jul 28 18:53:42.412764 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.412741    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54msq\" (UniqueName: \"kubernetes.io/projected/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-kube-api-access-54msq\") pod \"maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4\" (UID: \"3ec72956-0f6b-46ba-877c-fe8a9c80fb82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"
Jul 28 18:53:42.412977 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.412792    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4\" (UID: \"3ec72956-0f6b-46ba-877c-fe8a9c80fb82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"
Jul 28 18:53:42.413425 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.413401    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4\" (UID: \"3ec72956-0f6b-46ba-877c-fe8a9c80fb82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"
Jul 28 18:53:42.414991 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.414969    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4\" (UID: \"3ec72956-0f6b-46ba-877c-fe8a9c80fb82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"
Jul 28 18:53:42.426052 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.426033    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54msq\" (UniqueName: \"kubernetes.io/projected/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-kube-api-access-54msq\") pod \"maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4\" (UID: \"3ec72956-0f6b-46ba-877c-fe8a9c80fb82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"
Jul 28 18:53:42.476592 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.476564    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"
Jul 28 18:53:42.605548 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.605521    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"]
Jul 28 18:53:42.608240 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:53:42.608210    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec72956_0f6b_46ba_877c_fe8a9c80fb82.slice/crio-51afc45c6b23f782ade269a9a40d8c097fcde506c972198e7e8ce004b7b02e4e WatchSource:0}: Error finding container 51afc45c6b23f782ade269a9a40d8c097fcde506c972198e7e8ce004b7b02e4e: Status 404 returned error can't find the container with id 51afc45c6b23f782ade269a9a40d8c097fcde506c972198e7e8ce004b7b02e4e
Jul 28 18:53:42.798495 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.798459    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4" event={"ID":"3ec72956-0f6b-46ba-877c-fe8a9c80fb82","Type":"ContainerStarted","Data":"51afc45c6b23f782ade269a9a40d8c097fcde506c972198e7e8ce004b7b02e4e"}
Jul 28 18:53:42.876472 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.876434    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"]
Jul 28 18:53:42.880971 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.880948    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"
Jul 28 18:53:42.883488 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.883465    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-51b7e8\""
Jul 28 18:53:42.883744 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.883730    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-51b7e8-dockercfg-ckbwd\""
Jul 28 18:53:42.887715 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:42.887641    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"]
Jul 28 18:53:43.017416 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:43.017382    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b74226-facb-49b4-ac25-48082d7ceab6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj\" (UID: \"c8b74226-facb-49b4-ac25-48082d7ceab6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"
Jul 28 18:53:43.017587 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:43.017447    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvc8j\" (UniqueName: \"kubernetes.io/projected/c8b74226-facb-49b4-ac25-48082d7ceab6-kube-api-access-wvc8j\") pod \"payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj\" (UID: \"c8b74226-facb-49b4-ac25-48082d7ceab6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"
Jul 28 18:53:43.118679 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:43.118649    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b74226-facb-49b4-ac25-48082d7ceab6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj\" (UID: \"c8b74226-facb-49b4-ac25-48082d7ceab6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"
Jul 28 18:53:43.118786 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:43.118717    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvc8j\" (UniqueName: \"kubernetes.io/projected/c8b74226-facb-49b4-ac25-48082d7ceab6-kube-api-access-wvc8j\") pod \"payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj\" (UID: \"c8b74226-facb-49b4-ac25-48082d7ceab6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"
Jul 28 18:53:43.119347 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:43.119330    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b74226-facb-49b4-ac25-48082d7ceab6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj\" (UID: \"c8b74226-facb-49b4-ac25-48082d7ceab6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"
Jul 28 18:53:43.126334 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:43.126310    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvc8j\" (UniqueName: \"kubernetes.io/projected/c8b74226-facb-49b4-ac25-48082d7ceab6-kube-api-access-wvc8j\") pod \"payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj\" (UID: \"c8b74226-facb-49b4-ac25-48082d7ceab6\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"
Jul 28 18:53:43.192531 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:43.192499    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"
Jul 28 18:53:43.334710 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:43.334684    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"]
Jul 28 18:53:43.336552 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:53:43.336529    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b74226_facb_49b4_ac25_48082d7ceab6.slice/crio-999121333bcd6d408e217f9c529ee61e4719cbbc32104883da223b18d2733c71 WatchSource:0}: Error finding container 999121333bcd6d408e217f9c529ee61e4719cbbc32104883da223b18d2733c71: Status 404 returned error can't find the container with id 999121333bcd6d408e217f9c529ee61e4719cbbc32104883da223b18d2733c71
Jul 28 18:53:43.804576 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:43.804538    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj" event={"ID":"c8b74226-facb-49b4-ac25-48082d7ceab6","Type":"ContainerStarted","Data":"93c73cea1db1137f769b7a391368331d50bf8932b2e933d3776b24268869a530"}
Jul 28 18:53:43.804781 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:43.804610    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"
Jul 28 18:53:43.804781 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:43.804633    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj" event={"ID":"c8b74226-facb-49b4-ac25-48082d7ceab6","Type":"ContainerStarted","Data":"999121333bcd6d408e217f9c529ee61e4719cbbc32104883da223b18d2733c71"}
Jul 28 18:53:43.806075 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:43.806047    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4" event={"ID":"3ec72956-0f6b-46ba-877c-fe8a9c80fb82","Type":"ContainerStarted","Data":"8d148c05edaf4cfac9337ec01b64899b126b93791cfde6585f84e504cde7486e"}
Jul 28 18:53:43.806186 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:43.806123    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"
Jul 28 18:53:43.820451 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:43.820404    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj" podStartSLOduration=1.820389295 podStartE2EDuration="1.820389295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:53:43.81985585 +0000 UTC m=+3159.135357984" watchObservedRunningTime="2026-07-28 18:53:43.820389295 +0000 UTC m=+3159.135891408"
Jul 28 18:53:43.838629 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:43.838575    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4" podStartSLOduration=1.408903305 podStartE2EDuration="1.83856043s" totalImagesPullingTime="429.657125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:53:43.836501081 +0000 UTC m=+3159.152003218" watchObservedRunningTime="2026-07-28 18:53:43.83856043 +0000 UTC m=+3159.154062542"
Jul 28 18:53:49.819823 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:49.819782    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"
Jul 28 18:53:54.814741 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:53:54.814712    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"
Jul 28 18:54:12.809035 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:12.809003    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"]
Jul 28 18:54:12.809614 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:12.809251    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4" podUID="3ec72956-0f6b-46ba-877c-fe8a9c80fb82" containerName="maas-api" containerID="cri-o://8d148c05edaf4cfac9337ec01b64899b126b93791cfde6585f84e504cde7486e" gracePeriod=30
Jul 28 18:54:12.855202 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:12.855169    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"]
Jul 28 18:54:12.855444 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:12.855392    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj" podUID="c8b74226-facb-49b4-ac25-48082d7ceab6" containerName="payload-processing" containerID="cri-o://93c73cea1db1137f769b7a391368331d50bf8932b2e933d3776b24268869a530" gracePeriod=30
Jul 28 18:54:12.952405 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:12.952370    2564 generic.go:356] "Generic (PLEG): container finished" podID="3ec72956-0f6b-46ba-877c-fe8a9c80fb82" containerID="8d148c05edaf4cfac9337ec01b64899b126b93791cfde6585f84e504cde7486e" exitCode=0
Jul 28 18:54:12.952562 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:12.952398    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4" event={"ID":"3ec72956-0f6b-46ba-877c-fe8a9c80fb82","Type":"ContainerDied","Data":"8d148c05edaf4cfac9337ec01b64899b126b93791cfde6585f84e504cde7486e"}
Jul 28 18:54:13.103082 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.103056    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"
Jul 28 18:54:13.134732 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.134707    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"
Jul 28 18:54:13.185105 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.185077    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54msq\" (UniqueName: \"kubernetes.io/projected/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-kube-api-access-54msq\") pod \"3ec72956-0f6b-46ba-877c-fe8a9c80fb82\" (UID: \"3ec72956-0f6b-46ba-877c-fe8a9c80fb82\") "
Jul 28 18:54:13.185248 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.185176    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-openshift-service-ca\") pod \"3ec72956-0f6b-46ba-877c-fe8a9c80fb82\" (UID: \"3ec72956-0f6b-46ba-877c-fe8a9c80fb82\") "
Jul 28 18:54:13.185248 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.185198    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-maas-api-tls\") pod \"3ec72956-0f6b-46ba-877c-fe8a9c80fb82\" (UID: \"3ec72956-0f6b-46ba-877c-fe8a9c80fb82\") "
Jul 28 18:54:13.185543 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.185518    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3ec72956-0f6b-46ba-877c-fe8a9c80fb82" (UID: "3ec72956-0f6b-46ba-877c-fe8a9c80fb82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:54:13.187253 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.187232    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-kube-api-access-54msq" (OuterVolumeSpecName: "kube-api-access-54msq") pod "3ec72956-0f6b-46ba-877c-fe8a9c80fb82" (UID: "3ec72956-0f6b-46ba-877c-fe8a9c80fb82"). InnerVolumeSpecName "kube-api-access-54msq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:54:13.187253 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.187241    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3ec72956-0f6b-46ba-877c-fe8a9c80fb82" (UID: "3ec72956-0f6b-46ba-877c-fe8a9c80fb82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:54:13.286061 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.286036    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvc8j\" (UniqueName: \"kubernetes.io/projected/c8b74226-facb-49b4-ac25-48082d7ceab6-kube-api-access-wvc8j\") pod \"c8b74226-facb-49b4-ac25-48082d7ceab6\" (UID: \"c8b74226-facb-49b4-ac25-48082d7ceab6\") "
Jul 28 18:54:13.286198 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.286128    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b74226-facb-49b4-ac25-48082d7ceab6-plugins-config-volume\") pod \"c8b74226-facb-49b4-ac25-48082d7ceab6\" (UID: \"c8b74226-facb-49b4-ac25-48082d7ceab6\") "
Jul 28 18:54:13.286285 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.286273    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:54:13.286344 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.286286    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:54:13.286344 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.286296    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54msq\" (UniqueName: \"kubernetes.io/projected/3ec72956-0f6b-46ba-877c-fe8a9c80fb82-kube-api-access-54msq\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:54:13.286467 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.286446    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b74226-facb-49b4-ac25-48082d7ceab6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c8b74226-facb-49b4-ac25-48082d7ceab6" (UID: "c8b74226-facb-49b4-ac25-48082d7ceab6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:54:13.288101 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.288079    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b74226-facb-49b4-ac25-48082d7ceab6-kube-api-access-wvc8j" (OuterVolumeSpecName: "kube-api-access-wvc8j") pod "c8b74226-facb-49b4-ac25-48082d7ceab6" (UID: "c8b74226-facb-49b4-ac25-48082d7ceab6"). InnerVolumeSpecName "kube-api-access-wvc8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:54:13.387720 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.387644    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b74226-facb-49b4-ac25-48082d7ceab6-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:54:13.387720 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.387669    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvc8j\" (UniqueName: \"kubernetes.io/projected/c8b74226-facb-49b4-ac25-48082d7ceab6-kube-api-access-wvc8j\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:54:13.958728 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.958696    2564 generic.go:356] "Generic (PLEG): container finished" podID="c8b74226-facb-49b4-ac25-48082d7ceab6" containerID="93c73cea1db1137f769b7a391368331d50bf8932b2e933d3776b24268869a530" exitCode=0
Jul 28 18:54:13.959164 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.958742    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"
Jul 28 18:54:13.959164 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.958786    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj" event={"ID":"c8b74226-facb-49b4-ac25-48082d7ceab6","Type":"ContainerDied","Data":"93c73cea1db1137f769b7a391368331d50bf8932b2e933d3776b24268869a530"}
Jul 28 18:54:13.959164 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.958851    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj" event={"ID":"c8b74226-facb-49b4-ac25-48082d7ceab6","Type":"ContainerDied","Data":"999121333bcd6d408e217f9c529ee61e4719cbbc32104883da223b18d2733c71"}
Jul 28 18:54:13.959164 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.958880    2564 scope.go:117] "RemoveContainer" containerID="93c73cea1db1137f769b7a391368331d50bf8932b2e933d3776b24268869a530"
Jul 28 18:54:13.960294 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.960278    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"
Jul 28 18:54:13.960389 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.960275    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4" event={"ID":"3ec72956-0f6b-46ba-877c-fe8a9c80fb82","Type":"ContainerDied","Data":"51afc45c6b23f782ade269a9a40d8c097fcde506c972198e7e8ce004b7b02e4e"}
Jul 28 18:54:13.967938 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.967913    2564 scope.go:117] "RemoveContainer" containerID="93c73cea1db1137f769b7a391368331d50bf8932b2e933d3776b24268869a530"
Jul 28 18:54:13.968183 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:54:13.968162    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c73cea1db1137f769b7a391368331d50bf8932b2e933d3776b24268869a530\": container with ID starting with 93c73cea1db1137f769b7a391368331d50bf8932b2e933d3776b24268869a530 not found: ID does not exist" containerID="93c73cea1db1137f769b7a391368331d50bf8932b2e933d3776b24268869a530"
Jul 28 18:54:13.968264 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.968191    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c73cea1db1137f769b7a391368331d50bf8932b2e933d3776b24268869a530"} err="failed to get container status \"93c73cea1db1137f769b7a391368331d50bf8932b2e933d3776b24268869a530\": rpc error: code = NotFound desc = could not find container \"93c73cea1db1137f769b7a391368331d50bf8932b2e933d3776b24268869a530\": container with ID starting with 93c73cea1db1137f769b7a391368331d50bf8932b2e933d3776b24268869a530 not found: ID does not exist"
Jul 28 18:54:13.968264 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.968207    2564 scope.go:117] "RemoveContainer" containerID="8d148c05edaf4cfac9337ec01b64899b126b93791cfde6585f84e504cde7486e"
Jul 28 18:54:13.981006 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.980983    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"]
Jul 28 18:54:13.984638 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.984617    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51b7e8-6ddf8d5845-7plw4"]
Jul 28 18:54:13.995169 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:13.995147    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"]
Jul 28 18:54:14.001102 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:14.001082    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-51b7e8-7f79bb45c6-2fqxj"]
Jul 28 18:54:15.216338 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:15.216307    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ec72956-0f6b-46ba-877c-fe8a9c80fb82" path="/var/lib/kubelet/pods/3ec72956-0f6b-46ba-877c-fe8a9c80fb82/volumes"
Jul 28 18:54:15.216705 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:15.216655    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8b74226-facb-49b4-ac25-48082d7ceab6" path="/var/lib/kubelet/pods/c8b74226-facb-49b4-ac25-48082d7ceab6/volumes"
Jul 28 18:54:38.689006 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:38.688971    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"]
Jul 28 18:54:38.689453 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:38.689182    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592" podUID="3b91153a-180a-4825-9b9e-445848e64752" containerName="payload-processing" containerID="cri-o://6c59433b685ad990e221b537f069c0bc09488a13ecc82cc69f515b3ad9fd6ff5" gracePeriod=30
Jul 28 18:54:38.694002 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:38.693976    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"]
Jul 28 18:54:38.694351 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:38.694328    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp" podUID="d25c1c67-fd0e-43f3-9364-571fc86526a5" containerName="payload-pre-processing" containerID="cri-o://fb3d772b3245132ceeb6353c0076d9e978915e51ef208fa75bf119aeb1804154" gracePeriod=30
Jul 28 18:54:38.970252 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:38.970228    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"
Jul 28 18:54:38.973136 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:38.973118    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"
Jul 28 18:54:39.002506 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.002483    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b91153a-180a-4825-9b9e-445848e64752-plugins-config-volume\") pod \"3b91153a-180a-4825-9b9e-445848e64752\" (UID: \"3b91153a-180a-4825-9b9e-445848e64752\") "
Jul 28 18:54:39.002619 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.002523    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25c1c67-fd0e-43f3-9364-571fc86526a5-plugins-config-volume\") pod \"d25c1c67-fd0e-43f3-9364-571fc86526a5\" (UID: \"d25c1c67-fd0e-43f3-9364-571fc86526a5\") "
Jul 28 18:54:39.002619 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.002562    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mz8c\" (UniqueName: \"kubernetes.io/projected/3b91153a-180a-4825-9b9e-445848e64752-kube-api-access-4mz8c\") pod \"3b91153a-180a-4825-9b9e-445848e64752\" (UID: \"3b91153a-180a-4825-9b9e-445848e64752\") "
Jul 28 18:54:39.002619 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.002595    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsjv9\" (UniqueName: \"kubernetes.io/projected/d25c1c67-fd0e-43f3-9364-571fc86526a5-kube-api-access-gsjv9\") pod \"d25c1c67-fd0e-43f3-9364-571fc86526a5\" (UID: \"d25c1c67-fd0e-43f3-9364-571fc86526a5\") "
Jul 28 18:54:39.002910 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.002883    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b91153a-180a-4825-9b9e-445848e64752-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b91153a-180a-4825-9b9e-445848e64752" (UID: "3b91153a-180a-4825-9b9e-445848e64752"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:54:39.002973 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.002953    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25c1c67-fd0e-43f3-9364-571fc86526a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d25c1c67-fd0e-43f3-9364-571fc86526a5" (UID: "d25c1c67-fd0e-43f3-9364-571fc86526a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:54:39.004669 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.004640    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b91153a-180a-4825-9b9e-445848e64752-kube-api-access-4mz8c" (OuterVolumeSpecName: "kube-api-access-4mz8c") pod "3b91153a-180a-4825-9b9e-445848e64752" (UID: "3b91153a-180a-4825-9b9e-445848e64752"). InnerVolumeSpecName "kube-api-access-4mz8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:54:39.004752 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.004706    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25c1c67-fd0e-43f3-9364-571fc86526a5-kube-api-access-gsjv9" (OuterVolumeSpecName: "kube-api-access-gsjv9") pod "d25c1c67-fd0e-43f3-9364-571fc86526a5" (UID: "d25c1c67-fd0e-43f3-9364-571fc86526a5"). InnerVolumeSpecName "kube-api-access-gsjv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:54:39.082606 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.082571    2564 generic.go:356] "Generic (PLEG): container finished" podID="3b91153a-180a-4825-9b9e-445848e64752" containerID="6c59433b685ad990e221b537f069c0bc09488a13ecc82cc69f515b3ad9fd6ff5" exitCode=0
Jul 28 18:54:39.082727 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.082614    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"
Jul 28 18:54:39.082727 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.082662    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592" event={"ID":"3b91153a-180a-4825-9b9e-445848e64752","Type":"ContainerDied","Data":"6c59433b685ad990e221b537f069c0bc09488a13ecc82cc69f515b3ad9fd6ff5"}
Jul 28 18:54:39.082727 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.082710    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592" event={"ID":"3b91153a-180a-4825-9b9e-445848e64752","Type":"ContainerDied","Data":"76232eb02f49f37475c9fcf5b5853e6f16f70dc32998fa6a63252840865b44bf"}
Jul 28 18:54:39.082936 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.082738    2564 scope.go:117] "RemoveContainer" containerID="6c59433b685ad990e221b537f069c0bc09488a13ecc82cc69f515b3ad9fd6ff5"
Jul 28 18:54:39.084141 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.084120    2564 generic.go:356] "Generic (PLEG): container finished" podID="d25c1c67-fd0e-43f3-9364-571fc86526a5" containerID="fb3d772b3245132ceeb6353c0076d9e978915e51ef208fa75bf119aeb1804154" exitCode=0
Jul 28 18:54:39.084258 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.084156    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp" event={"ID":"d25c1c67-fd0e-43f3-9364-571fc86526a5","Type":"ContainerDied","Data":"fb3d772b3245132ceeb6353c0076d9e978915e51ef208fa75bf119aeb1804154"}
Jul 28 18:54:39.084258 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.084168    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"
Jul 28 18:54:39.084258 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.084192    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp" event={"ID":"d25c1c67-fd0e-43f3-9364-571fc86526a5","Type":"ContainerDied","Data":"155a38c98dfecf588c5944e775afb6c23dbef17df62e9fdad2703809196d08dd"}
Jul 28 18:54:39.101302 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.101283    2564 scope.go:117] "RemoveContainer" containerID="6c59433b685ad990e221b537f069c0bc09488a13ecc82cc69f515b3ad9fd6ff5"
Jul 28 18:54:39.101577 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:54:39.101556    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c59433b685ad990e221b537f069c0bc09488a13ecc82cc69f515b3ad9fd6ff5\": container with ID starting with 6c59433b685ad990e221b537f069c0bc09488a13ecc82cc69f515b3ad9fd6ff5 not found: ID does not exist" containerID="6c59433b685ad990e221b537f069c0bc09488a13ecc82cc69f515b3ad9fd6ff5"
Jul 28 18:54:39.101638 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.101584    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c59433b685ad990e221b537f069c0bc09488a13ecc82cc69f515b3ad9fd6ff5"} err="failed to get container status \"6c59433b685ad990e221b537f069c0bc09488a13ecc82cc69f515b3ad9fd6ff5\": rpc error: code = NotFound desc = could not find container \"6c59433b685ad990e221b537f069c0bc09488a13ecc82cc69f515b3ad9fd6ff5\": container with ID starting with 6c59433b685ad990e221b537f069c0bc09488a13ecc82cc69f515b3ad9fd6ff5 not found: ID does not exist"
Jul 28 18:54:39.101638 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.101602    2564 scope.go:117] "RemoveContainer" containerID="fb3d772b3245132ceeb6353c0076d9e978915e51ef208fa75bf119aeb1804154"
Jul 28 18:54:39.103824 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.103788    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsjv9\" (UniqueName: \"kubernetes.io/projected/d25c1c67-fd0e-43f3-9364-571fc86526a5-kube-api-access-gsjv9\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:54:39.103903 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.103869    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b91153a-180a-4825-9b9e-445848e64752-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:54:39.103903 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.103888    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25c1c67-fd0e-43f3-9364-571fc86526a5-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:54:39.103903 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.103897    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mz8c\" (UniqueName: \"kubernetes.io/projected/3b91153a-180a-4825-9b9e-445848e64752-kube-api-access-4mz8c\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:54:39.109843 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.109799    2564 scope.go:117] "RemoveContainer" containerID="fb3d772b3245132ceeb6353c0076d9e978915e51ef208fa75bf119aeb1804154"
Jul 28 18:54:39.110148 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:54:39.110123    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3d772b3245132ceeb6353c0076d9e978915e51ef208fa75bf119aeb1804154\": container with ID starting with fb3d772b3245132ceeb6353c0076d9e978915e51ef208fa75bf119aeb1804154 not found: ID does not exist" containerID="fb3d772b3245132ceeb6353c0076d9e978915e51ef208fa75bf119aeb1804154"
Jul 28 18:54:39.110224 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.110157    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3d772b3245132ceeb6353c0076d9e978915e51ef208fa75bf119aeb1804154"} err="failed to get container status \"fb3d772b3245132ceeb6353c0076d9e978915e51ef208fa75bf119aeb1804154\": rpc error: code = NotFound desc = could not find container \"fb3d772b3245132ceeb6353c0076d9e978915e51ef208fa75bf119aeb1804154\": container with ID starting with fb3d772b3245132ceeb6353c0076d9e978915e51ef208fa75bf119aeb1804154 not found: ID does not exist"
Jul 28 18:54:39.110409 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.110394    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"]
Jul 28 18:54:39.112122 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.112104    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c1237e-749786ff8d-22592"]
Jul 28 18:54:39.121983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.121962    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"]
Jul 28 18:54:39.125430 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.125412    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c1237e-7d4bfcbd4-mtqhp"]
Jul 28 18:54:39.217295 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.217202    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b91153a-180a-4825-9b9e-445848e64752" path="/var/lib/kubelet/pods/3b91153a-180a-4825-9b9e-445848e64752/volumes"
Jul 28 18:54:39.217660 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:39.217639    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d25c1c67-fd0e-43f3-9364-571fc86526a5" path="/var/lib/kubelet/pods/d25c1c67-fd0e-43f3-9364-571fc86526a5/volumes"
Jul 28 18:54:44.214322 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:44.214288    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"]
Jul 28 18:54:59.233113 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.233009    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565" podUID="db3167a8-ea04-4d63-9244-016c165243d7" containerName="main" containerID="cri-o://a6021bb1968ad22bda468d02741b330c3305599d2b630df17e3031c0ecf2dd7c" gracePeriod=45
Jul 28 18:54:59.473130 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.473106    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:54:59.584968 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.584941    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db3167a8-ea04-4d63-9244-016c165243d7-tls-certs\") pod \"db3167a8-ea04-4d63-9244-016c165243d7\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") "
Jul 28 18:54:59.585139 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.584997    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-tmp-dir\") pod \"db3167a8-ea04-4d63-9244-016c165243d7\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") "
Jul 28 18:54:59.585139 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.585020    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-home\") pod \"db3167a8-ea04-4d63-9244-016c165243d7\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") "
Jul 28 18:54:59.585139 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.585089    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-dshm\") pod \"db3167a8-ea04-4d63-9244-016c165243d7\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") "
Jul 28 18:54:59.585139 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.585108    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-model-cache\") pod \"db3167a8-ea04-4d63-9244-016c165243d7\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") "
Jul 28 18:54:59.585365 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.585144    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x5tk\" (UniqueName: \"kubernetes.io/projected/db3167a8-ea04-4d63-9244-016c165243d7-kube-api-access-5x5tk\") pod \"db3167a8-ea04-4d63-9244-016c165243d7\" (UID: \"db3167a8-ea04-4d63-9244-016c165243d7\") "
Jul 28 18:54:59.585365 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.585284    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db3167a8-ea04-4d63-9244-016c165243d7" (UID: "db3167a8-ea04-4d63-9244-016c165243d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:54:59.585365 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.585344    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-home" (OuterVolumeSpecName: "home") pod "db3167a8-ea04-4d63-9244-016c165243d7" (UID: "db3167a8-ea04-4d63-9244-016c165243d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:54:59.585497 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.585364    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "db3167a8-ea04-4d63-9244-016c165243d7" (UID: "db3167a8-ea04-4d63-9244-016c165243d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:54:59.585497 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.585447    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-tmp-dir\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:54:59.585497 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.585463    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-home\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:54:59.585497 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.585476    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-model-cache\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:54:59.587235 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.587204    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3167a8-ea04-4d63-9244-016c165243d7-kube-api-access-5x5tk" (OuterVolumeSpecName: "kube-api-access-5x5tk") pod "db3167a8-ea04-4d63-9244-016c165243d7" (UID: "db3167a8-ea04-4d63-9244-016c165243d7"). InnerVolumeSpecName "kube-api-access-5x5tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:54:59.587342 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.587208    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3167a8-ea04-4d63-9244-016c165243d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db3167a8-ea04-4d63-9244-016c165243d7" (UID: "db3167a8-ea04-4d63-9244-016c165243d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:54:59.587342 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.587275    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-dshm" (OuterVolumeSpecName: "dshm") pod "db3167a8-ea04-4d63-9244-016c165243d7" (UID: "db3167a8-ea04-4d63-9244-016c165243d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:54:59.686416 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.686373    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db3167a8-ea04-4d63-9244-016c165243d7-tls-certs\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:54:59.686416 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.686410    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db3167a8-ea04-4d63-9244-016c165243d7-dshm\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:54:59.686416 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:54:59.686421    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x5tk\" (UniqueName: \"kubernetes.io/projected/db3167a8-ea04-4d63-9244-016c165243d7-kube-api-access-5x5tk\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:55:00.188915 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:55:00.188881    2564 generic.go:356] "Generic (PLEG): container finished" podID="db3167a8-ea04-4d63-9244-016c165243d7" containerID="a6021bb1968ad22bda468d02741b330c3305599d2b630df17e3031c0ecf2dd7c" exitCode=0
Jul 28 18:55:00.189119 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:55:00.188923    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"
Jul 28 18:55:00.189119 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:55:00.188965    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565" event={"ID":"db3167a8-ea04-4d63-9244-016c165243d7","Type":"ContainerDied","Data":"a6021bb1968ad22bda468d02741b330c3305599d2b630df17e3031c0ecf2dd7c"}
Jul 28 18:55:00.189119 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:55:00.189007    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565" event={"ID":"db3167a8-ea04-4d63-9244-016c165243d7","Type":"ContainerDied","Data":"1187a0ffd1f6ce7d894725515ac0e05f348ff6d9137817e592949864df0d60fe"}
Jul 28 18:55:00.189119 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:55:00.189030    2564 scope.go:117] "RemoveContainer" containerID="a6021bb1968ad22bda468d02741b330c3305599d2b630df17e3031c0ecf2dd7c"
Jul 28 18:55:00.197790 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:55:00.197774    2564 scope.go:117] "RemoveContainer" containerID="a6021bb1968ad22bda468d02741b330c3305599d2b630df17e3031c0ecf2dd7c"
Jul 28 18:55:00.198062 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:55:00.198043    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6021bb1968ad22bda468d02741b330c3305599d2b630df17e3031c0ecf2dd7c\": container with ID starting with a6021bb1968ad22bda468d02741b330c3305599d2b630df17e3031c0ecf2dd7c not found: ID does not exist" containerID="a6021bb1968ad22bda468d02741b330c3305599d2b630df17e3031c0ecf2dd7c"
Jul 28 18:55:00.198111 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:55:00.198071    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6021bb1968ad22bda468d02741b330c3305599d2b630df17e3031c0ecf2dd7c"} err="failed to get container status \"a6021bb1968ad22bda468d02741b330c3305599d2b630df17e3031c0ecf2dd7c\": rpc error: code = NotFound desc = could not find container \"a6021bb1968ad22bda468d02741b330c3305599d2b630df17e3031c0ecf2dd7c\": container with ID starting with a6021bb1968ad22bda468d02741b330c3305599d2b630df17e3031c0ecf2dd7c not found: ID does not exist"
Jul 28 18:55:00.209456 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:55:00.209432    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"]
Jul 28 18:55:00.212920 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:55:00.212899    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-c1237e/ipp-route-c1237e-kserve-645bd9898c-h7565"]
Jul 28 18:55:01.216092 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:55:01.216061    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db3167a8-ea04-4d63-9244-016c165243d7" path="/var/lib/kubelet/pods/db3167a8-ea04-4d63-9244-016c165243d7/volumes"
Jul 28 18:56:01.199193 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.199159    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"]
Jul 28 18:56:01.199728 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.199366    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f" podUID="54e294e3-c8ae-4033-8ebf-bac42d3bbd81" containerName="maas-api" containerID="cri-o://570a1a108ccb6804b3deb3b36ac2c268067e932a767e65e1769328f102328d2c" gracePeriod=30
Jul 28 18:56:01.258942 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.258902    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"]
Jul 28 18:56:01.259128 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.259102    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw" podUID="7460543a-f55b-4b52-84a0-40d0ea267d97" containerName="payload-pre-processing" containerID="cri-o://4f6923958a6850fa731feae71b07569e48cbee9b7e985783e3380ea944e613c9" gracePeriod=30
Jul 28 18:56:01.465838 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.465788    2564 generic.go:356] "Generic (PLEG): container finished" podID="7460543a-f55b-4b52-84a0-40d0ea267d97" containerID="4f6923958a6850fa731feae71b07569e48cbee9b7e985783e3380ea944e613c9" exitCode=0
Jul 28 18:56:01.465983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.465854    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw" event={"ID":"7460543a-f55b-4b52-84a0-40d0ea267d97","Type":"ContainerDied","Data":"4f6923958a6850fa731feae71b07569e48cbee9b7e985783e3380ea944e613c9"}
Jul 28 18:56:01.467584 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.467559    2564 generic.go:356] "Generic (PLEG): container finished" podID="54e294e3-c8ae-4033-8ebf-bac42d3bbd81" containerID="570a1a108ccb6804b3deb3b36ac2c268067e932a767e65e1769328f102328d2c" exitCode=0
Jul 28 18:56:01.467718 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.467635    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f" event={"ID":"54e294e3-c8ae-4033-8ebf-bac42d3bbd81","Type":"ContainerDied","Data":"570a1a108ccb6804b3deb3b36ac2c268067e932a767e65e1769328f102328d2c"}
Jul 28 18:56:01.506588 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.506563    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"
Jul 28 18:56:01.544494 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.544464    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"
Jul 28 18:56:01.597777 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.597751    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-maas-api-tls\") pod \"54e294e3-c8ae-4033-8ebf-bac42d3bbd81\" (UID: \"54e294e3-c8ae-4033-8ebf-bac42d3bbd81\") "
Jul 28 18:56:01.597971 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.597843    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-openshift-service-ca\") pod \"54e294e3-c8ae-4033-8ebf-bac42d3bbd81\" (UID: \"54e294e3-c8ae-4033-8ebf-bac42d3bbd81\") "
Jul 28 18:56:01.597971 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.597904    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2gwl\" (UniqueName: \"kubernetes.io/projected/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-kube-api-access-g2gwl\") pod \"54e294e3-c8ae-4033-8ebf-bac42d3bbd81\" (UID: \"54e294e3-c8ae-4033-8ebf-bac42d3bbd81\") "
Jul 28 18:56:01.598208 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.598175    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "54e294e3-c8ae-4033-8ebf-bac42d3bbd81" (UID: "54e294e3-c8ae-4033-8ebf-bac42d3bbd81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:56:01.599855 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.599831    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "54e294e3-c8ae-4033-8ebf-bac42d3bbd81" (UID: "54e294e3-c8ae-4033-8ebf-bac42d3bbd81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:56:01.599939 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.599879    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-kube-api-access-g2gwl" (OuterVolumeSpecName: "kube-api-access-g2gwl") pod "54e294e3-c8ae-4033-8ebf-bac42d3bbd81" (UID: "54e294e3-c8ae-4033-8ebf-bac42d3bbd81"). InnerVolumeSpecName "kube-api-access-g2gwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:56:01.698707 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.698655    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfjbk\" (UniqueName: \"kubernetes.io/projected/7460543a-f55b-4b52-84a0-40d0ea267d97-kube-api-access-hfjbk\") pod \"7460543a-f55b-4b52-84a0-40d0ea267d97\" (UID: \"7460543a-f55b-4b52-84a0-40d0ea267d97\") "
Jul 28 18:56:01.698707 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.698713    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7460543a-f55b-4b52-84a0-40d0ea267d97-plugins-config-volume\") pod \"7460543a-f55b-4b52-84a0-40d0ea267d97\" (UID: \"7460543a-f55b-4b52-84a0-40d0ea267d97\") "
Jul 28 18:56:01.699031 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.699008    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2gwl\" (UniqueName: \"kubernetes.io/projected/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-kube-api-access-g2gwl\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.699077 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.699038    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.699077 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.699054    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e294e3-c8ae-4033-8ebf-bac42d3bbd81-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.699148 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.699134    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7460543a-f55b-4b52-84a0-40d0ea267d97-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7460543a-f55b-4b52-84a0-40d0ea267d97" (UID: "7460543a-f55b-4b52-84a0-40d0ea267d97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:56:01.700760 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.700737    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7460543a-f55b-4b52-84a0-40d0ea267d97-kube-api-access-hfjbk" (OuterVolumeSpecName: "kube-api-access-hfjbk") pod "7460543a-f55b-4b52-84a0-40d0ea267d97" (UID: "7460543a-f55b-4b52-84a0-40d0ea267d97"). InnerVolumeSpecName "kube-api-access-hfjbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:56:01.800374 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.800334    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfjbk\" (UniqueName: \"kubernetes.io/projected/7460543a-f55b-4b52-84a0-40d0ea267d97-kube-api-access-hfjbk\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.800374 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:01.800371    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7460543a-f55b-4b52-84a0-40d0ea267d97-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:02.473695 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:02.473622    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"
Jul 28 18:56:02.474106 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:02.473619    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw" event={"ID":"7460543a-f55b-4b52-84a0-40d0ea267d97","Type":"ContainerDied","Data":"1d12cbabc754022b5ed6cabbe9b139f9ccd43131517208a8e4c8315a6d113fce"}
Jul 28 18:56:02.474106 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:02.473747    2564 scope.go:117] "RemoveContainer" containerID="4f6923958a6850fa731feae71b07569e48cbee9b7e985783e3380ea944e613c9"
Jul 28 18:56:02.475198 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:02.475179    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f" event={"ID":"54e294e3-c8ae-4033-8ebf-bac42d3bbd81","Type":"ContainerDied","Data":"fecb9a0c900d3137db613ac7893ee6c854fcfc1c9af3a5f9c1bc211d9126c909"}
Jul 28 18:56:02.475293 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:02.475186    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"
Jul 28 18:56:02.483037 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:02.483020    2564 scope.go:117] "RemoveContainer" containerID="570a1a108ccb6804b3deb3b36ac2c268067e932a767e65e1769328f102328d2c"
Jul 28 18:56:02.497755 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:02.497718    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"]
Jul 28 18:56:02.502902 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:02.502882    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae953-68d9c6c8fb-h89zw"]
Jul 28 18:56:02.512589 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:02.512565    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"]
Jul 28 18:56:02.514765 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:02.514744    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae953-7d4f57dfd-hrf9f"]
Jul 28 18:56:03.216016 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:03.215987    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54e294e3-c8ae-4033-8ebf-bac42d3bbd81" path="/var/lib/kubelet/pods/54e294e3-c8ae-4033-8ebf-bac42d3bbd81/volumes"
Jul 28 18:56:03.216338 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:03.216326    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7460543a-f55b-4b52-84a0-40d0ea267d97" path="/var/lib/kubelet/pods/7460543a-f55b-4b52-84a0-40d0ea267d97/volumes"
Jul 28 18:56:08.295716 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.295675    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf"]
Jul 28 18:56:08.296313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296138    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d25c1c67-fd0e-43f3-9364-571fc86526a5" containerName="payload-pre-processing"
Jul 28 18:56:08.296313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296157    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d25c1c67-fd0e-43f3-9364-571fc86526a5" containerName="payload-pre-processing"
Jul 28 18:56:08.296313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296168    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db3167a8-ea04-4d63-9244-016c165243d7" containerName="main"
Jul 28 18:56:08.296313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296177    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="db3167a8-ea04-4d63-9244-016c165243d7" containerName="main"
Jul 28 18:56:08.296313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296192    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54e294e3-c8ae-4033-8ebf-bac42d3bbd81" containerName="maas-api"
Jul 28 18:56:08.296313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296203    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="54e294e3-c8ae-4033-8ebf-bac42d3bbd81" containerName="maas-api"
Jul 28 18:56:08.296313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296222    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec72956-0f6b-46ba-877c-fe8a9c80fb82" containerName="maas-api"
Jul 28 18:56:08.296313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296230    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec72956-0f6b-46ba-877c-fe8a9c80fb82" containerName="maas-api"
Jul 28 18:56:08.296313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296244    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7460543a-f55b-4b52-84a0-40d0ea267d97" containerName="payload-pre-processing"
Jul 28 18:56:08.296313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296254    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7460543a-f55b-4b52-84a0-40d0ea267d97" containerName="payload-pre-processing"
Jul 28 18:56:08.296313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296273    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b91153a-180a-4825-9b9e-445848e64752" containerName="payload-processing"
Jul 28 18:56:08.296313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296281    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b91153a-180a-4825-9b9e-445848e64752" containerName="payload-processing"
Jul 28 18:56:08.296313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296292    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8b74226-facb-49b4-ac25-48082d7ceab6" containerName="payload-processing"
Jul 28 18:56:08.296313 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296301    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8b74226-facb-49b4-ac25-48082d7ceab6" containerName="payload-processing"
Jul 28 18:56:08.297042 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296381    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="db3167a8-ea04-4d63-9244-016c165243d7" containerName="main"
Jul 28 18:56:08.297042 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296394    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d25c1c67-fd0e-43f3-9364-571fc86526a5" containerName="payload-pre-processing"
Jul 28 18:56:08.297042 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296404    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8b74226-facb-49b4-ac25-48082d7ceab6" containerName="payload-processing"
Jul 28 18:56:08.297042 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296414    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7460543a-f55b-4b52-84a0-40d0ea267d97" containerName="payload-pre-processing"
Jul 28 18:56:08.297042 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296423    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec72956-0f6b-46ba-877c-fe8a9c80fb82" containerName="maas-api"
Jul 28 18:56:08.297042 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296432    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="54e294e3-c8ae-4033-8ebf-bac42d3bbd81" containerName="maas-api"
Jul 28 18:56:08.297042 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.296440    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b91153a-180a-4825-9b9e-445848e64752" containerName="payload-processing"
Jul 28 18:56:08.301233 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.301210    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf"
Jul 28 18:56:08.304108 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.304081    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\""
Jul 28 18:56:08.309947 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.309921    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf"]
Jul 28 18:56:08.458655 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.458616    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00c52ee-6690-4343-afd1-25776c9f9514-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf\" (UID: \"d00c52ee-6690-4343-afd1-25776c9f9514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf"
Jul 28 18:56:08.559933 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.559847    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00c52ee-6690-4343-afd1-25776c9f9514-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf\" (UID: \"d00c52ee-6690-4343-afd1-25776c9f9514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf"
Jul 28 18:56:08.560461 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.560440    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00c52ee-6690-4343-afd1-25776c9f9514-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf\" (UID: \"d00c52ee-6690-4343-afd1-25776c9f9514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf"
Jul 28 18:56:08.613847 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.613821    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf"
Jul 28 18:56:08.744837 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:08.744791    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf"]
Jul 28 18:56:08.746818 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:56:08.746780    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00c52ee_6690_4343_afd1_25776c9f9514.slice/crio-f8d9711b80d5724533c7eefa6db939dddcf8722417842d927011a4543d8ddf61 WatchSource:0}: Error finding container f8d9711b80d5724533c7eefa6db939dddcf8722417842d927011a4543d8ddf61: Status 404 returned error can't find the container with id f8d9711b80d5724533c7eefa6db939dddcf8722417842d927011a4543d8ddf61
Jul 28 18:56:09.509857 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:09.509832    2564 generic.go:356] "Generic (PLEG): container finished" podID="d00c52ee-6690-4343-afd1-25776c9f9514" containerID="d56161972f1f47a9139411ebdd5c3d2bdf3b4d005bf9f76afdfb0adbb602fc37" exitCode=0
Jul 28 18:56:09.510220 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:09.509943    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf" event={"ID":"d00c52ee-6690-4343-afd1-25776c9f9514","Type":"ContainerDied","Data":"d56161972f1f47a9139411ebdd5c3d2bdf3b4d005bf9f76afdfb0adbb602fc37"}
Jul 28 18:56:09.510220 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:09.509980    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf" event={"ID":"d00c52ee-6690-4343-afd1-25776c9f9514","Type":"ContainerStarted","Data":"f8d9711b80d5724533c7eefa6db939dddcf8722417842d927011a4543d8ddf61"}
Jul 28 18:56:10.644386 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:10.644361    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf"
Jul 28 18:56:10.782291 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:10.782193    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00c52ee-6690-4343-afd1-25776c9f9514-maas-api-service-ca\") pod \"d00c52ee-6690-4343-afd1-25776c9f9514\" (UID: \"d00c52ee-6690-4343-afd1-25776c9f9514\") "
Jul 28 18:56:10.782576 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:10.782551    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00c52ee-6690-4343-afd1-25776c9f9514-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d00c52ee-6690-4343-afd1-25776c9f9514" (UID: "d00c52ee-6690-4343-afd1-25776c9f9514"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:56:10.883624 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:10.883592    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00c52ee-6690-4343-afd1-25776c9f9514-maas-api-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:11.520346 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:11.520320    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf"
Jul 28 18:56:11.520503 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:11.520317    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf" event={"ID":"d00c52ee-6690-4343-afd1-25776c9f9514","Type":"ContainerDied","Data":"f8d9711b80d5724533c7eefa6db939dddcf8722417842d927011a4543d8ddf61"}
Jul 28 18:56:11.520503 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:11.520470    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8d9711b80d5724533c7eefa6db939dddcf8722417842d927011a4543d8ddf61"
Jul 28 18:56:18.712048 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:18.712018    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"]
Jul 28 18:56:18.712480 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:18.712256    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8" podUID="a5f538fb-a15e-4b81-812e-5091d8475ca9" containerName="maas-api" containerID="cri-o://d2c6dc71182f8182f5d0993420f0fbd29073eac19d8bb1a9690abd5bd963cd5d" gracePeriod=30
Jul 28 18:56:18.769555 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:18.769525    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"]
Jul 28 18:56:18.769773 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:18.769744    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9" podUID="68309e6c-2f60-4fb6-ba40-bfda26857446" containerName="payload-processing" containerID="cri-o://094cfbfd5d0fd124570b0cd7d089f0eae00c5207daad406aa2786cc33a383174" gracePeriod=30
Jul 28 18:56:18.774646 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:18.774562    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"]
Jul 28 18:56:18.774842 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:18.774798    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt" podUID="3d1e757b-a627-4946-9184-7e6633fe0716" containerName="payload-pre-processing" containerID="cri-o://d5ebc8b935951cf1b5ace7e282680b8a18230476df9d8af79625785d3eaf8ed4" gracePeriod=30
Jul 28 18:56:18.779638 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:18.779488    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt" podUID="3d1e757b-a627-4946-9184-7e6633fe0716" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.68:9004: connect: connection refused"
Jul 28 18:56:18.779638 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:18.779528    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9" podUID="68309e6c-2f60-4fb6-ba40-bfda26857446" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.69:9004: connect: connection refused"
Jul 28 18:56:18.779638 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:18.779575    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8" podUID="a5f538fb-a15e-4b81-812e-5091d8475ca9" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.67:8443/health\": dial tcp 10.134.0.67:8443: connect: connection refused"
Jul 28 18:56:18.882046 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:18.881891    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf"]
Jul 28 18:56:18.883935 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:18.883914    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-37e11d-m49rf"]
Jul 28 18:56:19.027695 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.027665    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"
Jul 28 18:56:19.031012 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.030991    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"
Jul 28 18:56:19.054047 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.054020    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcv26\" (UniqueName: \"kubernetes.io/projected/a5f538fb-a15e-4b81-812e-5091d8475ca9-kube-api-access-wcv26\") pod \"a5f538fb-a15e-4b81-812e-5091d8475ca9\" (UID: \"a5f538fb-a15e-4b81-812e-5091d8475ca9\") "
Jul 28 18:56:19.054228 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.054066    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68309e6c-2f60-4fb6-ba40-bfda26857446-plugins-config-volume\") pod \"68309e6c-2f60-4fb6-ba40-bfda26857446\" (UID: \"68309e6c-2f60-4fb6-ba40-bfda26857446\") "
Jul 28 18:56:19.054228 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.054102    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f538fb-a15e-4b81-812e-5091d8475ca9-openshift-service-ca\") pod \"a5f538fb-a15e-4b81-812e-5091d8475ca9\" (UID: \"a5f538fb-a15e-4b81-812e-5091d8475ca9\") "
Jul 28 18:56:19.054228 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.054201    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f538fb-a15e-4b81-812e-5091d8475ca9-maas-api-tls\") pod \"a5f538fb-a15e-4b81-812e-5091d8475ca9\" (UID: \"a5f538fb-a15e-4b81-812e-5091d8475ca9\") "
Jul 28 18:56:19.054228 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.054225    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsq88\" (UniqueName: \"kubernetes.io/projected/68309e6c-2f60-4fb6-ba40-bfda26857446-kube-api-access-lsq88\") pod \"68309e6c-2f60-4fb6-ba40-bfda26857446\" (UID: \"68309e6c-2f60-4fb6-ba40-bfda26857446\") "
Jul 28 18:56:19.054549 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.054481    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68309e6c-2f60-4fb6-ba40-bfda26857446-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68309e6c-2f60-4fb6-ba40-bfda26857446" (UID: "68309e6c-2f60-4fb6-ba40-bfda26857446"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:56:19.054794 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.054757    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f538fb-a15e-4b81-812e-5091d8475ca9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a5f538fb-a15e-4b81-812e-5091d8475ca9" (UID: "a5f538fb-a15e-4b81-812e-5091d8475ca9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:56:19.056427 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.056400    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f538fb-a15e-4b81-812e-5091d8475ca9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a5f538fb-a15e-4b81-812e-5091d8475ca9" (UID: "a5f538fb-a15e-4b81-812e-5091d8475ca9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:56:19.056511 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.056439    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68309e6c-2f60-4fb6-ba40-bfda26857446-kube-api-access-lsq88" (OuterVolumeSpecName: "kube-api-access-lsq88") pod "68309e6c-2f60-4fb6-ba40-bfda26857446" (UID: "68309e6c-2f60-4fb6-ba40-bfda26857446"). InnerVolumeSpecName "kube-api-access-lsq88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:56:19.056576 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.056562    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f538fb-a15e-4b81-812e-5091d8475ca9-kube-api-access-wcv26" (OuterVolumeSpecName: "kube-api-access-wcv26") pod "a5f538fb-a15e-4b81-812e-5091d8475ca9" (UID: "a5f538fb-a15e-4b81-812e-5091d8475ca9"). InnerVolumeSpecName "kube-api-access-wcv26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:56:19.063291 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.063264    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"
Jul 28 18:56:19.155473 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.155424    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1e757b-a627-4946-9184-7e6633fe0716-plugins-config-volume\") pod \"3d1e757b-a627-4946-9184-7e6633fe0716\" (UID: \"3d1e757b-a627-4946-9184-7e6633fe0716\") "
Jul 28 18:56:19.155642 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.155543    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wxtw\" (UniqueName: \"kubernetes.io/projected/3d1e757b-a627-4946-9184-7e6633fe0716-kube-api-access-4wxtw\") pod \"3d1e757b-a627-4946-9184-7e6633fe0716\" (UID: \"3d1e757b-a627-4946-9184-7e6633fe0716\") "
Jul 28 18:56:19.155729 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.155713    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f538fb-a15e-4b81-812e-5091d8475ca9-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:19.155766 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.155738    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsq88\" (UniqueName: \"kubernetes.io/projected/68309e6c-2f60-4fb6-ba40-bfda26857446-kube-api-access-lsq88\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:19.155766 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.155752    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcv26\" (UniqueName: \"kubernetes.io/projected/a5f538fb-a15e-4b81-812e-5091d8475ca9-kube-api-access-wcv26\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:19.155867 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.155765    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68309e6c-2f60-4fb6-ba40-bfda26857446-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:19.155867 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.155778    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f538fb-a15e-4b81-812e-5091d8475ca9-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:19.155867 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.155801    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1e757b-a627-4946-9184-7e6633fe0716-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d1e757b-a627-4946-9184-7e6633fe0716" (UID: "3d1e757b-a627-4946-9184-7e6633fe0716"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:56:19.157561 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.157541    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1e757b-a627-4946-9184-7e6633fe0716-kube-api-access-4wxtw" (OuterVolumeSpecName: "kube-api-access-4wxtw") pod "3d1e757b-a627-4946-9184-7e6633fe0716" (UID: "3d1e757b-a627-4946-9184-7e6633fe0716"). InnerVolumeSpecName "kube-api-access-4wxtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:56:19.217116 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.217084    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d00c52ee-6690-4343-afd1-25776c9f9514" path="/var/lib/kubelet/pods/d00c52ee-6690-4343-afd1-25776c9f9514/volumes"
Jul 28 18:56:19.256314 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.256289    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wxtw\" (UniqueName: \"kubernetes.io/projected/3d1e757b-a627-4946-9184-7e6633fe0716-kube-api-access-4wxtw\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:19.256450 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.256317    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1e757b-a627-4946-9184-7e6633fe0716-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:19.558113 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.558029    2564 generic.go:356] "Generic (PLEG): container finished" podID="a5f538fb-a15e-4b81-812e-5091d8475ca9" containerID="d2c6dc71182f8182f5d0993420f0fbd29073eac19d8bb1a9690abd5bd963cd5d" exitCode=0
Jul 28 18:56:19.558113 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.558079    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"
Jul 28 18:56:19.558334 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.558117    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8" event={"ID":"a5f538fb-a15e-4b81-812e-5091d8475ca9","Type":"ContainerDied","Data":"d2c6dc71182f8182f5d0993420f0fbd29073eac19d8bb1a9690abd5bd963cd5d"}
Jul 28 18:56:19.558334 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.558173    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8" event={"ID":"a5f538fb-a15e-4b81-812e-5091d8475ca9","Type":"ContainerDied","Data":"b172e3419605a8505abd62c030eba549b80a8f6153e2161d890571e5678a4eca"}
Jul 28 18:56:19.558334 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.558203    2564 scope.go:117] "RemoveContainer" containerID="d2c6dc71182f8182f5d0993420f0fbd29073eac19d8bb1a9690abd5bd963cd5d"
Jul 28 18:56:19.559490 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.559471    2564 generic.go:356] "Generic (PLEG): container finished" podID="68309e6c-2f60-4fb6-ba40-bfda26857446" containerID="094cfbfd5d0fd124570b0cd7d089f0eae00c5207daad406aa2786cc33a383174" exitCode=0
Jul 28 18:56:19.559611 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.559504    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9" event={"ID":"68309e6c-2f60-4fb6-ba40-bfda26857446","Type":"ContainerDied","Data":"094cfbfd5d0fd124570b0cd7d089f0eae00c5207daad406aa2786cc33a383174"}
Jul 28 18:56:19.559611 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.559522    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"
Jul 28 18:56:19.559611 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.559561    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9" event={"ID":"68309e6c-2f60-4fb6-ba40-bfda26857446","Type":"ContainerDied","Data":"45f866802c0753b22c1631aafebded2f9863d1106cb67631ac77d56259582878"}
Jul 28 18:56:19.561083 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.561057    2564 generic.go:356] "Generic (PLEG): container finished" podID="3d1e757b-a627-4946-9184-7e6633fe0716" containerID="d5ebc8b935951cf1b5ace7e282680b8a18230476df9d8af79625785d3eaf8ed4" exitCode=0
Jul 28 18:56:19.561195 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.561098    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"
Jul 28 18:56:19.561269 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.561100    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt" event={"ID":"3d1e757b-a627-4946-9184-7e6633fe0716","Type":"ContainerDied","Data":"d5ebc8b935951cf1b5ace7e282680b8a18230476df9d8af79625785d3eaf8ed4"}
Jul 28 18:56:19.561269 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.561245    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt" event={"ID":"3d1e757b-a627-4946-9184-7e6633fe0716","Type":"ContainerDied","Data":"9c446904a3e9503e5509705ede5498abc39c5f5b4e4c263195c14f97a8e14a9e"}
Jul 28 18:56:19.569602 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.569573    2564 scope.go:117] "RemoveContainer" containerID="d2c6dc71182f8182f5d0993420f0fbd29073eac19d8bb1a9690abd5bd963cd5d"
Jul 28 18:56:19.569891 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:56:19.569864    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c6dc71182f8182f5d0993420f0fbd29073eac19d8bb1a9690abd5bd963cd5d\": container with ID starting with d2c6dc71182f8182f5d0993420f0fbd29073eac19d8bb1a9690abd5bd963cd5d not found: ID does not exist" containerID="d2c6dc71182f8182f5d0993420f0fbd29073eac19d8bb1a9690abd5bd963cd5d"
Jul 28 18:56:19.569973 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.569903    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c6dc71182f8182f5d0993420f0fbd29073eac19d8bb1a9690abd5bd963cd5d"} err="failed to get container status \"d2c6dc71182f8182f5d0993420f0fbd29073eac19d8bb1a9690abd5bd963cd5d\": rpc error: code = NotFound desc = could not find container \"d2c6dc71182f8182f5d0993420f0fbd29073eac19d8bb1a9690abd5bd963cd5d\": container with ID starting with d2c6dc71182f8182f5d0993420f0fbd29073eac19d8bb1a9690abd5bd963cd5d not found: ID does not exist"
Jul 28 18:56:19.569973 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.569926    2564 scope.go:117] "RemoveContainer" containerID="094cfbfd5d0fd124570b0cd7d089f0eae00c5207daad406aa2786cc33a383174"
Jul 28 18:56:19.577222 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.577205    2564 scope.go:117] "RemoveContainer" containerID="094cfbfd5d0fd124570b0cd7d089f0eae00c5207daad406aa2786cc33a383174"
Jul 28 18:56:19.577457 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:56:19.577440    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094cfbfd5d0fd124570b0cd7d089f0eae00c5207daad406aa2786cc33a383174\": container with ID starting with 094cfbfd5d0fd124570b0cd7d089f0eae00c5207daad406aa2786cc33a383174 not found: ID does not exist" containerID="094cfbfd5d0fd124570b0cd7d089f0eae00c5207daad406aa2786cc33a383174"
Jul 28 18:56:19.577515 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.577470    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094cfbfd5d0fd124570b0cd7d089f0eae00c5207daad406aa2786cc33a383174"} err="failed to get container status \"094cfbfd5d0fd124570b0cd7d089f0eae00c5207daad406aa2786cc33a383174\": rpc error: code = NotFound desc = could not find container \"094cfbfd5d0fd124570b0cd7d089f0eae00c5207daad406aa2786cc33a383174\": container with ID starting with 094cfbfd5d0fd124570b0cd7d089f0eae00c5207daad406aa2786cc33a383174 not found: ID does not exist"
Jul 28 18:56:19.577515 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.577486    2564 scope.go:117] "RemoveContainer" containerID="d5ebc8b935951cf1b5ace7e282680b8a18230476df9d8af79625785d3eaf8ed4"
Jul 28 18:56:19.580686 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.580663    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"]
Jul 28 18:56:19.584286 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.584266    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-37e11d-946c5bf55-h72l8"]
Jul 28 18:56:19.585185 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.585165    2564 scope.go:117] "RemoveContainer" containerID="d5ebc8b935951cf1b5ace7e282680b8a18230476df9d8af79625785d3eaf8ed4"
Jul 28 18:56:19.585459 ip-10-0-139-99 kubenswrapper[2564]: E0728 18:56:19.585441    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5ebc8b935951cf1b5ace7e282680b8a18230476df9d8af79625785d3eaf8ed4\": container with ID starting with d5ebc8b935951cf1b5ace7e282680b8a18230476df9d8af79625785d3eaf8ed4 not found: ID does not exist" containerID="d5ebc8b935951cf1b5ace7e282680b8a18230476df9d8af79625785d3eaf8ed4"
Jul 28 18:56:19.585512 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.585467    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5ebc8b935951cf1b5ace7e282680b8a18230476df9d8af79625785d3eaf8ed4"} err="failed to get container status \"d5ebc8b935951cf1b5ace7e282680b8a18230476df9d8af79625785d3eaf8ed4\": rpc error: code = NotFound desc = could not find container \"d5ebc8b935951cf1b5ace7e282680b8a18230476df9d8af79625785d3eaf8ed4\": container with ID starting with d5ebc8b935951cf1b5ace7e282680b8a18230476df9d8af79625785d3eaf8ed4 not found: ID does not exist"
Jul 28 18:56:19.594011 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.593988    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"]
Jul 28 18:56:19.597963 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.597941    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-37e11d-7dcf986488-tqkjt"]
Jul 28 18:56:19.609054 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.609026    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"]
Jul 28 18:56:19.611055 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:19.611037    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-37e11d-56df47d77d-zdgh9"]
Jul 28 18:56:21.216190 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:21.216158    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d1e757b-a627-4946-9184-7e6633fe0716" path="/var/lib/kubelet/pods/3d1e757b-a627-4946-9184-7e6633fe0716/volumes"
Jul 28 18:56:21.216545 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:21.216492    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68309e6c-2f60-4fb6-ba40-bfda26857446" path="/var/lib/kubelet/pods/68309e6c-2f60-4fb6-ba40-bfda26857446/volumes"
Jul 28 18:56:21.216872 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:21.216858    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5f538fb-a15e-4b81-812e-5091d8475ca9" path="/var/lib/kubelet/pods/a5f538fb-a15e-4b81-812e-5091d8475ca9/volumes"
Jul 28 18:56:36.451948 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.451915    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"]
Jul 28 18:56:36.452352 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.452176    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8" podUID="7eb97487-cd38-4561-840c-33f153df2164" containerName="maas-api" containerID="cri-o://84ddf526ae1af11558108472e07156f4858b0fc6043725d0dbfbcfbd91ce8ebc" gracePeriod=30
Jul 28 18:56:36.506526 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.506494    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"]
Jul 28 18:56:36.506775 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.506728    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh" podUID="fa4bebc7-ab42-44fc-aed7-b4b455c3ea26" containerName="payload-processing" containerID="cri-o://fd5e402554c7dbff9e0035149b5a31b7a66eddc6a1d2de2146ba6ebdf96ce087" gracePeriod=30
Jul 28 18:56:36.643262 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.643221    2564 generic.go:356] "Generic (PLEG): container finished" podID="fa4bebc7-ab42-44fc-aed7-b4b455c3ea26" containerID="fd5e402554c7dbff9e0035149b5a31b7a66eddc6a1d2de2146ba6ebdf96ce087" exitCode=0
Jul 28 18:56:36.643423 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.643311    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh" event={"ID":"fa4bebc7-ab42-44fc-aed7-b4b455c3ea26","Type":"ContainerDied","Data":"fd5e402554c7dbff9e0035149b5a31b7a66eddc6a1d2de2146ba6ebdf96ce087"}
Jul 28 18:56:36.646051 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.646020    2564 generic.go:356] "Generic (PLEG): container finished" podID="7eb97487-cd38-4561-840c-33f153df2164" containerID="84ddf526ae1af11558108472e07156f4858b0fc6043725d0dbfbcfbd91ce8ebc" exitCode=0
Jul 28 18:56:36.646198 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.646091    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8" event={"ID":"7eb97487-cd38-4561-840c-33f153df2164","Type":"ContainerDied","Data":"84ddf526ae1af11558108472e07156f4858b0fc6043725d0dbfbcfbd91ce8ebc"}
Jul 28 18:56:36.746275 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.746249    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"
Jul 28 18:56:36.778172 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.778147    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"
Jul 28 18:56:36.806391 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.806360    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmlnb\" (UniqueName: \"kubernetes.io/projected/7eb97487-cd38-4561-840c-33f153df2164-kube-api-access-kmlnb\") pod \"7eb97487-cd38-4561-840c-33f153df2164\" (UID: \"7eb97487-cd38-4561-840c-33f153df2164\") "
Jul 28 18:56:36.806566 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.806406    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eb97487-cd38-4561-840c-33f153df2164-maas-api-tls\") pod \"7eb97487-cd38-4561-840c-33f153df2164\" (UID: \"7eb97487-cd38-4561-840c-33f153df2164\") "
Jul 28 18:56:36.806566 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.806508    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb97487-cd38-4561-840c-33f153df2164-openshift-service-ca\") pod \"7eb97487-cd38-4561-840c-33f153df2164\" (UID: \"7eb97487-cd38-4561-840c-33f153df2164\") "
Jul 28 18:56:36.806983 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.806948    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb97487-cd38-4561-840c-33f153df2164-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7eb97487-cd38-4561-840c-33f153df2164" (UID: "7eb97487-cd38-4561-840c-33f153df2164"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:56:36.808511 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.808485    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb97487-cd38-4561-840c-33f153df2164-kube-api-access-kmlnb" (OuterVolumeSpecName: "kube-api-access-kmlnb") pod "7eb97487-cd38-4561-840c-33f153df2164" (UID: "7eb97487-cd38-4561-840c-33f153df2164"). InnerVolumeSpecName "kube-api-access-kmlnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:56:36.808593 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.808491    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb97487-cd38-4561-840c-33f153df2164-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7eb97487-cd38-4561-840c-33f153df2164" (UID: "7eb97487-cd38-4561-840c-33f153df2164"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:56:36.907695 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.907654    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb47x\" (UniqueName: \"kubernetes.io/projected/fa4bebc7-ab42-44fc-aed7-b4b455c3ea26-kube-api-access-hb47x\") pod \"fa4bebc7-ab42-44fc-aed7-b4b455c3ea26\" (UID: \"fa4bebc7-ab42-44fc-aed7-b4b455c3ea26\") "
Jul 28 18:56:36.907900 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.907838    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4bebc7-ab42-44fc-aed7-b4b455c3ea26-plugins-config-volume\") pod \"fa4bebc7-ab42-44fc-aed7-b4b455c3ea26\" (UID: \"fa4bebc7-ab42-44fc-aed7-b4b455c3ea26\") "
Jul 28 18:56:36.907974 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.907958    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb97487-cd38-4561-840c-33f153df2164-openshift-service-ca\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:36.907974 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.907970    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmlnb\" (UniqueName: \"kubernetes.io/projected/7eb97487-cd38-4561-840c-33f153df2164-kube-api-access-kmlnb\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:36.908089 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.907980    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eb97487-cd38-4561-840c-33f153df2164-maas-api-tls\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:36.908259 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.908233    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4bebc7-ab42-44fc-aed7-b4b455c3ea26-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa4bebc7-ab42-44fc-aed7-b4b455c3ea26" (UID: "fa4bebc7-ab42-44fc-aed7-b4b455c3ea26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:56:36.909694 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:36.909672    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4bebc7-ab42-44fc-aed7-b4b455c3ea26-kube-api-access-hb47x" (OuterVolumeSpecName: "kube-api-access-hb47x") pod "fa4bebc7-ab42-44fc-aed7-b4b455c3ea26" (UID: "fa4bebc7-ab42-44fc-aed7-b4b455c3ea26"). InnerVolumeSpecName "kube-api-access-hb47x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:56:37.008652 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:37.008621    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4bebc7-ab42-44fc-aed7-b4b455c3ea26-plugins-config-volume\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:37.008652 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:37.008651    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb47x\" (UniqueName: \"kubernetes.io/projected/fa4bebc7-ab42-44fc-aed7-b4b455c3ea26-kube-api-access-hb47x\") on node \"ip-10-0-139-99.ec2.internal\" DevicePath \"\""
Jul 28 18:56:37.651792 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:37.651761    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"
Jul 28 18:56:37.652280 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:37.651802    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8" event={"ID":"7eb97487-cd38-4561-840c-33f153df2164","Type":"ContainerDied","Data":"f60b7951fd9cb8422417728fc1600a385e6d9ff822eabccb1d609a4d6fbab062"}
Jul 28 18:56:37.652280 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:37.651877    2564 scope.go:117] "RemoveContainer" containerID="84ddf526ae1af11558108472e07156f4858b0fc6043725d0dbfbcfbd91ce8ebc"
Jul 28 18:56:37.653206 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:37.653179    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh" event={"ID":"fa4bebc7-ab42-44fc-aed7-b4b455c3ea26","Type":"ContainerDied","Data":"99da9b37394b6f51a42f362a0c793c197742213ea7fe1326b20aa38bac4ecb0d"}
Jul 28 18:56:37.653310 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:37.653190    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"
Jul 28 18:56:37.660763 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:37.660748    2564 scope.go:117] "RemoveContainer" containerID="fd5e402554c7dbff9e0035149b5a31b7a66eddc6a1d2de2146ba6ebdf96ce087"
Jul 28 18:56:37.670762 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:37.670742    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"]
Jul 28 18:56:37.676638 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:37.676614    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f36101-5b6bd68d68-9jbb8"]
Jul 28 18:56:37.686224 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:37.686200    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"]
Jul 28 18:56:37.689263 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:37.689243    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f36101-b867cd89d-vkgzh"]
Jul 28 18:56:39.215750 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:39.215705    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eb97487-cd38-4561-840c-33f153df2164" path="/var/lib/kubelet/pods/7eb97487-cd38-4561-840c-33f153df2164/volumes"
Jul 28 18:56:39.216206 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:39.216073    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa4bebc7-ab42-44fc-aed7-b4b455c3ea26" path="/var/lib/kubelet/pods/fa4bebc7-ab42-44fc-aed7-b4b455c3ea26/volumes"
Jul 28 18:56:43.951089 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:43.951054    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b756f6565-8jn68_d61b5f56-4792-4b6e-8e5c-29735e3bc334/authorino/0.log"
Jul 28 18:56:49.492504 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:49.492473    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56cfbfb6cd-xs7wh_1ef0fed9-0db5-440c-a8bf-513f51f8b6a8/manager/0.log"
Jul 28 18:56:50.084926 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:50.084895    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7795fd7c74-xkjx4_fde36536-6bc4-4cd7-8450-230e73b5ba59/copy-manifests/0.log"
Jul 28 18:56:50.091280 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:50.091253    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7795fd7c74-xkjx4_fde36536-6bc4-4cd7-8450-230e73b5ba59/manager/0.log"
Jul 28 18:56:50.530887 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:50.530861    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754375-wbszx_4c7264ab-2557-4578-b1d0-b5b7c959c67a/cleanup/0.log"
Jul 28 18:56:50.648882 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:50.648856    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754390-5ww2d_bc7660f4-d709-4018-99cb-5fea6c35c65f/cleanup/0.log"
Jul 28 18:56:52.149229 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:52.149200    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b756f6565-8jn68_d61b5f56-4792-4b6e-8e5c-29735e3bc334/authorino/0.log"
Jul 28 18:56:52.490615 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:52.490541    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-7t2mz_eb616377-2a6b-4fbf-a590-50d759166bfc/kuadrant-console-plugin/0.log"
Jul 28 18:56:53.639668 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:53.639636    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79b97f775b-tcccj_6f52a446-8fe8-40a6-88e8-74c05a817ffc/kube-auth-proxy/0.log"
Jul 28 18:56:54.460939 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:54.460910    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-pgq89_4bdebdea-05f6-4988-b87a-eef055328634/main/0.log"
Jul 28 18:56:54.581523 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:54.581488    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-6z8w4_84807c9f-a8ab-4964-94a9-6ba802476aa6/main/0.log"
Jul 28 18:56:55.014587 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:56:55.014553    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-tgszt_4e933fba-818a-45e5-9825-e2d73798e776/main/0.log"
Jul 28 18:57:00.446283 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446247    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-22fmk/must-gather-48kln"]
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446521    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68309e6c-2f60-4fb6-ba40-bfda26857446" containerName="payload-processing"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446532    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="68309e6c-2f60-4fb6-ba40-bfda26857446" containerName="payload-processing"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446546    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d1e757b-a627-4946-9184-7e6633fe0716" containerName="payload-pre-processing"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446553    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d1e757b-a627-4946-9184-7e6633fe0716" containerName="payload-pre-processing"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446565    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4bebc7-ab42-44fc-aed7-b4b455c3ea26" containerName="payload-processing"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446571    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4bebc7-ab42-44fc-aed7-b4b455c3ea26" containerName="payload-processing"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446580    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f538fb-a15e-4b81-812e-5091d8475ca9" containerName="maas-api"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446585    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f538fb-a15e-4b81-812e-5091d8475ca9" containerName="maas-api"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446594    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eb97487-cd38-4561-840c-33f153df2164" containerName="maas-api"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446601    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eb97487-cd38-4561-840c-33f153df2164" containerName="maas-api"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446606    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00c52ee-6690-4343-afd1-25776c9f9514" containerName="revoke-keys"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446612    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00c52ee-6690-4343-afd1-25776c9f9514" containerName="revoke-keys"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446655    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eb97487-cd38-4561-840c-33f153df2164" containerName="maas-api"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446662    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa4bebc7-ab42-44fc-aed7-b4b455c3ea26" containerName="payload-processing"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446669    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="68309e6c-2f60-4fb6-ba40-bfda26857446" containerName="payload-processing"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446676    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5f538fb-a15e-4b81-812e-5091d8475ca9" containerName="maas-api"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446682    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d00c52ee-6690-4343-afd1-25776c9f9514" containerName="revoke-keys"
Jul 28 18:57:00.446737 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.446688    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d1e757b-a627-4946-9184-7e6633fe0716" containerName="payload-pre-processing"
Jul 28 18:57:00.449641 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.449625    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-22fmk/must-gather-48kln"
Jul 28 18:57:00.452450 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.452430    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-22fmk\"/\"openshift-service-ca.crt\""
Jul 28 18:57:00.452591 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.452437    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-22fmk\"/\"kube-root-ca.crt\""
Jul 28 18:57:00.459447 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.459426    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-22fmk/must-gather-48kln"]
Jul 28 18:57:00.599710 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.599667    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx8gm\" (UniqueName: \"kubernetes.io/projected/35e0e71d-7907-4526-9c3b-b310d8ab32db-kube-api-access-vx8gm\") pod \"must-gather-48kln\" (UID: \"35e0e71d-7907-4526-9c3b-b310d8ab32db\") " pod="openshift-must-gather-22fmk/must-gather-48kln"
Jul 28 18:57:00.599936 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.599718    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/35e0e71d-7907-4526-9c3b-b310d8ab32db-must-gather-output\") pod \"must-gather-48kln\" (UID: \"35e0e71d-7907-4526-9c3b-b310d8ab32db\") " pod="openshift-must-gather-22fmk/must-gather-48kln"
Jul 28 18:57:00.701140 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.701048    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx8gm\" (UniqueName: \"kubernetes.io/projected/35e0e71d-7907-4526-9c3b-b310d8ab32db-kube-api-access-vx8gm\") pod \"must-gather-48kln\" (UID: \"35e0e71d-7907-4526-9c3b-b310d8ab32db\") " pod="openshift-must-gather-22fmk/must-gather-48kln"
Jul 28 18:57:00.701140 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.701091    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/35e0e71d-7907-4526-9c3b-b310d8ab32db-must-gather-output\") pod \"must-gather-48kln\" (UID: \"35e0e71d-7907-4526-9c3b-b310d8ab32db\") " pod="openshift-must-gather-22fmk/must-gather-48kln"
Jul 28 18:57:00.701393 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.701377    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/35e0e71d-7907-4526-9c3b-b310d8ab32db-must-gather-output\") pod \"must-gather-48kln\" (UID: \"35e0e71d-7907-4526-9c3b-b310d8ab32db\") " pod="openshift-must-gather-22fmk/must-gather-48kln"
Jul 28 18:57:00.723280 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.723247    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx8gm\" (UniqueName: \"kubernetes.io/projected/35e0e71d-7907-4526-9c3b-b310d8ab32db-kube-api-access-vx8gm\") pod \"must-gather-48kln\" (UID: \"35e0e71d-7907-4526-9c3b-b310d8ab32db\") " pod="openshift-must-gather-22fmk/must-gather-48kln"
Jul 28 18:57:00.758848 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.758795    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-22fmk/must-gather-48kln"
Jul 28 18:57:00.889289 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.889256    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-22fmk/must-gather-48kln"]
Jul 28 18:57:00.892978 ip-10-0-139-99 kubenswrapper[2564]: W0728 18:57:00.892952    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35e0e71d_7907_4526_9c3b_b310d8ab32db.slice/crio-54e63a22705606520edf77e6fe186b0b4d47adf2a51b5dcdaed02f56bbf850d9 WatchSource:0}: Error finding container 54e63a22705606520edf77e6fe186b0b4d47adf2a51b5dcdaed02f56bbf850d9: Status 404 returned error can't find the container with id 54e63a22705606520edf77e6fe186b0b4d47adf2a51b5dcdaed02f56bbf850d9
Jul 28 18:57:00.894717 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:00.894694    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:57:01.772515 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:01.772481    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-22fmk/must-gather-48kln" event={"ID":"35e0e71d-7907-4526-9c3b-b310d8ab32db","Type":"ContainerStarted","Data":"c9ff13c71a8ae3a5f434ddd652cc1c0ec6e2e7a69b0583fe2878f146f4fbb647"}
Jul 28 18:57:01.772886 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:01.772529    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-22fmk/must-gather-48kln" event={"ID":"35e0e71d-7907-4526-9c3b-b310d8ab32db","Type":"ContainerStarted","Data":"25c41cb050e157a22c77b7ae11c0d1ad7daf9a3fb6f09fa441ae51b37040ab97"}
Jul 28 18:57:01.772886 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:01.772541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-22fmk/must-gather-48kln" event={"ID":"35e0e71d-7907-4526-9c3b-b310d8ab32db","Type":"ContainerStarted","Data":"54e63a22705606520edf77e6fe186b0b4d47adf2a51b5dcdaed02f56bbf850d9"}
Jul 28 18:57:01.791564 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:01.791506    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-22fmk/must-gather-48kln" podStartSLOduration=1.1332583999999999 podStartE2EDuration="1.791490462s" totalImagesPullingTime="658.232062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:57:01.789823107 +0000 UTC m=+3357.105325220" watchObservedRunningTime="2026-07-28 18:57:01.791490462 +0000 UTC m=+3357.106992614"
Jul 28 18:57:03.116179 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:03.116142    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pfrhs_e69399ee-0e2d-477a-857d-79c7c39543a0/global-pull-secret-syncer/0.log"
Jul 28 18:57:03.278682 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:03.278650    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pq64d_6f4b2f27-907c-4aa3-af81-d2b4a221e24d/konnectivity-agent/0.log"
Jul 28 18:57:03.356431 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:03.356397    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-99.ec2.internal_6da522a8a76aa2b396d458299e956b18/haproxy/0.log"
Jul 28 18:57:08.511822 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:08.511767    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b756f6565-8jn68_d61b5f56-4792-4b6e-8e5c-29735e3bc334/authorino/0.log"
Jul 28 18:57:08.611156 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:08.611118    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-7t2mz_eb616377-2a6b-4fbf-a590-50d759166bfc/kuadrant-console-plugin/0.log"
Jul 28 18:57:11.123036 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:11.123000    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7khcj_61a722d1-355b-4055-adc5-7aa9ee3fcd5b/node-exporter/0.log"
Jul 28 18:57:11.144129 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:11.144105    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7khcj_61a722d1-355b-4055-adc5-7aa9ee3fcd5b/kube-rbac-proxy/0.log"
Jul 28 18:57:11.165794 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:11.165765    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7khcj_61a722d1-355b-4055-adc5-7aa9ee3fcd5b/init-textfile/0.log"
Jul 28 18:57:13.071733 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.071699    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"]
Jul 28 18:57:13.078936 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.078904    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.082488 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.082459    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-22fmk\"/\"default-dockercfg-fc5m6\""
Jul 28 18:57:13.082748 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.082726    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"]
Jul 28 18:57:13.132180 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.132136    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7c41a91-1b71-4b25-9f3d-131c6f3dbefa-sys\") pod \"perf-node-gather-daemonset-xp7vj\" (UID: \"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.132476 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.132457    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7c41a91-1b71-4b25-9f3d-131c6f3dbefa-proc\") pod \"perf-node-gather-daemonset-xp7vj\" (UID: \"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.132618 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.132603    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl94r\" (UniqueName: \"kubernetes.io/projected/d7c41a91-1b71-4b25-9f3d-131c6f3dbefa-kube-api-access-jl94r\") pod \"perf-node-gather-daemonset-xp7vj\" (UID: \"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.132736 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.132722    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7c41a91-1b71-4b25-9f3d-131c6f3dbefa-lib-modules\") pod \"perf-node-gather-daemonset-xp7vj\" (UID: \"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.132949 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.132919    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7c41a91-1b71-4b25-9f3d-131c6f3dbefa-podres\") pod \"perf-node-gather-daemonset-xp7vj\" (UID: \"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.234010 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.233973    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7c41a91-1b71-4b25-9f3d-131c6f3dbefa-sys\") pod \"perf-node-gather-daemonset-xp7vj\" (UID: \"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.234259 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.234239    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7c41a91-1b71-4b25-9f3d-131c6f3dbefa-proc\") pod \"perf-node-gather-daemonset-xp7vj\" (UID: \"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.234385 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.234368    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl94r\" (UniqueName: \"kubernetes.io/projected/d7c41a91-1b71-4b25-9f3d-131c6f3dbefa-kube-api-access-jl94r\") pod \"perf-node-gather-daemonset-xp7vj\" (UID: \"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.234494 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.234481    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7c41a91-1b71-4b25-9f3d-131c6f3dbefa-lib-modules\") pod \"perf-node-gather-daemonset-xp7vj\" (UID: \"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.234613 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.234600    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7c41a91-1b71-4b25-9f3d-131c6f3dbefa-podres\") pod \"perf-node-gather-daemonset-xp7vj\" (UID: \"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.235209 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.234089    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7c41a91-1b71-4b25-9f3d-131c6f3dbefa-sys\") pod \"perf-node-gather-daemonset-xp7vj\" (UID: \"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.235693 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.235565    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7c41a91-1b71-4b25-9f3d-131c6f3dbefa-proc\") pod \"perf-node-gather-daemonset-xp7vj\" (UID: \"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.235693 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.235665    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7c41a91-1b71-4b25-9f3d-131c6f3dbefa-podres\") pod \"perf-node-gather-daemonset-xp7vj\" (UID: \"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.235954 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.235936    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7c41a91-1b71-4b25-9f3d-131c6f3dbefa-lib-modules\") pod \"perf-node-gather-daemonset-xp7vj\" (UID: \"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.245134 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.244203    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl94r\" (UniqueName: \"kubernetes.io/projected/d7c41a91-1b71-4b25-9f3d-131c6f3dbefa-kube-api-access-jl94r\") pod \"perf-node-gather-daemonset-xp7vj\" (UID: \"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.394051 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.393968    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.582494 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.581140    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"]
Jul 28 18:57:13.861235 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.861196    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj" event={"ID":"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa","Type":"ContainerStarted","Data":"b273e924e7b6a98aa15349713c69a66c0c097088317ad2cf9fc9dce0e4c64361"}
Jul 28 18:57:13.861418 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.861259    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj" event={"ID":"d7c41a91-1b71-4b25-9f3d-131c6f3dbefa","Type":"ContainerStarted","Data":"730af88751b29f2c7fa477c03f4c8e486d0c25de6c256128580810e0d389d2e1"}
Jul 28 18:57:13.861989 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.861965    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:13.879187 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.879138    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj" podStartSLOduration=0.879122433 podStartE2EDuration="879.122433ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:57:13.877878738 +0000 UTC m=+3369.193380886" watchObservedRunningTime="2026-07-28 18:57:13.879122433 +0000 UTC m=+3369.194624546"
Jul 28 18:57:13.995728 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:13.995648    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-68ccc96945-prcbk_f6a74d1f-741b-431e-a60d-120aaa22e5ce/console/0.log"
Jul 28 18:57:15.645842 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:15.645796    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rz8gm_615e5794-659e-4726-b961-8e5be52f2272/dns/0.log"
Jul 28 18:57:15.665926 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:15.665902    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rz8gm_615e5794-659e-4726-b961-8e5be52f2272/kube-rbac-proxy/0.log"
Jul 28 18:57:15.688070 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:15.688047    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4b5g6_c73d0621-b609-4cea-a1c9-bf66bd5c0156/dns-node-resolver/0.log"
Jul 28 18:57:16.305934 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:16.305902    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xfhqj_8fd60e32-0794-41d2-ad51-83789cb97b4c/node-ca/0.log"
Jul 28 18:57:17.393518 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:17.393488    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79b97f775b-tcccj_6f52a446-8fe8-40a6-88e8-74c05a817ffc/kube-auth-proxy/0.log"
Jul 28 18:57:18.351684 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:18.351653    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kr5wn_726cdb79-f52a-4194-923b-bb594c60490d/serve-healthcheck-canary/0.log"
Jul 28 18:57:18.966556 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:18.966524    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xvdpr_c765ba9a-acaf-44e5-890a-42e943220b81/kube-rbac-proxy/0.log"
Jul 28 18:57:18.987222 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:18.987197    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xvdpr_c765ba9a-acaf-44e5-890a-42e943220b81/exporter/0.log"
Jul 28 18:57:19.009825 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:19.009774    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xvdpr_c765ba9a-acaf-44e5-890a-42e943220b81/extractor/0.log"
Jul 28 18:57:20.883189 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:20.883161    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-xp7vj"
Jul 28 18:57:21.746833 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:21.746777    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56cfbfb6cd-xs7wh_1ef0fed9-0db5-440c-a8bf-513f51f8b6a8/manager/0.log"
Jul 28 18:57:22.051936 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:22.051903    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7795fd7c74-xkjx4_fde36536-6bc4-4cd7-8450-230e73b5ba59/manager/0.log"
Jul 28 18:57:22.072167 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:22.072141    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7795fd7c74-xkjx4_fde36536-6bc4-4cd7-8450-230e73b5ba59/copy-manifests/0.log"
Jul 28 18:57:23.467224 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:23.467172    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-84df6f5b9b-r6fst_72ad3238-44e8-44f8-b1da-3050b2256bf6/manager/0.log"
Jul 28 18:57:23.495540 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:23.495504    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79b97f775b-tcccj_6f52a446-8fe8-40a6-88e8-74c05a817ffc/kube-auth-proxy/0.log"
Jul 28 18:57:25.103779 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:25.103742    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79b97f775b-tcccj_6f52a446-8fe8-40a6-88e8-74c05a817ffc/kube-auth-proxy/0.log"
Jul 28 18:57:28.288223 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:28.288186    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-9wgst_02c57bb5-3113-47ed-8052-15154fb5b1a1/migrator/0.log"
Jul 28 18:57:28.326589 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:28.326563    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-9wgst_02c57bb5-3113-47ed-8052-15154fb5b1a1/graceful-termination/0.log"
Jul 28 18:57:29.587853 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:29.587820    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6vpvk_7a63ca3e-bad2-4221-8493-38c09b9ad532/kube-multus-additional-cni-plugins/0.log"
Jul 28 18:57:29.628745 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:29.628716    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6vpvk_7a63ca3e-bad2-4221-8493-38c09b9ad532/egress-router-binary-copy/0.log"
Jul 28 18:57:29.667732 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:29.667700    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6vpvk_7a63ca3e-bad2-4221-8493-38c09b9ad532/cni-plugins/0.log"
Jul 28 18:57:29.701524 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:29.701498    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6vpvk_7a63ca3e-bad2-4221-8493-38c09b9ad532/bond-cni-plugin/0.log"
Jul 28 18:57:29.738879 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:29.738846    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6vpvk_7a63ca3e-bad2-4221-8493-38c09b9ad532/routeoverride-cni/0.log"
Jul 28 18:57:29.779137 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:29.779108    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6vpvk_7a63ca3e-bad2-4221-8493-38c09b9ad532/whereabouts-cni-bincopy/0.log"
Jul 28 18:57:29.816820 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:29.816781    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6vpvk_7a63ca3e-bad2-4221-8493-38c09b9ad532/whereabouts-cni/0.log"
Jul 28 18:57:30.263826 ip-10-0-139-99 kubenswrapper[2564]: I0728 18:57:30.263778    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zpkp7_7bb978d6-b37d-40bf-86ca-937cd0ee1f03/kube-multus/0.log"