Sep 18 13:44:22.612816 ip-10-0-128-141 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 13:44:26.032324 ip-10-0-128-141 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:44:26.032324 ip-10-0-128-141 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 13:44:26.032324 ip-10-0-128-141 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:44:26.032324 ip-10-0-128-141 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 13:44:26.032324 ip-10-0-128-141 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:44:26.033147 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.032401    2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 13:44:26.035936 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035921    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:26.035936 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035936    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035941    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035945    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035948    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035953    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035958    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035962    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035965    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035968    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035972    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035975    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035977    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035980    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035983    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035986    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035988    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035991    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035994    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.035996    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:26.036000 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036005    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036008    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036011    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036013    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036016    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036019    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036021    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036024    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036027    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036031    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036034    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036037    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036039    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036042    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036045    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036047    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036050    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036052    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036055    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:26.036450 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036057    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036060    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036063    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036066    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036069    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036071    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036074    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036077    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036079    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036082    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036084    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036087    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036090    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036093    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036096    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036099    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036102    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036104    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036107    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036110    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:26.036950 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036113    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036115    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036118    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036121    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036123    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036126    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036128    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036131    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036133    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036137    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036141    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036144    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036147    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036150    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036153    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036156    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036160    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036162    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036165    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036168    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:26.037442 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036170    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036173    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036175    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036179    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036182    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036185    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036188    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036190    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036193    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036196    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036199    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036201    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036204    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036207    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036210    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036213    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036215    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036218    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036220    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036222    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:26.037982 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036900    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036907    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036910    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036913    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036915    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036918    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036921    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036931    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036935    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036937    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036940    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036943    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036946    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036949    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036952    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036954    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036957    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036961    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036964    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036967    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:26.038488 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036970    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036973    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036977    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036981    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036984    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036987    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036990    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036993    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036996    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.036999    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037002    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037004    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037007    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037010    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037012    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037015    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037017    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037020    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037023    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037025    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:26.039030 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037036    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037039    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037042    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037045    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037048    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037050    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037052    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037055    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037058    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037061    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037063    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037066    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037069    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037072    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037074    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037077    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037079    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037082    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037084    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:26.039514 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037087    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037089    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037092    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037095    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037097    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037101    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037105    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037108    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037111    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037113    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037116    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037119    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037123    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037126    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037136    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037140    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037144    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037147    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037149    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:26.039990 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037152    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037155    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037157    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037160    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037163    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037165    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037168    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037170    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037173    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037175    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037178    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037181    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037183    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037186    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037188    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037191    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037194    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037196    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037199    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037201    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:26.040453 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037204    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037572    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037578    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037581    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037583    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037586    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037592    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037595    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037598    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037601    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037604    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037607    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037610    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037613    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037615    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037618    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037620    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037623    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037626    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037629    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:26.040951 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037631    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037634    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037637    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037639    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037642    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037644    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037647    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037650    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037654    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037658    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037661    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037664    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037667    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037669    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037672    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037675    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037679    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037683    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037690    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:26.041436 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037693    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037696    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037702    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037705    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037708    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037711    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037713    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037716    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037719    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037721    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037724    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037727    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037729    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037732    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037735    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037738    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037741    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037745    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037748    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037750    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:26.041920 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037753    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037756    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037758    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037761    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037763    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037766    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037769    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037772    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037775    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037777    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037780    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037784    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037787    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037789    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037794    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037796    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037799    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037802    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037805    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:26.042408 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037807    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037810    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037812    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037815    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037818    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037820    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037823    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037826    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037828    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037831    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037834    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037836    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037839    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037842    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037844    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037847    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037849    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037852    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037855    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037857    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:26.042873 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037860    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.037862    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.037964    2572 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.037998    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038005    2572 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038009    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038014    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038018    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038023    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038027    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038030    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038033    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038036    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038040    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038042    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038045    2572 flags.go:64] FLAG: --cgroup-root=""
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038048    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038051    2572 flags.go:64] FLAG: --client-ca-file=""
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038054    2572 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038057    2572 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038061    2572 flags.go:64] FLAG: --cluster-domain=""
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038064    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038067    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038070    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 13:44:26.043363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038073    2572 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038077    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038080    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038082    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038085    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038088    2572 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038091    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038095    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038098    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038101    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038105    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038108    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038111    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038114    2572 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038117    2572 flags.go:64] FLAG: --enable-server="true"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038121    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038126    2572 flags.go:64] FLAG: --event-burst="100"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038130    2572 flags.go:64] FLAG: --event-qps="50"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038133    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038136    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038139    2572 flags.go:64] FLAG: --eviction-hard=""
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038143    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038146    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038148    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038151    2572 flags.go:64] FLAG: --eviction-soft=""
Sep 18 13:44:26.043978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038155    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038157    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038160    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038164    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038166    2572 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038169    2572 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038172    2572 flags.go:64] FLAG: --feature-gates=""
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038175    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038179    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038182    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038185    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038188    2572 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038191    2572 flags.go:64] FLAG: --help="false"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038194    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038196    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038199    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038202    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038205    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038209    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038212    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038215    2572 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038218    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038222    2572 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 13:44:26.044641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038225    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038228    2572 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038231    2572 flags.go:64] FLAG: --kube-reserved=""
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038234    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038237    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038241    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038243    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038246    2572 flags.go:64] FLAG: --lock-file=""
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038249    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038252    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038255    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038260    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038264    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038267    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038270    2572 flags.go:64] FLAG: --logging-format="text"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038273    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038276    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038279    2572 flags.go:64] FLAG: --manifest-url=""
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038282    2572 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038286    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038289    2572 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038293    2572 flags.go:64] FLAG: --max-pods="110"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038296    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038299    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038302    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 13:44:26.045252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038305    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038308    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038311    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038314    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038321    2572 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038324    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038328    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038332    2572 flags.go:64] FLAG: --pod-cidr=""
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038334    2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038339    2572 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038343    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038346    2572 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038349    2572 flags.go:64] FLAG: --port="10250"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038352    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038355    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0858e30db19930121"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038358    2572 flags.go:64] FLAG: --qos-reserved=""
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038361    2572 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038364    2572 flags.go:64] FLAG: --register-node="true"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038367    2572 flags.go:64] FLAG: --register-with-taints=""
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038370    2572 flags.go:64] FLAG: --registry-burst="10"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038373    2572 flags.go:64] FLAG: --registry-qps="5"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038376    2572 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038379    2572 flags.go:64] FLAG: --reserved-memory=""
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038382    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038385    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 13:44:26.045966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038388    2572 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038391    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038394    2572 flags.go:64] FLAG: --runonce="false"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038397    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038399    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038402    2572 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038405    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038409    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038412    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038415    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038419    2572 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038422    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038425    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038429    2572 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038432    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038435    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038438    2572 flags.go:64] FLAG: --system-cgroups=""
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038441    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038446    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038450    2572 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038453    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038457    2572 flags.go:64] FLAG: --tls-min-version=""
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038460    2572 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038462    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038465    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 13:44:26.046868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038468    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038471    2572 flags.go:64] FLAG: --v="2"
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038475    2572 flags.go:64] FLAG: --version="false"
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038479    2572 flags.go:64] FLAG: --vmodule=""
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038482    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038486    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038609    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038613    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038617    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038620    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038623    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038627    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038629    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038632    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038638    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038641    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038644    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038647    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038651    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038654    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038657    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:26.047481 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038660    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038663    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038666    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038668    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038671    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038674    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038677    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038680    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038682    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038686    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038689    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038693    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038695    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038698    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038700    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038703    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038706    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038709    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038711    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038714    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:26.048046 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038717    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038719    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038722    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038725    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038727    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038732    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038735    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038738    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038741    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038743    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038746    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038749    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038752    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038754    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038757    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038759    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038762    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038766    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038769    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:26.048548 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038772    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038774    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038777    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038780    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038782    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038785    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038787    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038790    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038793    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038796    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038798    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038801    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038804    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038806    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038809    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038812    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038815    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038818    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038822    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038824    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:26.049020 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038827    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038830    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038832    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038835    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038838    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038841    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038843    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038846    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038849    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038851    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038854    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038858    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038862    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038866    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038868    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038872    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038875    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038877    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038880    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038883    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:26.049513 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038885    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:26.050018 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038888    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:26.050018 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038891    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:26.050018 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038894    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:26.050018 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.038896    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:26.050018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.038901    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:44:26.050018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.047321    2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 13:44:26.050018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.047336    2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 13:44:26.050018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.047357    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 13:44:26.050018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.047364    2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 13:44:26.050018 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047419    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:26.050018 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047424    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:26.050018 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047428    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:26.050018 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047431    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:26.050018 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047434    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:26.050018 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047436    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047439    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047442    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047445    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047448    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047451    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047453    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047456    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047459    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047462    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047464    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047467    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047470    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047472    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047475    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047478    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047480    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047484    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047488    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:26.050402 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047491    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047494    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047498    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047503    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047506    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047510    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047513    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047516    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047519    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047523    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047525    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047528    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047546    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047549    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047552    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047555    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047557    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047560    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047563    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047566    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:26.050885 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047568    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047571    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047574    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047576    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047579    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047582    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047584    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047587    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047590    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047593    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047595    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047599    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047602    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047604    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047607    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047610    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047613    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047616    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047619    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047622    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:26.051417 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047625    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047628    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047630    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047633    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047636    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047638    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047641    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047644    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047646    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047650    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047653    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047656    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047659    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047662    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047665    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047668    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047671    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047674    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047677    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:26.051919 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047680    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:26.052376 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047683    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:26.052376 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047685    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:26.052376 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047688    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:26.052376 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047690    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:26.052376 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047693    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:26.052376 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047696    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:26.052376 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047699    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:26.052376 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047701    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:26.052376 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047704    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:26.052376 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047707    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:26.052376 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047710    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:26.052376 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047712    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:26.052376 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047715    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:26.052376 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047718    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:26.052376 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047720    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:26.052863 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.047725    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:44:26.052863 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047832    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:26.052863 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047837    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:26.052863 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047841    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:26.052863 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047844    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:26.052863 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047847    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:26.052863 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047850    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:26.052863 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047853    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:26.052863 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047855    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:26.052863 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047858    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:26.052863 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047861    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:26.052863 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047864    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:26.052863 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047867    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:26.052863 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047870    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:26.052863 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047873    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047876    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047878    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047881    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047883    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047886    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047889    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047892    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047895    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047897    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047900    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047903    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047905    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047908    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047911    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047913    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047916    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047918    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047921    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047924    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:26.053237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047926    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047929    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047932    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047934    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047937    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047939    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047942    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047944    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047947    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047950    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047952    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047955    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047958    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047961    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047964    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047967    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047969    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047972    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047974    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047977    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:26.053738 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047980    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047983    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047985    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047988    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047990    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047993    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047996    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.047998    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048001    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048004    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048006    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048009    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048011    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048015    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048017    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048019    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048022    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048025    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048027    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048030    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:26.054219 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048033    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048035    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048038    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048041    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048043    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048047    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048050    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048053    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048056    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048059    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048062    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048065    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048069    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048072    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048076    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048080    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048083    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048086    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048089    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:26.054815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048093    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:26.055293 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048096    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:26.055293 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048099    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:26.055293 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048102    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:26.055293 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048105    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:26.055293 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048108    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:26.055293 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:26.048111    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:26.055293 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.048116    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:44:26.055293 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.048216    2572 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 13:44:26.055293 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.050237    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 13:44:26.055293 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.051085    2572 server.go:1019] "Starting client certificate rotation"
Sep 18 13:44:26.055293 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.051181    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 13:44:26.055293 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.051214    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 13:44:26.055293 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.053356    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 13:44:26.055652 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.054441    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 13:44:26.059990 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.059976    2572 log.go:25] "Validated CRI v1 runtime API"
Sep 18 13:44:26.061459 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.061444    2572 log.go:25] "Validated CRI v1 image API"
Sep 18 13:44:26.062200 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.062185    2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 13:44:26.067931 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.067906    2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a34c96d1-a245-4d4f-97f3-27d80bdb9d17:/dev/nvme0n1p4 a73e888d-ed48-466f-a3a0-ff16578a3c1a:/dev/nvme0n1p3]
Sep 18 13:44:26.067931 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.067923    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 13:44:26.073577 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.073447    2572 manager.go:216] Machine: {Timestamp:2026-09-18 13:44:26.072956905 +0000 UTC m=+2.007922041 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3083508 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29055e0a5229ab9deef04b44d3f2db SystemUUID:ec29055e-0a52-29ab-9dee-f04b44d3f2db BootID:a1f6e9ca-7d51-41b5-a871-7af12e85c1aa Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fc:26:72:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fc:26:72:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:0d:c7:86:f8:9f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 13:44:26.073577 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.073572    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 13:44:26.073678 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.073645    2572 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 13:44:26.076417 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.075412    2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 13:44:26.076743 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.076427    2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-141.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 13:44:26.077326 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.077297    2572 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 13:44:26.077491 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.077482    2572 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 13:44:26.077568 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.077494    2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 13:44:26.077568 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.077510    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 13:44:26.077653 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.077598    2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 13:44:26.081388 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.081377    2572 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 13:44:26.081506 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.081496    2572 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 13:44:26.081568 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.081550    2572 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 13:44:26.081568 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.081562    2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 13:44:26.081635 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.081577    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 13:44:26.081635 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.081586    2572 kubelet.go:398] "Adding apiserver pod source"
Sep 18 13:44:26.081635 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.081594    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 13:44:26.081907 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.081896    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 13:44:26.081952 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.081917    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 13:44:26.084473 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.084452    2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 13:44:26.086956 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.086942    2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 13:44:26.087006 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.086973    2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 13:44:26.088010 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.087999    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 13:44:26.088052 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.088014    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 13:44:26.088052 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.088020    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 13:44:26.088052 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.088026    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 13:44:26.088052 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.088035    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 13:44:26.088052 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.088040    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 13:44:26.088052 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.088054    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 13:44:26.088205 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.088060    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 13:44:26.088205 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.088066    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 13:44:26.088205 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.088071    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 13:44:26.088205 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.088079    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 13:44:26.088205 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.088084    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 13:44:26.088205 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.088106    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 13:44:26.088205 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.088111    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 13:44:26.088638 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.088621    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 13:44:26.095887 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.095870    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-141.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 13:44:26.095887 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.095870    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 13:44:26.096019 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.096001    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-141.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 13:44:26.096217 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.096206    2572 server.go:1268] "Started kubelet"
Sep 18 13:44:26.096640 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.096615    2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 13:44:26.096892 ip-10-0-128-141 systemd[1]: Started Kubernetes Kubelet.
Sep 18 13:44:26.097418 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.097190    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 13:44:26.097418 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.097224    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 13:44:26.098256 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.098243    2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 13:44:26.099257 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.099241    2572 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 13:44:26.103974 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.103953    2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 13:44:26.103974 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.103967    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 13:44:26.104271 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.104113    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-141.ec2.internal\" not found"
Sep 18 13:44:26.104595 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.104579    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 13:44:26.104692 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.104648    2572 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 13:44:26.104692 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.104663    2572 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 13:44:26.104785 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.104731    2572 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 13:44:26.104785 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.104741    2572 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 13:44:26.105171 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.105156    2572 factory.go:55] Registering systemd factory
Sep 18 13:44:26.105171 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.105171    2572 factory.go:223] Registration of the systemd container factory successfully
Sep 18 13:44:26.105418 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.105402    2572 factory.go:153] Registering CRI-O factory
Sep 18 13:44:26.105418 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.105418    2572 factory.go:223] Registration of the crio container factory successfully
Sep 18 13:44:26.105569 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.105463    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 13:44:26.105569 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.105482    2572 factory.go:103] Registering Raw factory
Sep 18 13:44:26.105569 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.105502    2572 manager.go:1196] Started watching for new ooms in manager
Sep 18 13:44:26.106110 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.106086    2572 manager.go:349] Starting recovery of all containers
Sep 18 13:44:26.107594 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.107566    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 13:44:26.107737 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.107702    2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-141.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 13:44:26.107794 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.107777    2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 13:44:26.135095 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.134975    2572 manager.go:354] Recovery completed
Sep 18 13:44:26.138752 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.138735    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-25wzs"
Sep 18 13:44:26.139196 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.139184    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:26.141219 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.141203    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:26.141280 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.141226    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:26.141280 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.141236    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:26.141747 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.141735    2572 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 13:44:26.141815 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.141747    2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 13:44:26.141815 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.141765    2572 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 13:44:26.141815 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.141785    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:26.142516 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.142500    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:26.142575 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.142527    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:26.142575 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.142553    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:26.143986 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.143973    2572 policy_none.go:49] "None policy: Start"
Sep 18 13:44:26.144040 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.143991    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 13:44:26.144040 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.144004    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 13:44:26.145976 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.145964    2572 policy_none.go:47] "Start"
Sep 18 13:44:26.149175 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.147695    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-141.ec2.internal.18d66e32bb4d3ccf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-141.ec2.internal,UID:ip-10-0-128-141.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-141.ec2.internal,},FirstTimestamp:2026-09-18 13:44:26.096172239 +0000 UTC m=+2.031137374,LastTimestamp:2026-09-18 13:44:26.096172239 +0000 UTC m=+2.031137374,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-141.ec2.internal,}"
Sep 18 13:44:26.154408 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.154393    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-25wzs"
Sep 18 13:44:26.158328 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.158260    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-141.ec2.internal.18d66e32bdfc91d6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-141.ec2.internal,UID:ip-10-0-128-141.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-141.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-141.ec2.internal,},FirstTimestamp:2026-09-18 13:44:26.141217238 +0000 UTC m=+2.076182374,LastTimestamp:2026-09-18 13:44:26.141217238 +0000 UTC m=+2.076182374,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-141.ec2.internal,}"
Sep 18 13:44:26.181114 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.181097    2572 manager.go:337] "Starting Device Plugin manager"
Sep 18 13:44:26.196776 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.181161    2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 13:44:26.196776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.181176    2572 server.go:87] "Starting device plugin registration server"
Sep 18 13:44:26.196776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.181418    2572 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 13:44:26.196776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.181439    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 13:44:26.196776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.181515    2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 13:44:26.196776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.181648    2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 13:44:26.196776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.181657    2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 13:44:26.196776 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.182167    2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 13:44:26.196776 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.182206    2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-141.ec2.internal\" not found"
Sep 18 13:44:26.196776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.189361    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 13:44:26.196776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.190524    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 13:44:26.196776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.190577    2572 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 13:44:26.196776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.190596    2572 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 13:44:26.196776 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.190631    2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 13:44:26.196776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.193613    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:44:26.281918 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.281892    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:26.284698 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.284651    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:26.284698 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.284680    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:26.284698 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.284692    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:26.284836 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.284722    2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.290797 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.290780    2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-141.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal"]
Sep 18 13:44:26.290848 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.290840    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:26.291922 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.291909    2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.291961 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.291930    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-141.ec2.internal\": node \"ip-10-0-128-141.ec2.internal\" not found"
Sep 18 13:44:26.297152 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.297136    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:26.297222 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.297163    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:26.297222 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.297174    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:26.299325 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.299313    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:26.299436 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.299414    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.299486 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.299441    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:26.300094 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.300077    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:26.300162 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.300105    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:26.300162 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.300115    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:26.300162 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.300082    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:26.300271 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.300176    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:26.300271 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.300185    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:26.302090 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.302074    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.302186 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.302099    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:26.302812 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.302797    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:26.302889 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.302823    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:26.302889 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.302834    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:26.305568 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.305440    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f61923dd52bddbca0af64d8f40dd2b70-config\") pod \"kube-apiserver-proxy-ip-10-0-128-141.ec2.internal\" (UID: \"f61923dd52bddbca0af64d8f40dd2b70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.305568 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.305472    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a9f766cd8a606fc3e2659119349a7a0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal\" (UID: \"7a9f766cd8a606fc3e2659119349a7a0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.305568 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.305501    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a9f766cd8a606fc3e2659119349a7a0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal\" (UID: \"7a9f766cd8a606fc3e2659119349a7a0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.319263 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.319245    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-141.ec2.internal\" not found"
Sep 18 13:44:26.331310 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.331291    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-141.ec2.internal\" not found" node="ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.335938 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.335921    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-141.ec2.internal\" not found" node="ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.405740 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.405711    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f61923dd52bddbca0af64d8f40dd2b70-config\") pod \"kube-apiserver-proxy-ip-10-0-128-141.ec2.internal\" (UID: \"f61923dd52bddbca0af64d8f40dd2b70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.405740 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.405738    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a9f766cd8a606fc3e2659119349a7a0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal\" (UID: \"7a9f766cd8a606fc3e2659119349a7a0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.405922 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.405755    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a9f766cd8a606fc3e2659119349a7a0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal\" (UID: \"7a9f766cd8a606fc3e2659119349a7a0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.405922 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.405781    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a9f766cd8a606fc3e2659119349a7a0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal\" (UID: \"7a9f766cd8a606fc3e2659119349a7a0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.405922 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.405818    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f61923dd52bddbca0af64d8f40dd2b70-config\") pod \"kube-apiserver-proxy-ip-10-0-128-141.ec2.internal\" (UID: \"f61923dd52bddbca0af64d8f40dd2b70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.405922 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.405823    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a9f766cd8a606fc3e2659119349a7a0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal\" (UID: \"7a9f766cd8a606fc3e2659119349a7a0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.419829 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.419812    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-141.ec2.internal\" not found"
Sep 18 13:44:26.520094 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.520066    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-141.ec2.internal\" not found"
Sep 18 13:44:26.621173 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.621113    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-141.ec2.internal\" not found"
Sep 18 13:44:26.635294 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.635273    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.639214 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:26.639195    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal"
Sep 18 13:44:26.721223 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.721197    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-141.ec2.internal\" not found"
Sep 18 13:44:26.821736 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.821708    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-141.ec2.internal\" not found"
Sep 18 13:44:26.922269 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:26.922184    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-141.ec2.internal\" not found"
Sep 18 13:44:27.022868 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:27.022841    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-141.ec2.internal\" not found"
Sep 18 13:44:27.051020 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:27.050996    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 13:44:27.051550 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:27.051152    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 13:44:27.090489 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:27.090465    2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:44:27.105043 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:27.105025    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal"
Sep 18 13:44:27.105147 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:27.105060    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 13:44:27.117510 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:27.117494    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 13:44:27.119097 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:27.119077    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-141.ec2.internal"
Sep 18 13:44:27.120028 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:27.120008    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 13:44:27.129835 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:27.129821    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 13:44:27.143683 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:27.143664    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pln6r"
Sep 18 13:44:27.155611 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:27.155592    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pln6r"
Sep 18 13:44:27.159590 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:27.159566    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 13:39:26 +0000 UTC" deadline="2028-02-18 02:41:02.469857048 +0000 UTC"
Sep 18 13:44:27.159590 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:27.159589    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12420h56m35.310270354s"
Sep 18 13:44:27.244797 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:27.243640    2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:44:27.252281 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:27.252242    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9f766cd8a606fc3e2659119349a7a0.slice/crio-c2681bd7f63751a9631cf4f6e6bfdb3296832f05bcf81fbdb225665368ca98ec WatchSource:0}: Error finding container c2681bd7f63751a9631cf4f6e6bfdb3296832f05bcf81fbdb225665368ca98ec: Status 404 returned error can't find the container with id c2681bd7f63751a9631cf4f6e6bfdb3296832f05bcf81fbdb225665368ca98ec
Sep 18 13:44:27.252758 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:27.252741    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61923dd52bddbca0af64d8f40dd2b70.slice/crio-28638c4363cd3f40de411bc3972fdaa9bf1ab4a5b9ecf69e3b2dfa4df09906b7 WatchSource:0}: Error finding container 28638c4363cd3f40de411bc3972fdaa9bf1ab4a5b9ecf69e3b2dfa4df09906b7: Status 404 returned error can't find the container with id 28638c4363cd3f40de411bc3972fdaa9bf1ab4a5b9ecf69e3b2dfa4df09906b7
Sep 18 13:44:27.258238 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:27.258223    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:44:27.702705 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:27.702527    2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:44:28.071730 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.071650    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:44:28.082478 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.082458    2572 apiserver.go:52] "Watching apiserver"
Sep 18 13:44:28.089106 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.089087    2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 13:44:28.089461 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.089429    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-4w9mg","openshift-ovn-kubernetes/ovnkube-node-8tg7h","kube-system/konnectivity-agent-kpjj7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj","openshift-cluster-node-tuning-operator/tuned-dbq8n","openshift-dns/node-resolver-txrvf","openshift-image-registry/node-ca-n2zrq","openshift-multus/multus-9lx9g","openshift-multus/multus-additional-cni-plugins-tqjnl","openshift-network-diagnostics/network-check-target-wstlx","kube-system/global-pull-secret-syncer-stclq","kube-system/kube-apiserver-proxy-ip-10-0-128-141.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal","openshift-multus/network-metrics-daemon-d2v6t"]
Sep 18 13:44:28.091794 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.091779    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n2zrq"
Sep 18 13:44:28.093973 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.093953    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 13:44:28.094067 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.093955    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 13:44:28.094067 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.094043    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sf5lw\""
Sep 18 13:44:28.094067 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.094045    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:28.094219 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.093960    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 13:44:28.094219 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.094121    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:28.096197 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.096179    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kpjj7"
Sep 18 13:44:28.097908 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.097884    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jbg5b\""
Sep 18 13:44:28.097986 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.097961    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 13:44:28.098242 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.098223    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 13:44:28.098349 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.098327    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.100221 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.100200    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 13:44:28.100309 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.100264    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 13:44:28.100418 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.100395    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 13:44:28.100504 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.100489    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wp5dc\""
Sep 18 13:44:28.100852 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.100834    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.103217 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.102858    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9zpqq\""
Sep 18 13:44:28.103217 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.102912    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:44:28.103217 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.102933    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 13:44:28.103457 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.103281    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-txrvf"
Sep 18 13:44:28.105313 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.105297    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7h7h4\""
Sep 18 13:44:28.105441 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.105346    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 13:44:28.105495 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.105470    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 13:44:28.105811 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.105793    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.108294 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.108271    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 13:44:28.108457 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.108442    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 13:44:28.108526 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.108512    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v6m4q\""
Sep 18 13:44:28.108732 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.108717    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 13:44:28.108807 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.108794    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 13:44:28.108957 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.108943    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 13:44:28.109104 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.109090    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:28.109192 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.109171    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2v6t" podUID="40156dc5-57f0-409f-a03f-2f0a80718d1d"
Sep 18 13:44:28.111606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.111587    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.113322 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.113304    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 13:44:28.113495 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.113481    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 13:44:28.113748 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.113730    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 13:44:28.113885 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.113749    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-msttp\""
Sep 18 13:44:28.113885 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.113756    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 13:44:28.113997 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.113913    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 13:44:28.113997 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.113942    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:28.113997 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.113979    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpr94\" (UniqueName: \"kubernetes.io/projected/af772cbf-6bc7-4051-a503-ac1ce786aa9d-kube-api-access-hpr94\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.114135 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.113998    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-modprobe-d\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.114135 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.114007    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wstlx" podUID="07d3a346-ad98-4e7a-9ebc-77bfe00b06be"
Sep 18 13:44:28.114135 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114014    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-socket-dir\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.114135 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114058    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-device-dir\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.114135 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114084    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-run\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.114135 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114108    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-cni-binary-copy\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.114352 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114152    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.114352 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114175    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-sysctl-d\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.114352 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114204    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-lib-modules\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.114352 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114232    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b7d8ad0-cdab-41f1-81d7-de60c5b2d692-serviceca\") pod \"node-ca-n2zrq\" (UID: \"3b7d8ad0-cdab-41f1-81d7-de60c5b2d692\") " pod="openshift-image-registry/node-ca-n2zrq"
Sep 18 13:44:28.114352 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114254    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrqcz\" (UniqueName: \"kubernetes.io/projected/3b7d8ad0-cdab-41f1-81d7-de60c5b2d692-kube-api-access-zrqcz\") pod \"node-ca-n2zrq\" (UID: \"3b7d8ad0-cdab-41f1-81d7-de60c5b2d692\") " pod="openshift-image-registry/node-ca-n2zrq"
Sep 18 13:44:28.114352 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114273    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbb05c7c-6add-4a50-82c5-ca67901571f7-kubelet-config\") pod \"global-pull-secret-syncer-stclq\" (UID: \"cbb05c7c-6add-4a50-82c5-ca67901571f7\") " pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:28.114352 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114288    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-registration-dir\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.114352 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114301    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-systemd\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.114352 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114318    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-var-lib-kubelet\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.114352 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114343    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a57abd9-a0ec-4d26-afe6-b74b8cc663a6-agent-certs\") pod \"konnectivity-agent-kpjj7\" (UID: \"2a57abd9-a0ec-4d26-afe6-b74b8cc663a6\") " pod="kube-system/konnectivity-agent-kpjj7"
Sep 18 13:44:28.114749 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114379    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b7d8ad0-cdab-41f1-81d7-de60c5b2d692-host\") pod \"node-ca-n2zrq\" (UID: \"3b7d8ad0-cdab-41f1-81d7-de60c5b2d692\") " pod="openshift-image-registry/node-ca-n2zrq"
Sep 18 13:44:28.114749 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114411    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.114749 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114436    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-kubernetes\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.114749 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114457    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0301c03b-a8d5-41c2-8320-e533954377db-etc-tuned\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.114749 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114481    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.114749 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114504    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2gcc\" (UniqueName: \"kubernetes.io/projected/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-kube-api-access-k2gcc\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.114749 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114543    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d34b196c-8eaa-40cf-a2e4-6b9c5a07652f-tmp-dir\") pod \"node-resolver-txrvf\" (UID: \"d34b196c-8eaa-40cf-a2e4-6b9c5a07652f\") " pod="openshift-dns/node-resolver-txrvf"
Sep 18 13:44:28.114749 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114570    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 13:44:28.114749 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114599    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h7kf\" (UniqueName: \"kubernetes.io/projected/0301c03b-a8d5-41c2-8320-e533954377db-kube-api-access-6h7kf\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.114749 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114652    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret\") pod \"global-pull-secret-syncer-stclq\" (UID: \"cbb05c7c-6add-4a50-82c5-ca67901571f7\") " pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:28.114749 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114668    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqcwt\" (UniqueName: \"kubernetes.io/projected/d34b196c-8eaa-40cf-a2e4-6b9c5a07652f-kube-api-access-lqcwt\") pod \"node-resolver-txrvf\" (UID: \"d34b196c-8eaa-40cf-a2e4-6b9c5a07652f\") " pod="openshift-dns/node-resolver-txrvf"
Sep 18 13:44:28.114749 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114714    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-sysctl-conf\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.114749 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114730    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-host\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.114749 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114745    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.115217 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114759    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d34b196c-8eaa-40cf-a2e4-6b9c5a07652f-hosts-file\") pod \"node-resolver-txrvf\" (UID: \"d34b196c-8eaa-40cf-a2e4-6b9c5a07652f\") " pod="openshift-dns/node-resolver-txrvf"
Sep 18 13:44:28.115217 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114788    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbb05c7c-6add-4a50-82c5-ca67901571f7-dbus\") pod \"global-pull-secret-syncer-stclq\" (UID: \"cbb05c7c-6add-4a50-82c5-ca67901571f7\") " pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:28.115217 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114811    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-sysconfig\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.115217 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114825    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-sys\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.115217 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114846    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a57abd9-a0ec-4d26-afe6-b74b8cc663a6-konnectivity-ca\") pod \"konnectivity-agent-kpjj7\" (UID: \"2a57abd9-a0ec-4d26-afe6-b74b8cc663a6\") " pod="kube-system/konnectivity-agent-kpjj7"
Sep 18 13:44:28.115217 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114880    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.115217 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114906    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-sys-fs\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.115217 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114942    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0301c03b-a8d5-41c2-8320-e533954377db-tmp\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.115217 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.114975    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-cnibin\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.115217 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.115001    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-system-cni-dir\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.116208 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.116194    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4w9mg"
Sep 18 13:44:28.118010 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.117983    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:44:28.118588 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.118464    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 13:44:28.118588 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.118485    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.118588 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.118510    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 13:44:28.118775 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.118470    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pbhgz\""
Sep 18 13:44:28.120345 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.120328    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 13:44:28.120548 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.120521    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6fj4k\""
Sep 18 13:44:28.159002 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.158957    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 13:39:27 +0000 UTC" deadline="2028-03-26 07:12:43.4559413 +0000 UTC"
Sep 18 13:44:28.159002 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.159001    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13313h28m15.296943493s"
Sep 18 13:44:28.195444 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.195397    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-141.ec2.internal" event={"ID":"f61923dd52bddbca0af64d8f40dd2b70","Type":"ContainerStarted","Data":"28638c4363cd3f40de411bc3972fdaa9bf1ab4a5b9ecf69e3b2dfa4df09906b7"}
Sep 18 13:44:28.196632 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.196608    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal" event={"ID":"7a9f766cd8a606fc3e2659119349a7a0","Type":"ContainerStarted","Data":"c2681bd7f63751a9631cf4f6e6bfdb3296832f05bcf81fbdb225665368ca98ec"}
Sep 18 13:44:28.205575 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.205557    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 13:44:28.215304 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.215281    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.215455 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.215440    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-cni-bin\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.215584 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.215567    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b7d8ad0-cdab-41f1-81d7-de60c5b2d692-serviceca\") pod \"node-ca-n2zrq\" (UID: \"3b7d8ad0-cdab-41f1-81d7-de60c5b2d692\") " pod="openshift-image-registry/node-ca-n2zrq"
Sep 18 13:44:28.215695 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.215682    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-system-cni-dir\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.215787 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.215775    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-cnibin\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.215874 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.215863    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-etc-openvswitch\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.215989 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.215976    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a57abd9-a0ec-4d26-afe6-b74b8cc663a6-agent-certs\") pod \"konnectivity-agent-kpjj7\" (UID: \"2a57abd9-a0ec-4d26-afe6-b74b8cc663a6\") " pod="kube-system/konnectivity-agent-kpjj7"
Sep 18 13:44:28.216088 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.216072    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b7d8ad0-cdab-41f1-81d7-de60c5b2d692-host\") pod \"node-ca-n2zrq\" (UID: \"3b7d8ad0-cdab-41f1-81d7-de60c5b2d692\") " pod="openshift-image-registry/node-ca-n2zrq"
Sep 18 13:44:28.216180 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.216169    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-hostroot\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.216268 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.216257    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0301c03b-a8d5-41c2-8320-e533954377db-etc-tuned\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.216355 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.216341    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2gcc\" (UniqueName: \"kubernetes.io/projected/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-kube-api-access-k2gcc\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.216453 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.216442    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d34b196c-8eaa-40cf-a2e4-6b9c5a07652f-tmp-dir\") pod \"node-resolver-txrvf\" (UID: \"d34b196c-8eaa-40cf-a2e4-6b9c5a07652f\") " pod="openshift-dns/node-resolver-txrvf"
Sep 18 13:44:28.216562 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.216548    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-slash\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.216671 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.216659    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h7kf\" (UniqueName: \"kubernetes.io/projected/0301c03b-a8d5-41c2-8320-e533954377db-kube-api-access-6h7kf\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.216779 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.216766    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret\") pod \"global-pull-secret-syncer-stclq\" (UID: \"cbb05c7c-6add-4a50-82c5-ca67901571f7\") " pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:28.216872 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.216861    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-systemd-units\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.216964 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.216952    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-sysctl-conf\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.217060 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.217046    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.217330 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.217312    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:28.217442 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.217361    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-multus-conf-dir\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.217585 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.217562    2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 13:44:28.217676 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.217632    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d34b196c-8eaa-40cf-a2e4-6b9c5a07652f-tmp-dir\") pod \"node-resolver-txrvf\" (UID: \"d34b196c-8eaa-40cf-a2e4-6b9c5a07652f\") " pod="openshift-dns/node-resolver-txrvf"
Sep 18 13:44:28.217738 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.217713    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-run-netns\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.217792 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.217758    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret podName:cbb05c7c-6add-4a50-82c5-ca67901571f7 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:28.71773605 +0000 UTC m=+4.652701192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret") pod "global-pull-secret-syncer-stclq" (UID: "cbb05c7c-6add-4a50-82c5-ca67901571f7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:28.217858 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.217786    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-sysconfig\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.217858 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.217824    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a57abd9-a0ec-4d26-afe6-b74b8cc663a6-konnectivity-ca\") pod \"konnectivity-agent-kpjj7\" (UID: \"2a57abd9-a0ec-4d26-afe6-b74b8cc663a6\") " pod="kube-system/konnectivity-agent-kpjj7"
Sep 18 13:44:28.217858 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.217851    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/10dae063-5c4a-45b7-bbec-98bdd47716b1-multus-daemon-config\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.218011 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.217881    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-etc-kubernetes\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.218011 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.217898    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-sysconfig\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.218011 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.217982    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-run-openvswitch\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.218158 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218017    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-node-log\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.218158 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218050    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.218158 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218056    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b7d8ad0-cdab-41f1-81d7-de60c5b2d692-serviceca\") pod \"node-ca-n2zrq\" (UID: \"3b7d8ad0-cdab-41f1-81d7-de60c5b2d692\") " pod="openshift-image-registry/node-ca-n2zrq"
Sep 18 13:44:28.218158 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218083    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fpbj\" (UniqueName: \"kubernetes.io/projected/9f9099a5-ee27-4d4a-a737-1d2881b211e1-kube-api-access-7fpbj\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.218158 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218121    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.218368 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218159    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfxqw\" (UniqueName: \"kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw\") pod \"network-check-target-wstlx\" (UID: \"07d3a346-ad98-4e7a-9ebc-77bfe00b06be\") " pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:28.218695 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218645    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-sysctl-conf\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.218793 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218692    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ad0ffbe-d91b-4574-a7bd-978b1f600f37-host-slash\") pod \"iptables-alerter-4w9mg\" (UID: \"1ad0ffbe-d91b-4574-a7bd-978b1f600f37\") " pod="openshift-network-operator/iptables-alerter-4w9mg"
Sep 18 13:44:28.218793 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218700    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.218793 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218719    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b7d8ad0-cdab-41f1-81d7-de60c5b2d692-host\") pod \"node-ca-n2zrq\" (UID: \"3b7d8ad0-cdab-41f1-81d7-de60c5b2d692\") " pod="openshift-image-registry/node-ca-n2zrq"
Sep 18 13:44:28.218793 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218725    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-system-cni-dir\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.218793 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218753    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpr94\" (UniqueName: \"kubernetes.io/projected/af772cbf-6bc7-4051-a503-ac1ce786aa9d-kube-api-access-hpr94\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.219028 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218802    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-system-cni-dir\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.219028 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218835    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-modprobe-d\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.219028 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218930    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-socket-dir\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.219028 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218946    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a57abd9-a0ec-4d26-afe6-b74b8cc663a6-konnectivity-ca\") pod \"konnectivity-agent-kpjj7\" (UID: \"2a57abd9-a0ec-4d26-afe6-b74b8cc663a6\") " pod="kube-system/konnectivity-agent-kpjj7"
Sep 18 13:44:28.219028 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218957    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-run\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.219028 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218964    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.219028 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.218995    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.219382 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219031    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrqcz\" (UniqueName: \"kubernetes.io/projected/3b7d8ad0-cdab-41f1-81d7-de60c5b2d692-kube-api-access-zrqcz\") pod \"node-ca-n2zrq\" (UID: \"3b7d8ad0-cdab-41f1-81d7-de60c5b2d692\") " pod="openshift-image-registry/node-ca-n2zrq"
Sep 18 13:44:28.219382 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219067    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-modprobe-d\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.219382 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219164    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-kubelet\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.219382 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219190    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-sysctl-d\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.219382 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219195    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-run\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.219382 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219214    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-lib-modules\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.219382 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219237    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-cni-netd\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.219382 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219260    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f9099a5-ee27-4d4a-a737-1d2881b211e1-ovn-node-metrics-cert\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.219382 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219288    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-sysctl-d\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.219382 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219315    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbb05c7c-6add-4a50-82c5-ca67901571f7-kubelet-config\") pod \"global-pull-secret-syncer-stclq\" (UID: \"cbb05c7c-6add-4a50-82c5-ca67901571f7\") " pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:28.219382 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219339    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-registration-dir\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.219382 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219360    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-systemd\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.219382 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219381    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-var-lib-kubelet\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219398    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-lib-modules\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219405    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-run-netns\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219430    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-run-multus-certs\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219470    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb5bl\" (UniqueName: \"kubernetes.io/projected/10dae063-5c4a-45b7-bbec-98bdd47716b1-kube-api-access-jb5bl\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219485    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbb05c7c-6add-4a50-82c5-ca67901571f7-kubelet-config\") pod \"global-pull-secret-syncer-stclq\" (UID: \"cbb05c7c-6add-4a50-82c5-ca67901571f7\") " pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219485    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-socket-dir\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219509    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9f9099a5-ee27-4d4a-a737-1d2881b211e1-ovnkube-script-lib\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219525    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219556    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-registration-dir\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219549    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219528    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-systemd\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219594    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-var-lib-kubelet\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219619    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-kubernetes\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219659    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-etc-kubernetes\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219660    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219698    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.220018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219735    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-multus-cni-dir\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219777    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10dae063-5c4a-45b7-bbec-98bdd47716b1-cni-binary-copy\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219805    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-var-lib-cni-multus\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219839    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-log-socket\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219855    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219873    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqcwt\" (UniqueName: \"kubernetes.io/projected/d34b196c-8eaa-40cf-a2e4-6b9c5a07652f-kube-api-access-lqcwt\") pod \"node-resolver-txrvf\" (UID: \"d34b196c-8eaa-40cf-a2e4-6b9c5a07652f\") " pod="openshift-dns/node-resolver-txrvf"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.219904    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-var-lib-cni-bin\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220041    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-var-lib-kubelet\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220067    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs\") pod \"network-metrics-daemon-d2v6t\" (UID: \"40156dc5-57f0-409f-a03f-2f0a80718d1d\") " pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220182    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-run-ovn\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220226    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-host\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220270    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d34b196c-8eaa-40cf-a2e4-6b9c5a07652f-hosts-file\") pod \"node-resolver-txrvf\" (UID: \"d34b196c-8eaa-40cf-a2e4-6b9c5a07652f\") " pod="openshift-dns/node-resolver-txrvf"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220320    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmqlj\" (UniqueName: \"kubernetes.io/projected/40156dc5-57f0-409f-a03f-2f0a80718d1d-kube-api-access-gmqlj\") pod \"network-metrics-daemon-d2v6t\" (UID: \"40156dc5-57f0-409f-a03f-2f0a80718d1d\") " pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220327    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-host\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220347    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-var-lib-openvswitch\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220380    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9f9099a5-ee27-4d4a-a737-1d2881b211e1-ovnkube-config\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220406    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbb05c7c-6add-4a50-82c5-ca67901571f7-dbus\") pod \"global-pull-secret-syncer-stclq\" (UID: \"cbb05c7c-6add-4a50-82c5-ca67901571f7\") " pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:28.220783 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220428    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-sys\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220464    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9f9099a5-ee27-4d4a-a737-1d2881b211e1-env-overrides\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220476    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0301c03b-a8d5-41c2-8320-e533954377db-sys\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220498    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-sys-fs\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220527    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0301c03b-a8d5-41c2-8320-e533954377db-tmp\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220589    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-cnibin\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220614    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-run-systemd\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220649    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ad0ffbe-d91b-4574-a7bd-978b1f600f37-iptables-alerter-script\") pod \"iptables-alerter-4w9mg\" (UID: \"1ad0ffbe-d91b-4574-a7bd-978b1f600f37\") " pod="openshift-network-operator/iptables-alerter-4w9mg"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220652    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbb05c7c-6add-4a50-82c5-ca67901571f7-dbus\") pod \"global-pull-secret-syncer-stclq\" (UID: \"cbb05c7c-6add-4a50-82c5-ca67901571f7\") " pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220674    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlgxm\" (UniqueName: \"kubernetes.io/projected/1ad0ffbe-d91b-4574-a7bd-978b1f600f37-kube-api-access-jlgxm\") pod \"iptables-alerter-4w9mg\" (UID: \"1ad0ffbe-d91b-4574-a7bd-978b1f600f37\") " pod="openshift-network-operator/iptables-alerter-4w9mg"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220706    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-sys-fs\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220710    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d34b196c-8eaa-40cf-a2e4-6b9c5a07652f-hosts-file\") pod \"node-resolver-txrvf\" (UID: \"d34b196c-8eaa-40cf-a2e4-6b9c5a07652f\") " pod="openshift-dns/node-resolver-txrvf"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220735    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-device-dir\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220748    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-cnibin\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220779    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-cni-binary-copy\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220803    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-multus-socket-dir-parent\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.220841    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af772cbf-6bc7-4051-a503-ac1ce786aa9d-device-dir\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.221606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.221216    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-cni-binary-copy\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.222231 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.221289    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-run-k8s-cni-cncf-io\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.222231 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.221501    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0301c03b-a8d5-41c2-8320-e533954377db-etc-tuned\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.222231 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.221725    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a57abd9-a0ec-4d26-afe6-b74b8cc663a6-agent-certs\") pod \"konnectivity-agent-kpjj7\" (UID: \"2a57abd9-a0ec-4d26-afe6-b74b8cc663a6\") " pod="kube-system/konnectivity-agent-kpjj7"
Sep 18 13:44:28.222878 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.222857    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0301c03b-a8d5-41c2-8320-e533954377db-tmp\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.227583 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.227524    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2gcc\" (UniqueName: \"kubernetes.io/projected/4ffe1615-f9cc-4314-abe9-fbfafc8f67fd-kube-api-access-k2gcc\") pod \"multus-additional-cni-plugins-tqjnl\" (UID: \"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd\") " pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.231371 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.231329    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h7kf\" (UniqueName: \"kubernetes.io/projected/0301c03b-a8d5-41c2-8320-e533954377db-kube-api-access-6h7kf\") pod \"tuned-dbq8n\" (UID: \"0301c03b-a8d5-41c2-8320-e533954377db\") " pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.232520 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.232494    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqcwt\" (UniqueName: \"kubernetes.io/projected/d34b196c-8eaa-40cf-a2e4-6b9c5a07652f-kube-api-access-lqcwt\") pod \"node-resolver-txrvf\" (UID: \"d34b196c-8eaa-40cf-a2e4-6b9c5a07652f\") " pod="openshift-dns/node-resolver-txrvf"
Sep 18 13:44:28.232868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.232850    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrqcz\" (UniqueName: \"kubernetes.io/projected/3b7d8ad0-cdab-41f1-81d7-de60c5b2d692-kube-api-access-zrqcz\") pod \"node-ca-n2zrq\" (UID: \"3b7d8ad0-cdab-41f1-81d7-de60c5b2d692\") " pod="openshift-image-registry/node-ca-n2zrq"
Sep 18 13:44:28.233135 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.233117    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpr94\" (UniqueName: \"kubernetes.io/projected/af772cbf-6bc7-4051-a503-ac1ce786aa9d-kube-api-access-hpr94\") pod \"aws-ebs-csi-driver-node-n8lxj\" (UID: \"af772cbf-6bc7-4051-a503-ac1ce786aa9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.322518 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322437    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/10dae063-5c4a-45b7-bbec-98bdd47716b1-multus-daemon-config\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.322518 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322477    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-etc-kubernetes\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.322518 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322502    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-run-openvswitch\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.322799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322522    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-node-log\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.322799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322565    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.322799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322591    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fpbj\" (UniqueName: \"kubernetes.io/projected/9f9099a5-ee27-4d4a-a737-1d2881b211e1-kube-api-access-7fpbj\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.322799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322589    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-etc-kubernetes\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.322799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322605    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-node-log\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.322799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322615    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxqw\" (UniqueName: \"kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw\") pod \"network-check-target-wstlx\" (UID: \"07d3a346-ad98-4e7a-9ebc-77bfe00b06be\") " pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:28.322799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322625    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.322799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322588    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-run-openvswitch\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.322799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322653    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ad0ffbe-d91b-4574-a7bd-978b1f600f37-host-slash\") pod \"iptables-alerter-4w9mg\" (UID: \"1ad0ffbe-d91b-4574-a7bd-978b1f600f37\") " pod="openshift-network-operator/iptables-alerter-4w9mg"
Sep 18 13:44:28.322799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322683    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-kubelet\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.322799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322705    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-cni-netd\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.322799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322730    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f9099a5-ee27-4d4a-a737-1d2881b211e1-ovn-node-metrics-cert\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.322799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322758    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-run-netns\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.322799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322778    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-run-multus-certs\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.322799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322799    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb5bl\" (UniqueName: \"kubernetes.io/projected/10dae063-5c4a-45b7-bbec-98bdd47716b1-kube-api-access-jb5bl\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322821    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9f9099a5-ee27-4d4a-a737-1d2881b211e1-ovnkube-script-lib\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322847    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-multus-cni-dir\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322871    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10dae063-5c4a-45b7-bbec-98bdd47716b1-cni-binary-copy\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322897    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-var-lib-cni-multus\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322919    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-log-socket\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322946    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-var-lib-cni-bin\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.322986    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-var-lib-kubelet\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323008    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs\") pod \"network-metrics-daemon-d2v6t\" (UID: \"40156dc5-57f0-409f-a03f-2f0a80718d1d\") " pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323032    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-run-ovn\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323070    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-run-netns\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323083    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-run-ovn\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323103    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmqlj\" (UniqueName: \"kubernetes.io/projected/40156dc5-57f0-409f-a03f-2f0a80718d1d-kube-api-access-gmqlj\") pod \"network-metrics-daemon-d2v6t\" (UID: \"40156dc5-57f0-409f-a03f-2f0a80718d1d\") " pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323123    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-var-lib-openvswitch\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323123    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/10dae063-5c4a-45b7-bbec-98bdd47716b1-multus-daemon-config\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323142    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9f9099a5-ee27-4d4a-a737-1d2881b211e1-ovnkube-config\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323177    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-run-multus-certs\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323182    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9f9099a5-ee27-4d4a-a737-1d2881b211e1-env-overrides\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.323480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323179    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-kubelet\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323216    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-run-systemd\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323242    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ad0ffbe-d91b-4574-a7bd-978b1f600f37-iptables-alerter-script\") pod \"iptables-alerter-4w9mg\" (UID: \"1ad0ffbe-d91b-4574-a7bd-978b1f600f37\") " pod="openshift-network-operator/iptables-alerter-4w9mg"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323251    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ad0ffbe-d91b-4574-a7bd-978b1f600f37-host-slash\") pod \"iptables-alerter-4w9mg\" (UID: \"1ad0ffbe-d91b-4574-a7bd-978b1f600f37\") " pod="openshift-network-operator/iptables-alerter-4w9mg"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323251    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-var-lib-openvswitch\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323269    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlgxm\" (UniqueName: \"kubernetes.io/projected/1ad0ffbe-d91b-4574-a7bd-978b1f600f37-kube-api-access-jlgxm\") pod \"iptables-alerter-4w9mg\" (UID: \"1ad0ffbe-d91b-4574-a7bd-978b1f600f37\") " pod="openshift-network-operator/iptables-alerter-4w9mg"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323268    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-cni-netd\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323322    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-run-systemd\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323375    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-multus-cni-dir\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323382    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-var-lib-cni-multus\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323407    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-log-socket\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323452    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-var-lib-kubelet\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323491    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-var-lib-cni-bin\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323496    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-multus-socket-dir-parent\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323527    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-run-k8s-cni-cncf-io\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323582    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.323598    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323607    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-cni-bin\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.324252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323633    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-system-cni-dir\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323654    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9f9099a5-ee27-4d4a-a737-1d2881b211e1-ovnkube-script-lib\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.323663    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs podName:40156dc5-57f0-409f-a03f-2f0a80718d1d nodeName:}" failed. No retries permitted until 2026-09-18 13:44:28.823643875 +0000 UTC m=+4.758609001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs") pod "network-metrics-daemon-d2v6t" (UID: "40156dc5-57f0-409f-a03f-2f0a80718d1d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323702    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9f9099a5-ee27-4d4a-a737-1d2881b211e1-env-overrides\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323711    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-cnibin\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323745    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-etc-openvswitch\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323755    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-cnibin\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323773    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-hostroot\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323777    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-host-run-k8s-cni-cncf-io\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323807    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9f9099a5-ee27-4d4a-a737-1d2881b211e1-ovnkube-config\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323704    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-system-cni-dir\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323840    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323745    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-multus-socket-dir-parent\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323853    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-cni-bin\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323833    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10dae063-5c4a-45b7-bbec-98bdd47716b1-cni-binary-copy\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323840    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-hostroot\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323842    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-slash\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.324912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323803    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-slash\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.325454 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323926    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ad0ffbe-d91b-4574-a7bd-978b1f600f37-iptables-alerter-script\") pod \"iptables-alerter-4w9mg\" (UID: \"1ad0ffbe-d91b-4574-a7bd-978b1f600f37\") " pod="openshift-network-operator/iptables-alerter-4w9mg"
Sep 18 13:44:28.325454 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323944    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-systemd-units\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.325454 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323796    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-etc-openvswitch\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.325454 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.323972    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-multus-conf-dir\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.325454 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.324016    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10dae063-5c4a-45b7-bbec-98bdd47716b1-multus-conf-dir\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.325454 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.324018    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-run-netns\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.325454 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.324020    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-systemd-units\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.325454 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.324054    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f9099a5-ee27-4d4a-a737-1d2881b211e1-host-run-netns\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.325923 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.325907    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f9099a5-ee27-4d4a-a737-1d2881b211e1-ovn-node-metrics-cert\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.329736 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.329712    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:28.329736 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.329737    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:28.329917 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.329750    2572 projected.go:196] Error preparing data for projected volume kube-api-access-dfxqw for pod openshift-network-diagnostics/network-check-target-wstlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:28.329917 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.329807    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw podName:07d3a346-ad98-4e7a-9ebc-77bfe00b06be nodeName:}" failed. No retries permitted until 2026-09-18 13:44:28.829790912 +0000 UTC m=+4.764756035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dfxqw" (UniqueName: "kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw") pod "network-check-target-wstlx" (UID: "07d3a346-ad98-4e7a-9ebc-77bfe00b06be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:28.331900 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.331875    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlgxm\" (UniqueName: \"kubernetes.io/projected/1ad0ffbe-d91b-4574-a7bd-978b1f600f37-kube-api-access-jlgxm\") pod \"iptables-alerter-4w9mg\" (UID: \"1ad0ffbe-d91b-4574-a7bd-978b1f600f37\") " pod="openshift-network-operator/iptables-alerter-4w9mg"
Sep 18 13:44:28.332155 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.332110    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmqlj\" (UniqueName: \"kubernetes.io/projected/40156dc5-57f0-409f-a03f-2f0a80718d1d-kube-api-access-gmqlj\") pod \"network-metrics-daemon-d2v6t\" (UID: \"40156dc5-57f0-409f-a03f-2f0a80718d1d\") " pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:28.332155 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.332145    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb5bl\" (UniqueName: \"kubernetes.io/projected/10dae063-5c4a-45b7-bbec-98bdd47716b1-kube-api-access-jb5bl\") pod \"multus-9lx9g\" (UID: \"10dae063-5c4a-45b7-bbec-98bdd47716b1\") " pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.332288 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.332231    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fpbj\" (UniqueName: \"kubernetes.io/projected/9f9099a5-ee27-4d4a-a737-1d2881b211e1-kube-api-access-7fpbj\") pod \"ovnkube-node-8tg7h\" (UID: \"9f9099a5-ee27-4d4a-a737-1d2881b211e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.403353 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.403321    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n2zrq"
Sep 18 13:44:28.413175 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.413153    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kpjj7"
Sep 18 13:44:28.422392 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.422370    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj"
Sep 18 13:44:28.426846 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.426826    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dbq8n"
Sep 18 13:44:28.433313 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.433291    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-txrvf"
Sep 18 13:44:28.439950 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.439931    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tqjnl"
Sep 18 13:44:28.445385 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.445368    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:28.451920 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.451905    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4w9mg"
Sep 18 13:44:28.457463 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.457446    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9lx9g"
Sep 18 13:44:28.726254 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.726227    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret\") pod \"global-pull-secret-syncer-stclq\" (UID: \"cbb05c7c-6add-4a50-82c5-ca67901571f7\") " pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:28.726385 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.726367    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:28.726433 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.726426    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret podName:cbb05c7c-6add-4a50-82c5-ca67901571f7 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:29.726411333 +0000 UTC m=+5.661376456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret") pod "global-pull-secret-syncer-stclq" (UID: "cbb05c7c-6add-4a50-82c5-ca67901571f7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:28.827322 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.827289    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs\") pod \"network-metrics-daemon-d2v6t\" (UID: \"40156dc5-57f0-409f-a03f-2f0a80718d1d\") " pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:28.827499 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.827458    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:28.827579 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.827529    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs podName:40156dc5-57f0-409f-a03f-2f0a80718d1d nodeName:}" failed. No retries permitted until 2026-09-18 13:44:29.82751421 +0000 UTC m=+5.762479337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs") pod "network-metrics-daemon-d2v6t" (UID: "40156dc5-57f0-409f-a03f-2f0a80718d1d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:28.928291 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:28.928253    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxqw\" (UniqueName: \"kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw\") pod \"network-check-target-wstlx\" (UID: \"07d3a346-ad98-4e7a-9ebc-77bfe00b06be\") " pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:28.928463 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.928401    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:28.928463 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.928420    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:28.928463 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.928434    2572 projected.go:196] Error preparing data for projected volume kube-api-access-dfxqw for pod openshift-network-diagnostics/network-check-target-wstlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:28.928640 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:28.928497    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw podName:07d3a346-ad98-4e7a-9ebc-77bfe00b06be nodeName:}" failed. No retries permitted until 2026-09-18 13:44:29.928482743 +0000 UTC m=+5.863447866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dfxqw" (UniqueName: "kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw") pod "network-check-target-wstlx" (UID: "07d3a346-ad98-4e7a-9ebc-77bfe00b06be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:29.069805 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:29.069772    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7d8ad0_cdab_41f1_81d7_de60c5b2d692.slice/crio-3e1e09e2d4fe154c32f640cf6feef027545ba37aee56177d9fa18c811bb49802 WatchSource:0}: Error finding container 3e1e09e2d4fe154c32f640cf6feef027545ba37aee56177d9fa18c811bb49802: Status 404 returned error can't find the container with id 3e1e09e2d4fe154c32f640cf6feef027545ba37aee56177d9fa18c811bb49802
Sep 18 13:44:29.077431 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:29.077406    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ffe1615_f9cc_4314_abe9_fbfafc8f67fd.slice/crio-56ebb0855e552920aab49d6ca2679711f8c8c673d77c4fe091e7fb714d29e410 WatchSource:0}: Error finding container 56ebb0855e552920aab49d6ca2679711f8c8c673d77c4fe091e7fb714d29e410: Status 404 returned error can't find the container with id 56ebb0855e552920aab49d6ca2679711f8c8c673d77c4fe091e7fb714d29e410
Sep 18 13:44:29.078367 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:29.078341    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf772cbf_6bc7_4051_a503_ac1ce786aa9d.slice/crio-e5389e6ae209c8cfb397b910d0ee49f04cb04c04bda9296e5e76cf63ba2604f4 WatchSource:0}: Error finding container e5389e6ae209c8cfb397b910d0ee49f04cb04c04bda9296e5e76cf63ba2604f4: Status 404 returned error can't find the container with id e5389e6ae209c8cfb397b910d0ee49f04cb04c04bda9296e5e76cf63ba2604f4
Sep 18 13:44:29.079999 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:29.079917    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10dae063_5c4a_45b7_bbec_98bdd47716b1.slice/crio-01996e54b1095a37f6344e9c3719eea12235fcd4a04a0dcefbd99e7f77f01aba WatchSource:0}: Error finding container 01996e54b1095a37f6344e9c3719eea12235fcd4a04a0dcefbd99e7f77f01aba: Status 404 returned error can't find the container with id 01996e54b1095a37f6344e9c3719eea12235fcd4a04a0dcefbd99e7f77f01aba
Sep 18 13:44:29.101035 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:29.100926    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f9099a5_ee27_4d4a_a737_1d2881b211e1.slice/crio-4c7ec11223ae49beef63a76c5f7feae7d68d8a8119ff9f5dc2d8e00931a1cc5b WatchSource:0}: Error finding container 4c7ec11223ae49beef63a76c5f7feae7d68d8a8119ff9f5dc2d8e00931a1cc5b: Status 404 returned error can't find the container with id 4c7ec11223ae49beef63a76c5f7feae7d68d8a8119ff9f5dc2d8e00931a1cc5b
Sep 18 13:44:29.101606 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:29.101583    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad0ffbe_d91b_4574_a7bd_978b1f600f37.slice/crio-a2cd22fe951d17c2a7f4cc376aa37ca97da1f6855bf6eb3853064dba01f10fa2 WatchSource:0}: Error finding container a2cd22fe951d17c2a7f4cc376aa37ca97da1f6855bf6eb3853064dba01f10fa2: Status 404 returned error can't find the container with id a2cd22fe951d17c2a7f4cc376aa37ca97da1f6855bf6eb3853064dba01f10fa2
Sep 18 13:44:29.103237 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:29.103214    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0301c03b_a8d5_41c2_8320_e533954377db.slice/crio-f69546bf70e14937c9d90173644424a4cca82983b3cfe78a9c85573fc5f560bd WatchSource:0}: Error finding container f69546bf70e14937c9d90173644424a4cca82983b3cfe78a9c85573fc5f560bd: Status 404 returned error can't find the container with id f69546bf70e14937c9d90173644424a4cca82983b3cfe78a9c85573fc5f560bd
Sep 18 13:44:29.104631 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:29.104614    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a57abd9_a0ec_4d26_afe6_b74b8cc663a6.slice/crio-23baa0952aa835a6a963061dd0df4c15c00b9fa63aa0fc00365921111c05bdba WatchSource:0}: Error finding container 23baa0952aa835a6a963061dd0df4c15c00b9fa63aa0fc00365921111c05bdba: Status 404 returned error can't find the container with id 23baa0952aa835a6a963061dd0df4c15c00b9fa63aa0fc00365921111c05bdba
Sep 18 13:44:29.105714 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:44:29.105692    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34b196c_8eaa_40cf_a2e4_6b9c5a07652f.slice/crio-3872833614a75ebcf4a07100da9a4e7912e23ff8f750cfefc14e06ddaee0ea15 WatchSource:0}: Error finding container 3872833614a75ebcf4a07100da9a4e7912e23ff8f750cfefc14e06ddaee0ea15: Status 404 returned error can't find the container with id 3872833614a75ebcf4a07100da9a4e7912e23ff8f750cfefc14e06ddaee0ea15
Sep 18 13:44:29.159600 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:29.159434    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 13:39:27 +0000 UTC" deadline="2028-06-03 18:22:23.862871382 +0000 UTC"
Sep 18 13:44:29.159709 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:29.159602    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14980h37m54.70327453s"
Sep 18 13:44:29.201840 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:29.201804    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dbq8n" event={"ID":"0301c03b-a8d5-41c2-8320-e533954377db","Type":"ContainerStarted","Data":"f69546bf70e14937c9d90173644424a4cca82983b3cfe78a9c85573fc5f560bd"}
Sep 18 13:44:29.205478 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:29.205454    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4w9mg" event={"ID":"1ad0ffbe-d91b-4574-a7bd-978b1f600f37","Type":"ContainerStarted","Data":"a2cd22fe951d17c2a7f4cc376aa37ca97da1f6855bf6eb3853064dba01f10fa2"}
Sep 18 13:44:29.207502 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:29.207474    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h" event={"ID":"9f9099a5-ee27-4d4a-a737-1d2881b211e1","Type":"ContainerStarted","Data":"4c7ec11223ae49beef63a76c5f7feae7d68d8a8119ff9f5dc2d8e00931a1cc5b"}
Sep 18 13:44:29.209908 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:29.209883    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj" event={"ID":"af772cbf-6bc7-4051-a503-ac1ce786aa9d","Type":"ContainerStarted","Data":"e5389e6ae209c8cfb397b910d0ee49f04cb04c04bda9296e5e76cf63ba2604f4"}
Sep 18 13:44:29.212136 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:29.212110    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kpjj7" event={"ID":"2a57abd9-a0ec-4d26-afe6-b74b8cc663a6","Type":"ContainerStarted","Data":"23baa0952aa835a6a963061dd0df4c15c00b9fa63aa0fc00365921111c05bdba"}
Sep 18 13:44:29.214863 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:29.214843    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-txrvf" event={"ID":"d34b196c-8eaa-40cf-a2e4-6b9c5a07652f","Type":"ContainerStarted","Data":"3872833614a75ebcf4a07100da9a4e7912e23ff8f750cfefc14e06ddaee0ea15"}
Sep 18 13:44:29.216018 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:29.215999    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9lx9g" event={"ID":"10dae063-5c4a-45b7-bbec-98bdd47716b1","Type":"ContainerStarted","Data":"01996e54b1095a37f6344e9c3719eea12235fcd4a04a0dcefbd99e7f77f01aba"}
Sep 18 13:44:29.219264 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:29.219242    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tqjnl" event={"ID":"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd","Type":"ContainerStarted","Data":"56ebb0855e552920aab49d6ca2679711f8c8c673d77c4fe091e7fb714d29e410"}
Sep 18 13:44:29.220231 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:29.220213    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n2zrq" event={"ID":"3b7d8ad0-cdab-41f1-81d7-de60c5b2d692","Type":"ContainerStarted","Data":"3e1e09e2d4fe154c32f640cf6feef027545ba37aee56177d9fa18c811bb49802"}
Sep 18 13:44:29.225657 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:29.225637    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-141.ec2.internal" event={"ID":"f61923dd52bddbca0af64d8f40dd2b70","Type":"ContainerStarted","Data":"10542dbf5f6bb3898cff3151d91876499d3415a3e31c607fe660fc96d871dba1"}
Sep 18 13:44:29.240262 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:29.240220    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-141.ec2.internal" podStartSLOduration=2.2402076380000002 podStartE2EDuration="2.240207638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:29.239701019 +0000 UTC m=+5.174666163" watchObservedRunningTime="2026-09-18 13:44:29.240207638 +0000 UTC m=+5.175172784"
Sep 18 13:44:29.734465 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:29.733752    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret\") pod \"global-pull-secret-syncer-stclq\" (UID: \"cbb05c7c-6add-4a50-82c5-ca67901571f7\") " pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:29.734465 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:29.733967    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:29.734465 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:29.734040    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret podName:cbb05c7c-6add-4a50-82c5-ca67901571f7 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:31.734022823 +0000 UTC m=+7.668987961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret") pod "global-pull-secret-syncer-stclq" (UID: "cbb05c7c-6add-4a50-82c5-ca67901571f7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:29.835387 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:29.834789    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs\") pod \"network-metrics-daemon-d2v6t\" (UID: \"40156dc5-57f0-409f-a03f-2f0a80718d1d\") " pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:29.835387 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:29.834992    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:29.835387 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:29.835056    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs podName:40156dc5-57f0-409f-a03f-2f0a80718d1d nodeName:}" failed. No retries permitted until 2026-09-18 13:44:31.835038867 +0000 UTC m=+7.770003993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs") pod "network-metrics-daemon-d2v6t" (UID: "40156dc5-57f0-409f-a03f-2f0a80718d1d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:29.936231 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:29.936099    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxqw\" (UniqueName: \"kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw\") pod \"network-check-target-wstlx\" (UID: \"07d3a346-ad98-4e7a-9ebc-77bfe00b06be\") " pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:29.936411 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:29.936302    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:29.936411 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:29.936321    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:29.936411 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:29.936333    2572 projected.go:196] Error preparing data for projected volume kube-api-access-dfxqw for pod openshift-network-diagnostics/network-check-target-wstlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:29.936411 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:29.936390    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw podName:07d3a346-ad98-4e7a-9ebc-77bfe00b06be nodeName:}" failed. No retries permitted until 2026-09-18 13:44:31.936373023 +0000 UTC m=+7.871338151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dfxqw" (UniqueName: "kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw") pod "network-check-target-wstlx" (UID: "07d3a346-ad98-4e7a-9ebc-77bfe00b06be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:30.193244 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:30.193211    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:30.193721 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:30.193347    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:30.193721 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:30.193430    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:30.193721 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:30.193518    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2v6t" podUID="40156dc5-57f0-409f-a03f-2f0a80718d1d"
Sep 18 13:44:30.193721 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:30.193605    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:30.193721 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:30.193672    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wstlx" podUID="07d3a346-ad98-4e7a-9ebc-77bfe00b06be"
Sep 18 13:44:30.252956 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:30.252916    2572 generic.go:356] "Generic (PLEG): container finished" podID="7a9f766cd8a606fc3e2659119349a7a0" containerID="d3dfca75a59687bf6a1193479b25983181557d4a9e256564e12e064de64076d9" exitCode=0
Sep 18 13:44:30.253121 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:30.253059    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal" event={"ID":"7a9f766cd8a606fc3e2659119349a7a0","Type":"ContainerDied","Data":"d3dfca75a59687bf6a1193479b25983181557d4a9e256564e12e064de64076d9"}
Sep 18 13:44:31.278245 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:31.278070    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal" event={"ID":"7a9f766cd8a606fc3e2659119349a7a0","Type":"ContainerStarted","Data":"d831d8259bd7f99eb1b6e40fbb0dc3e3cf5b6b59a07ec8cf56295d6f68bd4ca4"}
Sep 18 13:44:31.751827 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:31.751792    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret\") pod \"global-pull-secret-syncer-stclq\" (UID: \"cbb05c7c-6add-4a50-82c5-ca67901571f7\") " pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:31.752009 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:31.751977    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:31.752073 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:31.752039    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret podName:cbb05c7c-6add-4a50-82c5-ca67901571f7 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:35.75202175 +0000 UTC m=+11.686986889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret") pod "global-pull-secret-syncer-stclq" (UID: "cbb05c7c-6add-4a50-82c5-ca67901571f7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:31.852269 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:31.852220    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs\") pod \"network-metrics-daemon-d2v6t\" (UID: \"40156dc5-57f0-409f-a03f-2f0a80718d1d\") " pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:31.852432 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:31.852374    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:31.852514 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:31.852434    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs podName:40156dc5-57f0-409f-a03f-2f0a80718d1d nodeName:}" failed. No retries permitted until 2026-09-18 13:44:35.852416965 +0000 UTC m=+11.787382095 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs") pod "network-metrics-daemon-d2v6t" (UID: "40156dc5-57f0-409f-a03f-2f0a80718d1d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:31.952965 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:31.952928    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxqw\" (UniqueName: \"kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw\") pod \"network-check-target-wstlx\" (UID: \"07d3a346-ad98-4e7a-9ebc-77bfe00b06be\") " pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:31.953134 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:31.953094    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:31.953134 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:31.953115    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:31.953134 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:31.953129    2572 projected.go:196] Error preparing data for projected volume kube-api-access-dfxqw for pod openshift-network-diagnostics/network-check-target-wstlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:31.953278 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:31.953193    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw podName:07d3a346-ad98-4e7a-9ebc-77bfe00b06be nodeName:}" failed. No retries permitted until 2026-09-18 13:44:35.953172223 +0000 UTC m=+11.888137346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dfxqw" (UniqueName: "kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw") pod "network-check-target-wstlx" (UID: "07d3a346-ad98-4e7a-9ebc-77bfe00b06be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:32.192670 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:32.192639    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:32.192838 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:32.192793    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2v6t" podUID="40156dc5-57f0-409f-a03f-2f0a80718d1d"
Sep 18 13:44:32.193142 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:32.193122    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:32.193247 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:32.193226    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:32.193301 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:32.193289    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:32.193376 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:32.193359    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wstlx" podUID="07d3a346-ad98-4e7a-9ebc-77bfe00b06be"
Sep 18 13:44:34.191624 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:34.191588    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:34.192087 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:34.191592    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:34.192087 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:34.191714    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:34.192087 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:34.191776    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2v6t" podUID="40156dc5-57f0-409f-a03f-2f0a80718d1d"
Sep 18 13:44:34.192087 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:34.191785    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:34.192087 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:34.191857    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wstlx" podUID="07d3a346-ad98-4e7a-9ebc-77bfe00b06be"
Sep 18 13:44:35.786883 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:35.786580    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret\") pod \"global-pull-secret-syncer-stclq\" (UID: \"cbb05c7c-6add-4a50-82c5-ca67901571f7\") " pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:35.786883 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:35.786754    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:35.786883 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:35.786820    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret podName:cbb05c7c-6add-4a50-82c5-ca67901571f7 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:43.786799914 +0000 UTC m=+19.721765039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret") pod "global-pull-secret-syncer-stclq" (UID: "cbb05c7c-6add-4a50-82c5-ca67901571f7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:35.887284 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:35.887245    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs\") pod \"network-metrics-daemon-d2v6t\" (UID: \"40156dc5-57f0-409f-a03f-2f0a80718d1d\") " pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:35.887468 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:35.887421    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:35.887528 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:35.887492    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs podName:40156dc5-57f0-409f-a03f-2f0a80718d1d nodeName:}" failed. No retries permitted until 2026-09-18 13:44:43.887472341 +0000 UTC m=+19.822437471 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs") pod "network-metrics-daemon-d2v6t" (UID: "40156dc5-57f0-409f-a03f-2f0a80718d1d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:35.988462 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:35.987914    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxqw\" (UniqueName: \"kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw\") pod \"network-check-target-wstlx\" (UID: \"07d3a346-ad98-4e7a-9ebc-77bfe00b06be\") " pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:35.988462 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:35.988119    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:35.988462 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:35.988141    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:35.988462 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:35.988156    2572 projected.go:196] Error preparing data for projected volume kube-api-access-dfxqw for pod openshift-network-diagnostics/network-check-target-wstlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:35.988462 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:35.988214    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw podName:07d3a346-ad98-4e7a-9ebc-77bfe00b06be nodeName:}" failed. No retries permitted until 2026-09-18 13:44:43.988195876 +0000 UTC m=+19.923161002 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dfxqw" (UniqueName: "kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw") pod "network-check-target-wstlx" (UID: "07d3a346-ad98-4e7a-9ebc-77bfe00b06be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:36.191949 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:36.191916    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:36.192118 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:36.192016    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:36.192427 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:36.192389    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2v6t" podUID="40156dc5-57f0-409f-a03f-2f0a80718d1d"
Sep 18 13:44:36.192958 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:36.192655    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wstlx" podUID="07d3a346-ad98-4e7a-9ebc-77bfe00b06be"
Sep 18 13:44:36.192958 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:36.192718    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:36.193696 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:36.193080    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:38.191385 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:38.191340    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:38.191935 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:38.191340    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:38.191935 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:38.191468    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:38.191935 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:38.191528    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:38.191935 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:38.191656    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2v6t" podUID="40156dc5-57f0-409f-a03f-2f0a80718d1d"
Sep 18 13:44:38.191935 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:38.191728    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wstlx" podUID="07d3a346-ad98-4e7a-9ebc-77bfe00b06be"
Sep 18 13:44:40.191271 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:40.191239    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:40.191843 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:40.191286    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:40.191843 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:40.191240    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:40.191843 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:40.191388    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2v6t" podUID="40156dc5-57f0-409f-a03f-2f0a80718d1d"
Sep 18 13:44:40.191843 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:40.191472    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wstlx" podUID="07d3a346-ad98-4e7a-9ebc-77bfe00b06be"
Sep 18 13:44:40.191843 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:40.191580    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:42.191425 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:42.191382    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:42.191425 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:42.191436    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:42.191879 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:42.191382    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:42.191879 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:42.191518    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wstlx" podUID="07d3a346-ad98-4e7a-9ebc-77bfe00b06be"
Sep 18 13:44:42.191879 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:42.191604    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:42.191879 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:42.191670    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2v6t" podUID="40156dc5-57f0-409f-a03f-2f0a80718d1d"
Sep 18 13:44:43.837306 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:43.837268    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret\") pod \"global-pull-secret-syncer-stclq\" (UID: \"cbb05c7c-6add-4a50-82c5-ca67901571f7\") " pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:43.837735 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:43.837389    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:43.837735 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:43.837459    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret podName:cbb05c7c-6add-4a50-82c5-ca67901571f7 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:59.83744084 +0000 UTC m=+35.772405963 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret") pod "global-pull-secret-syncer-stclq" (UID: "cbb05c7c-6add-4a50-82c5-ca67901571f7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:43.937741 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:43.937707    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs\") pod \"network-metrics-daemon-d2v6t\" (UID: \"40156dc5-57f0-409f-a03f-2f0a80718d1d\") " pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:43.937909 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:43.937854    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:43.937961 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:43.937925    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs podName:40156dc5-57f0-409f-a03f-2f0a80718d1d nodeName:}" failed. No retries permitted until 2026-09-18 13:44:59.93790425 +0000 UTC m=+35.872869395 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs") pod "network-metrics-daemon-d2v6t" (UID: "40156dc5-57f0-409f-a03f-2f0a80718d1d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:44.038326 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:44.038291    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxqw\" (UniqueName: \"kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw\") pod \"network-check-target-wstlx\" (UID: \"07d3a346-ad98-4e7a-9ebc-77bfe00b06be\") " pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:44.038493 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:44.038478    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:44.038573 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:44.038501    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:44.038573 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:44.038514    2572 projected.go:196] Error preparing data for projected volume kube-api-access-dfxqw for pod openshift-network-diagnostics/network-check-target-wstlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:44.038683 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:44.038600    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw podName:07d3a346-ad98-4e7a-9ebc-77bfe00b06be nodeName:}" failed. No retries permitted until 2026-09-18 13:45:00.038581111 +0000 UTC m=+35.973546252 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dfxqw" (UniqueName: "kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw") pod "network-check-target-wstlx" (UID: "07d3a346-ad98-4e7a-9ebc-77bfe00b06be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:44.191592 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:44.191558    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:44.191744 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:44.191600    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:44.191744 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:44.191561    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:44.191744 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:44.191706    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2v6t" podUID="40156dc5-57f0-409f-a03f-2f0a80718d1d"
Sep 18 13:44:44.191912 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:44.191792    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:44.191912 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:44.191874    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wstlx" podUID="07d3a346-ad98-4e7a-9ebc-77bfe00b06be"
Sep 18 13:44:46.191810 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:46.191758    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:46.192154 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:46.191878    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2v6t" podUID="40156dc5-57f0-409f-a03f-2f0a80718d1d"
Sep 18 13:44:46.192154 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:46.191891    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:46.192154 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:46.191997    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wstlx" podUID="07d3a346-ad98-4e7a-9ebc-77bfe00b06be"
Sep 18 13:44:46.192154 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:46.192031    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:46.192154 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:46.192122    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:46.310023 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:46.309989    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kpjj7" event={"ID":"2a57abd9-a0ec-4d26-afe6-b74b8cc663a6","Type":"ContainerStarted","Data":"6eeb1ccb55fa08c9732ebe72e962f15551d8c35f4b492879e73abb386702d64d"}
Sep 18 13:44:46.311582 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:46.311558    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dbq8n" event={"ID":"0301c03b-a8d5-41c2-8320-e533954377db","Type":"ContainerStarted","Data":"7e1544908be7e974c03d277ce82ce598931dd64c364c2f528e0f4fb24a6f2c39"}
Sep 18 13:44:46.329730 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:46.329691    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kpjj7" podStartSLOduration=3.300260417 podStartE2EDuration="20.329676903s" totalImagesPullingTime="17.029416486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:46.329599016 +0000 UTC m=+22.264564161" watchObservedRunningTime="2026-09-18 13:44:46.329676903 +0000 UTC m=+22.264642048"
Sep 18 13:44:46.330126 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:46.330105    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-141.ec2.internal" podStartSLOduration=19.330097805 podStartE2EDuration="19.330097805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:31.295979306 +0000 UTC m=+7.230944448" watchObservedRunningTime="2026-09-18 13:44:46.330097805 +0000 UTC m=+22.265062946"
Sep 18 13:44:47.314749 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:47.314576    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9lx9g" event={"ID":"10dae063-5c4a-45b7-bbec-98bdd47716b1","Type":"ContainerStarted","Data":"0d33a4cf1e656d6d2627dba9c6bf5e7574e4c6cde1c28f6084b607941730b058"}
Sep 18 13:44:47.316034 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:47.316008    2572 generic.go:356] "Generic (PLEG): container finished" podID="4ffe1615-f9cc-4314-abe9-fbfafc8f67fd" containerID="41fd60ec030d04d6fe2fb103da9e5f402b81263480eb378d1a9f5e845625e9b5" exitCode=0
Sep 18 13:44:47.316121 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:47.316088    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tqjnl" event={"ID":"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd","Type":"ContainerDied","Data":"41fd60ec030d04d6fe2fb103da9e5f402b81263480eb378d1a9f5e845625e9b5"}
Sep 18 13:44:47.317414 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:47.317397    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n2zrq" event={"ID":"3b7d8ad0-cdab-41f1-81d7-de60c5b2d692","Type":"ContainerStarted","Data":"72ef69d61e61c21e23e92d4f3bf8d0a569801a15d5bc93f75454b22863542494"}
Sep 18 13:44:47.318847 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:47.318827    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h" event={"ID":"9f9099a5-ee27-4d4a-a737-1d2881b211e1","Type":"ContainerStarted","Data":"9451fdc1fa7be4b816755c944b1e28fc5b12adc358674d0c50f1efb1bc892204"}
Sep 18 13:44:47.320072 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:47.320057    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj" event={"ID":"af772cbf-6bc7-4051-a503-ac1ce786aa9d","Type":"ContainerStarted","Data":"d8cd37e37071b1146631388c3b78d89c22f9e9aa7fedf2a73993c2e8b4cee080"}
Sep 18 13:44:47.321218 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:47.321201    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-txrvf" event={"ID":"d34b196c-8eaa-40cf-a2e4-6b9c5a07652f","Type":"ContainerStarted","Data":"2edd5485577d01ac68ac7823b268a9e739c5b2a85620fab62d8b3dda728582cf"}
Sep 18 13:44:47.342760 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:47.342713    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9lx9g" podStartSLOduration=4.280789243 podStartE2EDuration="21.342697062s" totalImagesPullingTime="17.061907819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:47.342124198 +0000 UTC m=+23.277089344" watchObservedRunningTime="2026-09-18 13:44:47.342697062 +0000 UTC m=+23.277662206"
Sep 18 13:44:47.356679 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:47.356630    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n2zrq" podStartSLOduration=4.29095104 podStartE2EDuration="21.356613834s" totalImagesPullingTime="17.065662794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:47.356582875 +0000 UTC m=+23.291548019" watchObservedRunningTime="2026-09-18 13:44:47.356613834 +0000 UTC m=+23.291578979"
Sep 18 13:44:47.376063 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:47.376023    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dbq8n" podStartSLOduration=4.346051461 podStartE2EDuration="21.375998188s" totalImagesPullingTime="17.029946727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:47.375874717 +0000 UTC m=+23.310839861" watchObservedRunningTime="2026-09-18 13:44:47.375998188 +0000 UTC m=+23.310963333"
Sep 18 13:44:47.392066 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:47.392027    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-txrvf" podStartSLOduration=4.364367245 podStartE2EDuration="21.39201558s" totalImagesPullingTime="17.027648335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:47.392005723 +0000 UTC m=+23.326970867" watchObservedRunningTime="2026-09-18 13:44:47.39201558 +0000 UTC m=+23.326980724"
Sep 18 13:44:47.545020 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:47.544874    2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 13:44:48.191790 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:48.191747    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:48.191790 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:48.191779    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:48.192022 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:48.191864    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:48.192022 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:48.191880    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:48.192022 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:48.191959    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wstlx" podUID="07d3a346-ad98-4e7a-9ebc-77bfe00b06be"
Sep 18 13:44:48.192128 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:48.192059    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2v6t" podUID="40156dc5-57f0-409f-a03f-2f0a80718d1d"
Sep 18 13:44:48.192734 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:48.192645    2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T13:44:47.545014245Z","UUID":"09bdec32-bcb8-4f6d-863b-3deb52073020","Handler":null,"Name":"","Endpoint":""}
Sep 18 13:44:48.194302 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:48.194283    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 13:44:48.194396 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:48.194316    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 13:44:48.327116 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:48.327079    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj" event={"ID":"af772cbf-6bc7-4051-a503-ac1ce786aa9d","Type":"ContainerStarted","Data":"d3c93c228b7b348246e199bacc0d6a1b7f443cd9ab4ec10d47e98274f6df65ff"}
Sep 18 13:44:48.329199 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:48.329167    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4w9mg" event={"ID":"1ad0ffbe-d91b-4574-a7bd-978b1f600f37","Type":"ContainerStarted","Data":"e931879786fda20e3e918588aff2a39e0c889acb34f4a33d4890443cd5b2b253"}
Sep 18 13:44:48.333849 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:48.333821    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h" event={"ID":"9f9099a5-ee27-4d4a-a737-1d2881b211e1","Type":"ContainerStarted","Data":"1b5d18ac1d24a257d367b84a2c1bf26672d3090d97af6f8e445ab6f5d6cd8ec3"}
Sep 18 13:44:48.333994 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:48.333891    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h" event={"ID":"9f9099a5-ee27-4d4a-a737-1d2881b211e1","Type":"ContainerStarted","Data":"fbc6629c4dc3d0c064b9d888840d1aae6ff3f038b5206ac6798988a735355fac"}
Sep 18 13:44:48.333994 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:48.333945    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h" event={"ID":"9f9099a5-ee27-4d4a-a737-1d2881b211e1","Type":"ContainerStarted","Data":"0eeaf2bb2a657d971a0147428c95756855f02859b70a691ca72b13082ca02c81"}
Sep 18 13:44:48.333994 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:48.333989    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h" event={"ID":"9f9099a5-ee27-4d4a-a737-1d2881b211e1","Type":"ContainerStarted","Data":"eba4b2b786f74572a5d3ed8d731882ee2df9fd646f53b969084abc6c395ff5ec"}
Sep 18 13:44:48.334133 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:48.334036    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h" event={"ID":"9f9099a5-ee27-4d4a-a737-1d2881b211e1","Type":"ContainerStarted","Data":"320e18e809c898d907e9f21d420c84d5602d10cee785db6244c4c22b85dcc7ff"}
Sep 18 13:44:48.344908 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:48.344857    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4w9mg" podStartSLOduration=5.309664872 podStartE2EDuration="22.344839746s" totalImagesPullingTime="17.035174874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:48.344036868 +0000 UTC m=+24.279002014" watchObservedRunningTime="2026-09-18 13:44:48.344839746 +0000 UTC m=+24.279804895"
Sep 18 13:44:49.337930 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:49.337725    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj" event={"ID":"af772cbf-6bc7-4051-a503-ac1ce786aa9d","Type":"ContainerStarted","Data":"c3abf29bf3ec131ce9eb913f03600591f83e038d764777c7b53ca86138d2249f"}
Sep 18 13:44:49.358978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:49.358928    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n8lxj" podStartSLOduration=4.437559785 podStartE2EDuration="23.358907684s" totalImagesPullingTime="18.921347899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:49.358581631 +0000 UTC m=+25.293546777" watchObservedRunningTime="2026-09-18 13:44:49.358907684 +0000 UTC m=+25.293872833"
Sep 18 13:44:49.439280 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:49.439250    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kpjj7"
Sep 18 13:44:49.439910 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:49.439885    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kpjj7"
Sep 18 13:44:50.191325 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:50.191293    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:50.191497 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:50.191302    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:50.191497 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:50.191415    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wstlx" podUID="07d3a346-ad98-4e7a-9ebc-77bfe00b06be"
Sep 18 13:44:50.191636 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:50.191498    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:50.191636 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:50.191306    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:50.191636 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:50.191610    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2v6t" podUID="40156dc5-57f0-409f-a03f-2f0a80718d1d"
Sep 18 13:44:50.343767 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:50.343727    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h" event={"ID":"9f9099a5-ee27-4d4a-a737-1d2881b211e1","Type":"ContainerStarted","Data":"99386d995ec4fbe20960e1c6d050656af3e1b2a560c21b946bcffd0c89f285ef"}
Sep 18 13:44:50.344421 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:50.344124    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kpjj7"
Sep 18 13:44:50.344622 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:50.344602    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kpjj7"
Sep 18 13:44:52.191164 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:52.190999    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:52.191711 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:52.190999    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:52.191711 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:52.191258    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:52.191711 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:52.191344    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wstlx" podUID="07d3a346-ad98-4e7a-9ebc-77bfe00b06be"
Sep 18 13:44:52.191711 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:52.191079    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:52.191711 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:52.191451    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2v6t" podUID="40156dc5-57f0-409f-a03f-2f0a80718d1d"
Sep 18 13:44:52.350574 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:52.350447    2572 generic.go:356] "Generic (PLEG): container finished" podID="4ffe1615-f9cc-4314-abe9-fbfafc8f67fd" containerID="bf2d93a35987484243f001fb6c9f4808edc5b8890118d266901acedd7a808da6" exitCode=0
Sep 18 13:44:52.350574 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:52.350551    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tqjnl" event={"ID":"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd","Type":"ContainerDied","Data":"bf2d93a35987484243f001fb6c9f4808edc5b8890118d266901acedd7a808da6"}
Sep 18 13:44:52.354385 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:52.354362    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h" event={"ID":"9f9099a5-ee27-4d4a-a737-1d2881b211e1","Type":"ContainerStarted","Data":"07ea9b37b34d7f6affc3a0834748c7fbb12eac28fbe9b4f501f925d8bcdbc997"}
Sep 18 13:44:52.399218 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:52.399171    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h" podStartSLOduration=8.333979923 podStartE2EDuration="26.399157546s" totalImagesPullingTime="18.065177623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:52.398986886 +0000 UTC m=+28.333952030" watchObservedRunningTime="2026-09-18 13:44:52.399157546 +0000 UTC m=+28.334122691"
Sep 18 13:44:53.359591 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:53.359497    2572 generic.go:356] "Generic (PLEG): container finished" podID="4ffe1615-f9cc-4314-abe9-fbfafc8f67fd" containerID="ef2406660894fad356f0ab3e2f363ca7406c2f750a03ba28ae5680fbecfd9386" exitCode=0
Sep 18 13:44:53.359591 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:53.359573    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tqjnl" event={"ID":"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd","Type":"ContainerDied","Data":"ef2406660894fad356f0ab3e2f363ca7406c2f750a03ba28ae5680fbecfd9386"}
Sep 18 13:44:53.360216 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:53.360106    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:53.360216 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:53.360180    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:53.360216 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:53.360220    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:53.375774 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:53.375748    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:53.375900 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:53.375873    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:44:53.938564 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:53.938383    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wstlx"]
Sep 18 13:44:53.938564 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:53.938517    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:53.938704 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:53.938623    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wstlx" podUID="07d3a346-ad98-4e7a-9ebc-77bfe00b06be"
Sep 18 13:44:53.941844 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:53.941817    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-stclq"]
Sep 18 13:44:53.942024 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:53.941940    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:53.942095 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:53.942027    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:53.942420 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:53.942398    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d2v6t"]
Sep 18 13:44:53.942518 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:53.942500    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:53.942661 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:53.942636    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2v6t" podUID="40156dc5-57f0-409f-a03f-2f0a80718d1d"
Sep 18 13:44:54.364852 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:54.364612    2572 generic.go:356] "Generic (PLEG): container finished" podID="4ffe1615-f9cc-4314-abe9-fbfafc8f67fd" containerID="3105eabe60a9d58f27715923f954de0d360b7ce0d10046096ef194aac74e1ae5" exitCode=0
Sep 18 13:44:54.364852 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:54.364691    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tqjnl" event={"ID":"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd","Type":"ContainerDied","Data":"3105eabe60a9d58f27715923f954de0d360b7ce0d10046096ef194aac74e1ae5"}
Sep 18 13:44:55.191775 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:55.191742    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:55.191950 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:55.191871    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:56.192083 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:56.192053    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:56.192083 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:56.192078    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:56.192700 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:56.192159    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wstlx" podUID="07d3a346-ad98-4e7a-9ebc-77bfe00b06be"
Sep 18 13:44:56.192700 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:56.192251    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2v6t" podUID="40156dc5-57f0-409f-a03f-2f0a80718d1d"
Sep 18 13:44:57.191525 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:57.191490    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:57.191723 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:57.191616    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:58.191113 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:58.191074    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:44:58.191709 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:58.191204    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wstlx" podUID="07d3a346-ad98-4e7a-9ebc-77bfe00b06be"
Sep 18 13:44:58.191709 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:58.191260    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:58.191709 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:58.191376    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2v6t" podUID="40156dc5-57f0-409f-a03f-2f0a80718d1d"
Sep 18 13:44:59.191432 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:59.191410    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:59.191880 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:59.191499    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stclq" podUID="cbb05c7c-6add-4a50-82c5-ca67901571f7"
Sep 18 13:44:59.380741 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:59.380711    2572 generic.go:356] "Generic (PLEG): container finished" podID="4ffe1615-f9cc-4314-abe9-fbfafc8f67fd" containerID="47edfc27d722b3feece5c5ddcd4f822b64e31ed3fe598bac376dc2fd987e10bc" exitCode=0
Sep 18 13:44:59.380980 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:59.380751    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tqjnl" event={"ID":"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd","Type":"ContainerDied","Data":"47edfc27d722b3feece5c5ddcd4f822b64e31ed3fe598bac376dc2fd987e10bc"}
Sep 18 13:44:59.854177 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:59.854085    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret\") pod \"global-pull-secret-syncer-stclq\" (UID: \"cbb05c7c-6add-4a50-82c5-ca67901571f7\") " pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:44:59.854443 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:59.854252    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:59.854443 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:59.854337    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret podName:cbb05c7c-6add-4a50-82c5-ca67901571f7 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:31.854315776 +0000 UTC m=+67.789280913 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret") pod "global-pull-secret-syncer-stclq" (UID: "cbb05c7c-6add-4a50-82c5-ca67901571f7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:59.955161 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:44:59.955125    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs\") pod \"network-metrics-daemon-d2v6t\" (UID: \"40156dc5-57f0-409f-a03f-2f0a80718d1d\") " pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:44:59.955331 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:59.955245    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:59.955331 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:44:59.955296    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs podName:40156dc5-57f0-409f-a03f-2f0a80718d1d nodeName:}" failed. No retries permitted until 2026-09-18 13:45:31.955282927 +0000 UTC m=+67.890248055 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs") pod "network-metrics-daemon-d2v6t" (UID: "40156dc5-57f0-409f-a03f-2f0a80718d1d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:45:00.056153 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.056115    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxqw\" (UniqueName: \"kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw\") pod \"network-check-target-wstlx\" (UID: \"07d3a346-ad98-4e7a-9ebc-77bfe00b06be\") " pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:45:00.056388 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:00.056276    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:45:00.056388 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:00.056297    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:45:00.056388 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:00.056307    2572 projected.go:196] Error preparing data for projected volume kube-api-access-dfxqw for pod openshift-network-diagnostics/network-check-target-wstlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:45:00.056388 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:00.056358    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw podName:07d3a346-ad98-4e7a-9ebc-77bfe00b06be nodeName:}" failed. No retries permitted until 2026-09-18 13:45:32.056344628 +0000 UTC m=+67.991309751 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dfxqw" (UniqueName: "kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw") pod "network-check-target-wstlx" (UID: "07d3a346-ad98-4e7a-9ebc-77bfe00b06be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:45:00.076876 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.076851    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-141.ec2.internal" event="NodeReady"
Sep 18 13:45:00.077011 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.076987    2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 13:45:00.175333 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.175266    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qmrw7"]
Sep 18 13:45:00.177024 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.177008    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:00.179710 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.179686    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jlrx5\""
Sep 18 13:45:00.179859 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.179844    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 13:45:00.179938 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.179924    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 13:45:00.185492 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.185471    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nm5v8"]
Sep 18 13:45:00.186994 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.186980    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nm5v8"
Sep 18 13:45:00.190990 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.190973    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:45:00.191946 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.191524    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:45:00.195743 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.195501    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 13:45:00.195743 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.195516    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 13:45:00.195743 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.195602    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 13:45:00.196023 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.195891    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p62gx\""
Sep 18 13:45:00.198061 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.198040    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7x5xj\""
Sep 18 13:45:00.198828 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.198813    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 13:45:00.198923 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.198842    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 13:45:00.198923 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.198913    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 13:45:00.199032 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.199016    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cttxw\""
Sep 18 13:45:00.202774 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.202753    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qmrw7"]
Sep 18 13:45:00.204485 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.204468    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nm5v8"]
Sep 18 13:45:00.257354 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.257326    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/233eb0fd-25e7-4116-88c8-c2aad00a3fac-config-volume\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:00.257474 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.257370    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:00.257474 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.257406    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b859p\" (UniqueName: \"kubernetes.io/projected/7f3f5b5d-0e90-4535-b2e9-d994388cbc51-kube-api-access-b859p\") pod \"ingress-canary-nm5v8\" (UID: \"7f3f5b5d-0e90-4535-b2e9-d994388cbc51\") " pod="openshift-ingress-canary/ingress-canary-nm5v8"
Sep 18 13:45:00.257474 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.257441    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f3f5b5d-0e90-4535-b2e9-d994388cbc51-cert\") pod \"ingress-canary-nm5v8\" (UID: \"7f3f5b5d-0e90-4535-b2e9-d994388cbc51\") " pod="openshift-ingress-canary/ingress-canary-nm5v8"
Sep 18 13:45:00.257630 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.257490    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/233eb0fd-25e7-4116-88c8-c2aad00a3fac-tmp-dir\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:00.257630 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.257527    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vlrk\" (UniqueName: \"kubernetes.io/projected/233eb0fd-25e7-4116-88c8-c2aad00a3fac-kube-api-access-6vlrk\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:00.358479 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.358437    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vlrk\" (UniqueName: \"kubernetes.io/projected/233eb0fd-25e7-4116-88c8-c2aad00a3fac-kube-api-access-6vlrk\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:00.358673 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.358591    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/233eb0fd-25e7-4116-88c8-c2aad00a3fac-config-volume\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:00.358673 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.358647    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:00.358780 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.358677    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b859p\" (UniqueName: \"kubernetes.io/projected/7f3f5b5d-0e90-4535-b2e9-d994388cbc51-kube-api-access-b859p\") pod \"ingress-canary-nm5v8\" (UID: \"7f3f5b5d-0e90-4535-b2e9-d994388cbc51\") " pod="openshift-ingress-canary/ingress-canary-nm5v8"
Sep 18 13:45:00.358780 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.358712    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f3f5b5d-0e90-4535-b2e9-d994388cbc51-cert\") pod \"ingress-canary-nm5v8\" (UID: \"7f3f5b5d-0e90-4535-b2e9-d994388cbc51\") " pod="openshift-ingress-canary/ingress-canary-nm5v8"
Sep 18 13:45:00.358780 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.358737    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/233eb0fd-25e7-4116-88c8-c2aad00a3fac-tmp-dir\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:00.358887 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:00.358829    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:45:00.358930 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:00.358919    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls podName:233eb0fd-25e7-4116-88c8-c2aad00a3fac nodeName:}" failed. No retries permitted until 2026-09-18 13:45:00.858900034 +0000 UTC m=+36.793865159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls") pod "dns-default-qmrw7" (UID: "233eb0fd-25e7-4116-88c8-c2aad00a3fac") : secret "dns-default-metrics-tls" not found
Sep 18 13:45:00.359776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.359751    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/233eb0fd-25e7-4116-88c8-c2aad00a3fac-tmp-dir\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:00.359981 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.359962    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/233eb0fd-25e7-4116-88c8-c2aad00a3fac-config-volume\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:00.362932 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.362913    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f3f5b5d-0e90-4535-b2e9-d994388cbc51-cert\") pod \"ingress-canary-nm5v8\" (UID: \"7f3f5b5d-0e90-4535-b2e9-d994388cbc51\") " pod="openshift-ingress-canary/ingress-canary-nm5v8"
Sep 18 13:45:00.372460 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.372440    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vlrk\" (UniqueName: \"kubernetes.io/projected/233eb0fd-25e7-4116-88c8-c2aad00a3fac-kube-api-access-6vlrk\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:00.372825 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.372806    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b859p\" (UniqueName: \"kubernetes.io/projected/7f3f5b5d-0e90-4535-b2e9-d994388cbc51-kube-api-access-b859p\") pod \"ingress-canary-nm5v8\" (UID: \"7f3f5b5d-0e90-4535-b2e9-d994388cbc51\") " pod="openshift-ingress-canary/ingress-canary-nm5v8"
Sep 18 13:45:00.385666 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.385639    2572 generic.go:356] "Generic (PLEG): container finished" podID="4ffe1615-f9cc-4314-abe9-fbfafc8f67fd" containerID="d3decf60eb6115d168193e55fba2776204b824744e9a53537eec76d6e21bc6f7" exitCode=0
Sep 18 13:45:00.385779 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.385676    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tqjnl" event={"ID":"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd","Type":"ContainerDied","Data":"d3decf60eb6115d168193e55fba2776204b824744e9a53537eec76d6e21bc6f7"}
Sep 18 13:45:00.499870 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.499712    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nm5v8"
Sep 18 13:45:00.773494 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.773454    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nm5v8"]
Sep 18 13:45:00.778461 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:45:00.778435    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3f5b5d_0e90_4535_b2e9_d994388cbc51.slice/crio-88771a2d2eee3e881a0d201c1db34d2a1fc595f568805225d569926920605e3c WatchSource:0}: Error finding container 88771a2d2eee3e881a0d201c1db34d2a1fc595f568805225d569926920605e3c: Status 404 returned error can't find the container with id 88771a2d2eee3e881a0d201c1db34d2a1fc595f568805225d569926920605e3c
Sep 18 13:45:00.864206 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:00.864164    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:00.864371 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:00.864300    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:45:00.864371 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:00.864350    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls podName:233eb0fd-25e7-4116-88c8-c2aad00a3fac nodeName:}" failed. No retries permitted until 2026-09-18 13:45:01.864337802 +0000 UTC m=+37.799302937 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls") pod "dns-default-qmrw7" (UID: "233eb0fd-25e7-4116-88c8-c2aad00a3fac") : secret "dns-default-metrics-tls" not found
Sep 18 13:45:01.191049 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:01.191026    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:45:01.193262 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:01.193245    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 13:45:01.392170 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:01.392133    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tqjnl" event={"ID":"4ffe1615-f9cc-4314-abe9-fbfafc8f67fd","Type":"ContainerStarted","Data":"0c0cb332a58852297d379009765e48365e028efc05fd4642d7cf1f4be6119bd4"}
Sep 18 13:45:01.393463 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:01.393432    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nm5v8" event={"ID":"7f3f5b5d-0e90-4535-b2e9-d994388cbc51","Type":"ContainerStarted","Data":"88771a2d2eee3e881a0d201c1db34d2a1fc595f568805225d569926920605e3c"}
Sep 18 13:45:01.418050 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:01.417980    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tqjnl" podStartSLOduration=8.449376472 podStartE2EDuration="35.417960908s" totalImagesPullingTime="26.968584436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:01.41405777 +0000 UTC m=+37.349022918" watchObservedRunningTime="2026-09-18 13:45:01.417960908 +0000 UTC m=+37.352926055"
Sep 18 13:45:01.870711 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:01.870673    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:01.870916 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:01.870831    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:45:01.870977 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:01.870929    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls podName:233eb0fd-25e7-4116-88c8-c2aad00a3fac nodeName:}" failed. No retries permitted until 2026-09-18 13:45:03.870905458 +0000 UTC m=+39.805870602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls") pod "dns-default-qmrw7" (UID: "233eb0fd-25e7-4116-88c8-c2aad00a3fac") : secret "dns-default-metrics-tls" not found
Sep 18 13:45:03.400469 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:03.400256    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nm5v8" event={"ID":"7f3f5b5d-0e90-4535-b2e9-d994388cbc51","Type":"ContainerStarted","Data":"58c114c6ecf1aa17ccfa10460c0b614448963fdb784006d44183acb16de8e591"}
Sep 18 13:45:03.415650 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:03.415610    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nm5v8" podStartSLOduration=1.182855339 podStartE2EDuration="3.415575535s" totalImagesPullingTime="2.232720196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:03.414909076 +0000 UTC m=+39.349874235" watchObservedRunningTime="2026-09-18 13:45:03.415575535 +0000 UTC m=+39.350540680"
Sep 18 13:45:03.884817 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:03.884765    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:03.885006 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:03.884913    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:45:03.885006 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:03.884976    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls podName:233eb0fd-25e7-4116-88c8-c2aad00a3fac nodeName:}" failed. No retries permitted until 2026-09-18 13:45:07.884959909 +0000 UTC m=+43.819925037 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls") pod "dns-default-qmrw7" (UID: "233eb0fd-25e7-4116-88c8-c2aad00a3fac") : secret "dns-default-metrics-tls" not found
Sep 18 13:45:07.909160 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:07.909112    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:07.909670 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:07.909267    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:45:07.909670 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:07.909331    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls podName:233eb0fd-25e7-4116-88c8-c2aad00a3fac nodeName:}" failed. No retries permitted until 2026-09-18 13:45:15.909317623 +0000 UTC m=+51.844282758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls") pod "dns-default-qmrw7" (UID: "233eb0fd-25e7-4116-88c8-c2aad00a3fac") : secret "dns-default-metrics-tls" not found
Sep 18 13:45:14.616349 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.616312    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz"]
Sep 18 13:45:14.660579 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.660526    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-lj48l"]
Sep 18 13:45:14.660717 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.660648    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz"
Sep 18 13:45:14.664854 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.664833    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 13:45:14.665611 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.665597    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:45:14.665709 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.665691    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 13:45:14.666056 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.666037    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8pgj8\""
Sep 18 13:45:14.666589 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.666575    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 13:45:14.672647 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.672625    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz"]
Sep 18 13:45:14.672723 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.672658    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-lj48l"]
Sep 18 13:45:14.672759 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.672726    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:14.675142 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.675126    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 13:45:14.675385 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.675370    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:45:14.675520 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.675507    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-b4ls7\""
Sep 18 13:45:14.675606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.675576    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 13:45:14.675672 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.675643    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 13:45:14.704375 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.704352    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 13:45:14.737815 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.737789    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77d48d976-6gts2"]
Sep 18 13:45:14.752900 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.752882    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24a56b4d-b52e-4af5-b13e-5e9d26a998e8-serving-cert\") pod \"console-operator-fd4bb46c-lj48l\" (UID: \"24a56b4d-b52e-4af5-b13e-5e9d26a998e8\") " pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:14.752989 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.752909    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2cb3e59e-5c9c-486d-89dd-99474a7edade-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-4wcgz\" (UID: \"2cb3e59e-5c9c-486d-89dd-99474a7edade\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz"
Sep 18 13:45:14.752989 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.752939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2cb3e59e-5c9c-486d-89dd-99474a7edade-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-4wcgz\" (UID: \"2cb3e59e-5c9c-486d-89dd-99474a7edade\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz"
Sep 18 13:45:14.753064 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.753049    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24a56b4d-b52e-4af5-b13e-5e9d26a998e8-config\") pod \"console-operator-fd4bb46c-lj48l\" (UID: \"24a56b4d-b52e-4af5-b13e-5e9d26a998e8\") " pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:14.753096 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.753082    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp97l\" (UniqueName: \"kubernetes.io/projected/24a56b4d-b52e-4af5-b13e-5e9d26a998e8-kube-api-access-kp97l\") pod \"console-operator-fd4bb46c-lj48l\" (UID: \"24a56b4d-b52e-4af5-b13e-5e9d26a998e8\") " pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:14.753128 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.753114    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24a56b4d-b52e-4af5-b13e-5e9d26a998e8-trusted-ca\") pod \"console-operator-fd4bb46c-lj48l\" (UID: \"24a56b4d-b52e-4af5-b13e-5e9d26a998e8\") " pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:14.753173 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.753157    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhq8m\" (UniqueName: \"kubernetes.io/projected/2cb3e59e-5c9c-486d-89dd-99474a7edade-kube-api-access-zhq8m\") pod \"kube-storage-version-migrator-operator-55889f7b47-4wcgz\" (UID: \"2cb3e59e-5c9c-486d-89dd-99474a7edade\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz"
Sep 18 13:45:14.755864 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.755842    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.761350 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.761318    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 13:45:14.761463 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.761444    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 13:45:14.761668 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.761647    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 13:45:14.761774 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.761735    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zf5xv\""
Sep 18 13:45:14.769608 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.769585    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"]
Sep 18 13:45:14.776657 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.776634    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 13:45:14.785286 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.785266    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lb7tl"]
Sep 18 13:45:14.785403 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.785390    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:14.789624 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.789608    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 13:45:14.789948 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.789934    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wtcms\""
Sep 18 13:45:14.790312 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.790297    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 13:45:14.791578 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.791563    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 13:45:14.793303 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.793289    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 13:45:14.806826 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.806807    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lb7tl"
Sep 18 13:45:14.806941 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.806810    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77d48d976-6gts2"]
Sep 18 13:45:14.807016 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.806981    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"]
Sep 18 13:45:14.809265 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.809242    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-c9ljv\""
Sep 18 13:45:14.809371 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.809243    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:45:14.809435 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.809426    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 13:45:14.813470 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.813451    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lb7tl"]
Sep 18 13:45:14.836706 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.836679    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-x95zg"]
Sep 18 13:45:14.852745 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.852723    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-x95zg"]
Sep 18 13:45:14.852896 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.852823    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-x95zg"
Sep 18 13:45:14.853931 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.853909    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb3669f1-1d97-46e9-b533-a4dbbb942be0-image-registry-private-configuration\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.854043 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.853939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.854043 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.853961    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb3669f1-1d97-46e9-b533-a4dbbb942be0-installation-pull-secrets\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.854043 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.853983    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6d2p\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-kube-api-access-z6d2p\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.854043 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.854015    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24a56b4d-b52e-4af5-b13e-5e9d26a998e8-config\") pod \"console-operator-fd4bb46c-lj48l\" (UID: \"24a56b4d-b52e-4af5-b13e-5e9d26a998e8\") " pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:14.854208 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.854112    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp97l\" (UniqueName: \"kubernetes.io/projected/24a56b4d-b52e-4af5-b13e-5e9d26a998e8-kube-api-access-kp97l\") pod \"console-operator-fd4bb46c-lj48l\" (UID: \"24a56b4d-b52e-4af5-b13e-5e9d26a998e8\") " pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:14.854208 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.854143    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24a56b4d-b52e-4af5-b13e-5e9d26a998e8-trusted-ca\") pod \"console-operator-fd4bb46c-lj48l\" (UID: \"24a56b4d-b52e-4af5-b13e-5e9d26a998e8\") " pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:14.854208 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.854181    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhq8m\" (UniqueName: \"kubernetes.io/projected/2cb3e59e-5c9c-486d-89dd-99474a7edade-kube-api-access-zhq8m\") pod \"kube-storage-version-migrator-operator-55889f7b47-4wcgz\" (UID: \"2cb3e59e-5c9c-486d-89dd-99474a7edade\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz"
Sep 18 13:45:14.854349 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.854223    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24a56b4d-b52e-4af5-b13e-5e9d26a998e8-serving-cert\") pod \"console-operator-fd4bb46c-lj48l\" (UID: \"24a56b4d-b52e-4af5-b13e-5e9d26a998e8\") " pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:14.854349 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.854254    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2cb3e59e-5c9c-486d-89dd-99474a7edade-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-4wcgz\" (UID: \"2cb3e59e-5c9c-486d-89dd-99474a7edade\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz"
Sep 18 13:45:14.854349 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.854280    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb3669f1-1d97-46e9-b533-a4dbbb942be0-trusted-ca\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.854349 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.854324    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2cb3e59e-5c9c-486d-89dd-99474a7edade-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-4wcgz\" (UID: \"2cb3e59e-5c9c-486d-89dd-99474a7edade\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz"
Sep 18 13:45:14.854349 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.854347    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-bound-sa-token\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.854601 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.854437    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb3669f1-1d97-46e9-b533-a4dbbb942be0-ca-trust-extracted\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.854800 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.854767    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24a56b4d-b52e-4af5-b13e-5e9d26a998e8-config\") pod \"console-operator-fd4bb46c-lj48l\" (UID: \"24a56b4d-b52e-4af5-b13e-5e9d26a998e8\") " pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:14.854872 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.854821    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-certificates\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.855074 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.855051    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2cb3e59e-5c9c-486d-89dd-99474a7edade-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-4wcgz\" (UID: \"2cb3e59e-5c9c-486d-89dd-99474a7edade\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz"
Sep 18 13:45:14.855251 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.855230    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hvgnn\""
Sep 18 13:45:14.855827 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.855803    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24a56b4d-b52e-4af5-b13e-5e9d26a998e8-trusted-ca\") pod \"console-operator-fd4bb46c-lj48l\" (UID: \"24a56b4d-b52e-4af5-b13e-5e9d26a998e8\") " pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:14.858517 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.858499    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2cb3e59e-5c9c-486d-89dd-99474a7edade-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-4wcgz\" (UID: \"2cb3e59e-5c9c-486d-89dd-99474a7edade\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz"
Sep 18 13:45:14.858627 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.858525    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24a56b4d-b52e-4af5-b13e-5e9d26a998e8-serving-cert\") pod \"console-operator-fd4bb46c-lj48l\" (UID: \"24a56b4d-b52e-4af5-b13e-5e9d26a998e8\") " pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:14.869904 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.869855    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhq8m\" (UniqueName: \"kubernetes.io/projected/2cb3e59e-5c9c-486d-89dd-99474a7edade-kube-api-access-zhq8m\") pod \"kube-storage-version-migrator-operator-55889f7b47-4wcgz\" (UID: \"2cb3e59e-5c9c-486d-89dd-99474a7edade\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz"
Sep 18 13:45:14.873498 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.873480    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp97l\" (UniqueName: \"kubernetes.io/projected/24a56b4d-b52e-4af5-b13e-5e9d26a998e8-kube-api-access-kp97l\") pod \"console-operator-fd4bb46c-lj48l\" (UID: \"24a56b4d-b52e-4af5-b13e-5e9d26a998e8\") " pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:14.937195 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.937169    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-77578d5fdb-w9ttn"]
Sep 18 13:45:14.956118 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.956095    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb3669f1-1d97-46e9-b533-a4dbbb942be0-ca-trust-extracted\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.956213 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.956123    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-certificates\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.956213 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.956144    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4bbr\" (UniqueName: \"kubernetes.io/projected/e3650f9e-a149-425c-9c57-ad099dedf32e-kube-api-access-b4bbr\") pod \"network-check-source-7cbc45fddc-x95zg\" (UID: \"e3650f9e-a149-425c-9c57-ad099dedf32e\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-x95zg"
Sep 18 13:45:14.956327 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.956304    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb3669f1-1d97-46e9-b533-a4dbbb942be0-image-registry-private-configuration\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.956379 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.956343    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.956431 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.956377    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb3669f1-1d97-46e9-b533-a4dbbb942be0-installation-pull-secrets\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.956431 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.956406    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6d2p\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-kube-api-access-z6d2p\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.956528 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.956443    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb3669f1-1d97-46e9-b533-a4dbbb942be0-ca-trust-extracted\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.956528 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.956452    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c25c7\" (UniqueName: \"kubernetes.io/projected/0722ceea-9544-4e73-84f2-aae4fbe4ff98-kube-api-access-c25c7\") pod \"cluster-monitoring-operator-847b8bd8d7-58q2l\" (UID: \"0722ceea-9544-4e73-84f2-aae4fbe4ff98\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:14.956528 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:14.956473    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:45:14.956528 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.956489    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kpv9\" (UniqueName: \"kubernetes.io/projected/1b1982f4-552c-47bb-8679-cfaf7091f1be-kube-api-access-8kpv9\") pod \"volume-data-source-validator-5c4db8bdf9-lb7tl\" (UID: \"1b1982f4-552c-47bb-8679-cfaf7091f1be\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lb7tl"
Sep 18 13:45:14.956741 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.956528    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0722ceea-9544-4e73-84f2-aae4fbe4ff98-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-58q2l\" (UID: \"0722ceea-9544-4e73-84f2-aae4fbe4ff98\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:14.956741 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.956590    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb3669f1-1d97-46e9-b533-a4dbbb942be0-trusted-ca\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.956741 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.956640    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-bound-sa-token\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.956741 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.956667    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-58q2l\" (UID: \"0722ceea-9544-4e73-84f2-aae4fbe4ff98\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:14.956741 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:14.956493    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77d48d976-6gts2: secret "image-registry-tls" not found
Sep 18 13:45:14.956741 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.956735    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-certificates\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.957037 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:14.956777    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls podName:cb3669f1-1d97-46e9-b533-a4dbbb942be0 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:15.456756564 +0000 UTC m=+51.391721717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls") pod "image-registry-77d48d976-6gts2" (UID: "cb3669f1-1d97-46e9-b533-a4dbbb942be0") : secret "image-registry-tls" not found
Sep 18 13:45:14.957592 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.957567    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb3669f1-1d97-46e9-b533-a4dbbb942be0-trusted-ca\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.958928 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.958903    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb3669f1-1d97-46e9-b533-a4dbbb942be0-image-registry-private-configuration\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.959177 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.959158    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb3669f1-1d97-46e9-b533-a4dbbb942be0-installation-pull-secrets\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.965471 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.965446    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77578d5fdb-w9ttn"]
Sep 18 13:45:14.965579 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.965569    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:14.965991 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.965969    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6d2p\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-kube-api-access-z6d2p\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.968416 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.968399    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 13:45:14.968519 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.968492    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 13:45:14.968589 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.968559    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 18 13:45:14.968681 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.968660    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 13:45:14.968870 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.968849    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-njvq6\""
Sep 18 13:45:14.968962 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.968854    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 13:45:14.969336 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.969321    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-bound-sa-token\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:14.969383 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.969323    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 13:45:14.970223 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.970209    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz"
Sep 18 13:45:14.981967 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:14.981947    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:15.057552 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.057301    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:15.057643 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.057604    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4bbr\" (UniqueName: \"kubernetes.io/projected/e3650f9e-a149-425c-9c57-ad099dedf32e-kube-api-access-b4bbr\") pod \"network-check-source-7cbc45fddc-x95zg\" (UID: \"e3650f9e-a149-425c-9c57-ad099dedf32e\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-x95zg"
Sep 18 13:45:15.057707 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.057685    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-stats-auth\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:15.057793 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.057768    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:15.057842 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.057832    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c25c7\" (UniqueName: \"kubernetes.io/projected/0722ceea-9544-4e73-84f2-aae4fbe4ff98-kube-api-access-c25c7\") pod \"cluster-monitoring-operator-847b8bd8d7-58q2l\" (UID: \"0722ceea-9544-4e73-84f2-aae4fbe4ff98\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:15.058056 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.057859    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kpv9\" (UniqueName: \"kubernetes.io/projected/1b1982f4-552c-47bb-8679-cfaf7091f1be-kube-api-access-8kpv9\") pod \"volume-data-source-validator-5c4db8bdf9-lb7tl\" (UID: \"1b1982f4-552c-47bb-8679-cfaf7091f1be\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lb7tl"
Sep 18 13:45:15.058153 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.058111    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0722ceea-9544-4e73-84f2-aae4fbe4ff98-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-58q2l\" (UID: \"0722ceea-9544-4e73-84f2-aae4fbe4ff98\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:15.058213 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.058185    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-58q2l\" (UID: \"0722ceea-9544-4e73-84f2-aae4fbe4ff98\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:15.058272 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.058238    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-default-certificate\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:15.058425 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.058329    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcwh6\" (UniqueName: \"kubernetes.io/projected/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-kube-api-access-xcwh6\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:15.058425 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:15.058355    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:15.058670 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:15.058435    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls podName:0722ceea-9544-4e73-84f2-aae4fbe4ff98 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:15.558413907 +0000 UTC m=+51.493379049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-58q2l" (UID: "0722ceea-9544-4e73-84f2-aae4fbe4ff98") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:15.058894 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.058873    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0722ceea-9544-4e73-84f2-aae4fbe4ff98-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-58q2l\" (UID: \"0722ceea-9544-4e73-84f2-aae4fbe4ff98\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:15.068270 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.067808    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4bbr\" (UniqueName: \"kubernetes.io/projected/e3650f9e-a149-425c-9c57-ad099dedf32e-kube-api-access-b4bbr\") pod \"network-check-source-7cbc45fddc-x95zg\" (UID: \"e3650f9e-a149-425c-9c57-ad099dedf32e\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-x95zg"
Sep 18 13:45:15.071178 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.071158    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kpv9\" (UniqueName: \"kubernetes.io/projected/1b1982f4-552c-47bb-8679-cfaf7091f1be-kube-api-access-8kpv9\") pod \"volume-data-source-validator-5c4db8bdf9-lb7tl\" (UID: \"1b1982f4-552c-47bb-8679-cfaf7091f1be\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lb7tl"
Sep 18 13:45:15.071304 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.071286    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c25c7\" (UniqueName: \"kubernetes.io/projected/0722ceea-9544-4e73-84f2-aae4fbe4ff98-kube-api-access-c25c7\") pod \"cluster-monitoring-operator-847b8bd8d7-58q2l\" (UID: \"0722ceea-9544-4e73-84f2-aae4fbe4ff98\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:15.115257 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.115235    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lb7tl"
Sep 18 13:45:15.158706 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.158685    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:15.158787 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.158717    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-stats-auth\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:15.158787 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.158739    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:15.158876 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.158788    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-default-certificate\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:15.158932 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:15.158887    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle podName:5b1b97a5-adb1-4a5f-89a9-785ea463af4e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:15.658871158 +0000 UTC m=+51.593836280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle") pod "router-default-77578d5fdb-w9ttn" (UID: "5b1b97a5-adb1-4a5f-89a9-785ea463af4e") : configmap references non-existent config key: service-ca.crt
Sep 18 13:45:15.158995 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:15.158951    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:45:15.158995 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.158974    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcwh6\" (UniqueName: \"kubernetes.io/projected/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-kube-api-access-xcwh6\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:15.159088 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:15.159024    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs podName:5b1b97a5-adb1-4a5f-89a9-785ea463af4e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:15.659006397 +0000 UTC m=+51.593971528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs") pod "router-default-77578d5fdb-w9ttn" (UID: "5b1b97a5-adb1-4a5f-89a9-785ea463af4e") : secret "router-metrics-certs-default" not found
Sep 18 13:45:15.160901 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.160879    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-default-certificate\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:15.161145 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.161126    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-stats-auth\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:15.166682 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.166649    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-x95zg"
Sep 18 13:45:15.167714 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.167691    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcwh6\" (UniqueName: \"kubernetes.io/projected/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-kube-api-access-xcwh6\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:15.258465 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.258441    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz"]
Sep 18 13:45:15.261192 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:45:15.261162    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb3e59e_5c9c_486d_89dd_99474a7edade.slice/crio-403657c3abc50d7bb616dbe7b18ff0074ecdf4beed2cc1be782c98760519b6f2 WatchSource:0}: Error finding container 403657c3abc50d7bb616dbe7b18ff0074ecdf4beed2cc1be782c98760519b6f2: Status 404 returned error can't find the container with id 403657c3abc50d7bb616dbe7b18ff0074ecdf4beed2cc1be782c98760519b6f2
Sep 18 13:45:15.275047 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.275025    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-lj48l"]
Sep 18 13:45:15.277439 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:45:15.277417    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a56b4d_b52e_4af5_b13e_5e9d26a998e8.slice/crio-68b39514a51dae920605db902990ad4bccdaae4a2ec0472bfbac5c2c199e6ade WatchSource:0}: Error finding container 68b39514a51dae920605db902990ad4bccdaae4a2ec0472bfbac5c2c199e6ade: Status 404 returned error can't find the container with id 68b39514a51dae920605db902990ad4bccdaae4a2ec0472bfbac5c2c199e6ade
Sep 18 13:45:15.412329 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.412253    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lb7tl"]
Sep 18 13:45:15.415174 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:45:15.415151    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1982f4_552c_47bb_8679_cfaf7091f1be.slice/crio-764458cf618620c2fe4b12f8d2fee0dc1dd913e66e0088c84847e19ded41ec17 WatchSource:0}: Error finding container 764458cf618620c2fe4b12f8d2fee0dc1dd913e66e0088c84847e19ded41ec17: Status 404 returned error can't find the container with id 764458cf618620c2fe4b12f8d2fee0dc1dd913e66e0088c84847e19ded41ec17
Sep 18 13:45:15.427770 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.427747    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lb7tl" event={"ID":"1b1982f4-552c-47bb-8679-cfaf7091f1be","Type":"ContainerStarted","Data":"764458cf618620c2fe4b12f8d2fee0dc1dd913e66e0088c84847e19ded41ec17"}
Sep 18 13:45:15.428884 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.428854    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l" event={"ID":"24a56b4d-b52e-4af5-b13e-5e9d26a998e8","Type":"ContainerStarted","Data":"68b39514a51dae920605db902990ad4bccdaae4a2ec0472bfbac5c2c199e6ade"}
Sep 18 13:45:15.429947 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.429929    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz" event={"ID":"2cb3e59e-5c9c-486d-89dd-99474a7edade","Type":"ContainerStarted","Data":"403657c3abc50d7bb616dbe7b18ff0074ecdf4beed2cc1be782c98760519b6f2"}
Sep 18 13:45:15.461456 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.461426    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:15.461608 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:15.461590    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:45:15.461653 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:15.461610    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77d48d976-6gts2: secret "image-registry-tls" not found
Sep 18 13:45:15.461691 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:15.461659    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls podName:cb3669f1-1d97-46e9-b533-a4dbbb942be0 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:16.461644287 +0000 UTC m=+52.396609410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls") pod "image-registry-77d48d976-6gts2" (UID: "cb3669f1-1d97-46e9-b533-a4dbbb942be0") : secret "image-registry-tls" not found
Sep 18 13:45:15.463714 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.463692    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-x95zg"]
Sep 18 13:45:15.466515 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:45:15.466495    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3650f9e_a149_425c_9c57_ad099dedf32e.slice/crio-5221b86e40a2e35995d8b211f3b082e4e715f0af10f216f301378ac5e7f3c397 WatchSource:0}: Error finding container 5221b86e40a2e35995d8b211f3b082e4e715f0af10f216f301378ac5e7f3c397: Status 404 returned error can't find the container with id 5221b86e40a2e35995d8b211f3b082e4e715f0af10f216f301378ac5e7f3c397
Sep 18 13:45:15.562397 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.562364    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-58q2l\" (UID: \"0722ceea-9544-4e73-84f2-aae4fbe4ff98\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:15.562610 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:15.562566    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:15.562679 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:15.562646    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls podName:0722ceea-9544-4e73-84f2-aae4fbe4ff98 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:16.562626739 +0000 UTC m=+52.497591864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-58q2l" (UID: "0722ceea-9544-4e73-84f2-aae4fbe4ff98") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:15.663656 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.663581    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:15.663656 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.663628    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:15.664183 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:15.663739    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle podName:5b1b97a5-adb1-4a5f-89a9-785ea463af4e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:16.663721917 +0000 UTC m=+52.598687077 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle") pod "router-default-77578d5fdb-w9ttn" (UID: "5b1b97a5-adb1-4a5f-89a9-785ea463af4e") : configmap references non-existent config key: service-ca.crt
Sep 18 13:45:15.664183 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:15.663787    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:45:15.664183 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:15.663836    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs podName:5b1b97a5-adb1-4a5f-89a9-785ea463af4e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:16.663825132 +0000 UTC m=+52.598790255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs") pod "router-default-77578d5fdb-w9ttn" (UID: "5b1b97a5-adb1-4a5f-89a9-785ea463af4e") : secret "router-metrics-certs-default" not found
Sep 18 13:45:15.966381 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:15.966269    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:15.966662 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:15.966433    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:45:15.966662 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:15.966507    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls podName:233eb0fd-25e7-4116-88c8-c2aad00a3fac nodeName:}" failed. No retries permitted until 2026-09-18 13:45:31.966484165 +0000 UTC m=+67.901449291 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls") pod "dns-default-qmrw7" (UID: "233eb0fd-25e7-4116-88c8-c2aad00a3fac") : secret "dns-default-metrics-tls" not found
Sep 18 13:45:16.436412 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:16.436327    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-x95zg" event={"ID":"e3650f9e-a149-425c-9c57-ad099dedf32e","Type":"ContainerStarted","Data":"5221b86e40a2e35995d8b211f3b082e4e715f0af10f216f301378ac5e7f3c397"}
Sep 18 13:45:16.472667 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:16.472006    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:16.472667 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:16.472234    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:45:16.472667 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:16.472253    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77d48d976-6gts2: secret "image-registry-tls" not found
Sep 18 13:45:16.472667 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:16.472316    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls podName:cb3669f1-1d97-46e9-b533-a4dbbb942be0 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:18.472297475 +0000 UTC m=+54.407262620 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls") pod "image-registry-77d48d976-6gts2" (UID: "cb3669f1-1d97-46e9-b533-a4dbbb942be0") : secret "image-registry-tls" not found
Sep 18 13:45:16.573281 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:16.572659    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-58q2l\" (UID: \"0722ceea-9544-4e73-84f2-aae4fbe4ff98\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:16.573281 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:16.572965    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:16.573281 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:16.573050    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls podName:0722ceea-9544-4e73-84f2-aae4fbe4ff98 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:18.573028845 +0000 UTC m=+54.507993969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-58q2l" (UID: "0722ceea-9544-4e73-84f2-aae4fbe4ff98") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:16.673591 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:16.673477    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:16.674065 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:16.673657    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:45:16.674065 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:16.673740    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:16.674065 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:16.673802    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs podName:5b1b97a5-adb1-4a5f-89a9-785ea463af4e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:18.673782325 +0000 UTC m=+54.608747449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs") pod "router-default-77578d5fdb-w9ttn" (UID: "5b1b97a5-adb1-4a5f-89a9-785ea463af4e") : secret "router-metrics-certs-default" not found
Sep 18 13:45:16.674065 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:16.673887    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle podName:5b1b97a5-adb1-4a5f-89a9-785ea463af4e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:18.67386945 +0000 UTC m=+54.608834576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle") pod "router-default-77578d5fdb-w9ttn" (UID: "5b1b97a5-adb1-4a5f-89a9-785ea463af4e") : configmap references non-existent config key: service-ca.crt
Sep 18 13:45:18.490682 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:18.490647    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:18.491112 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:18.490823    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:45:18.491112 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:18.490846    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77d48d976-6gts2: secret "image-registry-tls" not found
Sep 18 13:45:18.491112 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:18.490921    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls podName:cb3669f1-1d97-46e9-b533-a4dbbb942be0 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:22.49089325 +0000 UTC m=+58.425858375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls") pod "image-registry-77d48d976-6gts2" (UID: "cb3669f1-1d97-46e9-b533-a4dbbb942be0") : secret "image-registry-tls" not found
Sep 18 13:45:18.592051 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:18.592013    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-58q2l\" (UID: \"0722ceea-9544-4e73-84f2-aae4fbe4ff98\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:18.592228 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:18.592195    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:18.592294 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:18.592284    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls podName:0722ceea-9544-4e73-84f2-aae4fbe4ff98 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:22.592263224 +0000 UTC m=+58.527228365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-58q2l" (UID: "0722ceea-9544-4e73-84f2-aae4fbe4ff98") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:18.692903 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:18.692865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:18.693082 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:18.693015    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:18.693082 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:18.693035    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:45:18.693196 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:18.693107    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs podName:5b1b97a5-adb1-4a5f-89a9-785ea463af4e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:22.693087982 +0000 UTC m=+58.628053120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs") pod "router-default-77578d5fdb-w9ttn" (UID: "5b1b97a5-adb1-4a5f-89a9-785ea463af4e") : secret "router-metrics-certs-default" not found
Sep 18 13:45:18.693196 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:18.693129    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle podName:5b1b97a5-adb1-4a5f-89a9-785ea463af4e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:22.693113779 +0000 UTC m=+58.628078925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle") pod "router-default-77578d5fdb-w9ttn" (UID: "5b1b97a5-adb1-4a5f-89a9-785ea463af4e") : configmap references non-existent config key: service-ca.crt
Sep 18 13:45:19.447844 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.447653    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz" event={"ID":"2cb3e59e-5c9c-486d-89dd-99474a7edade","Type":"ContainerStarted","Data":"08faeddb5f9fab25a52724f7744a6ba8467af6a44060ba9991f524394f678b8b"}
Sep 18 13:45:19.449232 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.449204    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lb7tl" event={"ID":"1b1982f4-552c-47bb-8679-cfaf7091f1be","Type":"ContainerStarted","Data":"369fea82ad975ed9eda9a3cdb8b3654e364a27f0c557df97e96be9f067e99056"}
Sep 18 13:45:19.450643 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.450622    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l" event={"ID":"24a56b4d-b52e-4af5-b13e-5e9d26a998e8","Type":"ContainerStarted","Data":"cbe9752596e4b8fdfbafb2d645f87de697046f1b13ff158ef5661f37acad7cb4"}
Sep 18 13:45:19.450830 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.450813    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:19.452136 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.452116    2572 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-lj48l container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.9:8443/readyz\": dial tcp 10.133.0.9:8443: connect: connection refused" start-of-body=
Sep 18 13:45:19.452216 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.452155    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l" podUID="24a56b4d-b52e-4af5-b13e-5e9d26a998e8" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.9:8443/readyz\": dial tcp 10.133.0.9:8443: connect: connection refused"
Sep 18 13:45:19.470363 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.470325    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz" podStartSLOduration=1.490914281 podStartE2EDuration="5.47031252s" totalImagesPullingTime="3.979398239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:19.468911098 +0000 UTC m=+55.403876242" watchObservedRunningTime="2026-09-18 13:45:19.47031252 +0000 UTC m=+55.405277664"
Sep 18 13:45:19.488267 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.488196    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l" podStartSLOduration=1.5295677460000001 podStartE2EDuration="5.488183583s" totalImagesPullingTime="3.958615837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:19.487743415 +0000 UTC m=+55.422708561" watchObservedRunningTime="2026-09-18 13:45:19.488183583 +0000 UTC m=+55.423148718"
Sep 18 13:45:19.504382 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.504345    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lb7tl" podStartSLOduration=1.6846765270000001 podStartE2EDuration="5.504332785s" totalImagesPullingTime="3.819656258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:19.50319617 +0000 UTC m=+55.438161315" watchObservedRunningTime="2026-09-18 13:45:19.504332785 +0000 UTC m=+55.439297974"
Sep 18 13:45:19.691798 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.691773    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bzrnp"]
Sep 18 13:45:19.717887 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.717862    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bzrnp"]
Sep 18 13:45:19.718028 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.717979    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:19.720155 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.720127    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 13:45:19.720283 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.720188    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 13:45:19.720500 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.720462    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 13:45:19.720500 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.720471    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 13:45:19.720782 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.720767    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8k78s\""
Sep 18 13:45:19.905147 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.905055    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:19.905147 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.905115    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-data-volume\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:19.905356 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.905187    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-crio-socket\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:19.905356 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.905273    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:19.905356 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:19.905341    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fctmm\" (UniqueName: \"kubernetes.io/projected/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-api-access-fctmm\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:20.005877 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:20.005843    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:20.006028 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:20.005889    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-data-volume\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:20.006028 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:20.005929    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-crio-socket\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:20.006028 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:20.005962    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:45:20.006028 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:20.005966    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:20.006172 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:20.006040    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm podName:1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:20.50602016 +0000 UTC m=+56.440985299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzrnp" (UID: "1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e") : configmap "kube-rbac-proxy" not found
Sep 18 13:45:20.006172 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:20.006059    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:20.006172 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:20.006126    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fctmm\" (UniqueName: \"kubernetes.io/projected/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-api-access-fctmm\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:20.006320 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:20.006201    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls podName:1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:20.506189813 +0000 UTC m=+56.441154936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzrnp" (UID: "1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:20.006320 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:20.006221    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-crio-socket\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:20.006320 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:20.006262    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-data-volume\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:20.016157 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:20.016133    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fctmm\" (UniqueName: \"kubernetes.io/projected/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-api-access-fctmm\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:20.455058 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:20.455030    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/0.log"
Sep 18 13:45:20.455234 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:20.455067    2572 generic.go:356] "Generic (PLEG): container finished" podID="24a56b4d-b52e-4af5-b13e-5e9d26a998e8" containerID="cbe9752596e4b8fdfbafb2d645f87de697046f1b13ff158ef5661f37acad7cb4" exitCode=255
Sep 18 13:45:20.455234 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:20.455159    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l" event={"ID":"24a56b4d-b52e-4af5-b13e-5e9d26a998e8","Type":"ContainerDied","Data":"cbe9752596e4b8fdfbafb2d645f87de697046f1b13ff158ef5661f37acad7cb4"}
Sep 18 13:45:20.455362 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:20.455347    2572 scope.go:117] "RemoveContainer" containerID="cbe9752596e4b8fdfbafb2d645f87de697046f1b13ff158ef5661f37acad7cb4"
Sep 18 13:45:20.456713 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:20.456685    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-x95zg" event={"ID":"e3650f9e-a149-425c-9c57-ad099dedf32e","Type":"ContainerStarted","Data":"ca56ed1bfab33c4846e3567ee052e48744e0ef87bf41f446a60d787c35506f8e"}
Sep 18 13:45:20.490648 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:20.490605    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-x95zg" podStartSLOduration=2.374095032 podStartE2EDuration="6.490579788s" totalImagesPullingTime="4.116484756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:20.488167868 +0000 UTC m=+56.423133010" watchObservedRunningTime="2026-09-18 13:45:20.490579788 +0000 UTC m=+56.425544933"
Sep 18 13:45:20.510707 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:20.510671    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:20.511112 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:20.510803    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:20.511112 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:20.510808    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:20.511112 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:20.510854    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls podName:1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:21.510840895 +0000 UTC m=+57.445806021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzrnp" (UID: "1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:20.511112 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:20.510888    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:45:20.511112 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:20.510933    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm podName:1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:21.51091907 +0000 UTC m=+57.445884207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzrnp" (UID: "1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e") : configmap "kube-rbac-proxy" not found
Sep 18 13:45:21.222131 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:21.222101    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-txrvf_d34b196c-8eaa-40cf-a2e4-6b9c5a07652f/dns-node-resolver/0.log"
Sep 18 13:45:21.461660 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:21.461630    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 13:45:21.462044 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:21.462029    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/0.log"
Sep 18 13:45:21.462101 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:21.462067    2572 generic.go:356] "Generic (PLEG): container finished" podID="24a56b4d-b52e-4af5-b13e-5e9d26a998e8" containerID="8ce93ce6d8ab1b8f447fabce46a05a7d714786c4d934b24ce06b68676b43d073" exitCode=255
Sep 18 13:45:21.462173 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:21.462152    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l" event={"ID":"24a56b4d-b52e-4af5-b13e-5e9d26a998e8","Type":"ContainerDied","Data":"8ce93ce6d8ab1b8f447fabce46a05a7d714786c4d934b24ce06b68676b43d073"}
Sep 18 13:45:21.462241 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:21.462230    2572 scope.go:117] "RemoveContainer" containerID="cbe9752596e4b8fdfbafb2d645f87de697046f1b13ff158ef5661f37acad7cb4"
Sep 18 13:45:21.462464 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:21.462445    2572 scope.go:117] "RemoveContainer" containerID="8ce93ce6d8ab1b8f447fabce46a05a7d714786c4d934b24ce06b68676b43d073"
Sep 18 13:45:21.462663 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:21.462644    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-lj48l_openshift-console-operator(24a56b4d-b52e-4af5-b13e-5e9d26a998e8)\"" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l" podUID="24a56b4d-b52e-4af5-b13e-5e9d26a998e8"
Sep 18 13:45:21.518672 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:21.518604    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:21.518972 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:21.518710    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:21.518972 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:21.518726    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:45:21.518972 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:21.518786    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm podName:1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:23.518767463 +0000 UTC m=+59.453732606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzrnp" (UID: "1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e") : configmap "kube-rbac-proxy" not found
Sep 18 13:45:21.518972 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:21.518860    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:21.518972 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:21.518917    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls podName:1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:23.518901107 +0000 UTC m=+59.453866250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzrnp" (UID: "1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:22.220990 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:22.220962    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n2zrq_3b7d8ad0-cdab-41f1-81d7-de60c5b2d692/node-ca/0.log"
Sep 18 13:45:22.466172 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:22.466145    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 13:45:22.466472 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:22.466458    2572 scope.go:117] "RemoveContainer" containerID="8ce93ce6d8ab1b8f447fabce46a05a7d714786c4d934b24ce06b68676b43d073"
Sep 18 13:45:22.466639 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:22.466621    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-lj48l_openshift-console-operator(24a56b4d-b52e-4af5-b13e-5e9d26a998e8)\"" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l" podUID="24a56b4d-b52e-4af5-b13e-5e9d26a998e8"
Sep 18 13:45:22.526569 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:22.526468    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:22.526906 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:22.526644    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:45:22.526906 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:22.526672    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77d48d976-6gts2: secret "image-registry-tls" not found
Sep 18 13:45:22.526906 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:22.526726    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls podName:cb3669f1-1d97-46e9-b533-a4dbbb942be0 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:30.526712552 +0000 UTC m=+66.461677675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls") pod "image-registry-77d48d976-6gts2" (UID: "cb3669f1-1d97-46e9-b533-a4dbbb942be0") : secret "image-registry-tls" not found
Sep 18 13:45:22.627807 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:22.627772    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-58q2l\" (UID: \"0722ceea-9544-4e73-84f2-aae4fbe4ff98\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:22.627999 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:22.627980    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:22.628066 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:22.628055    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls podName:0722ceea-9544-4e73-84f2-aae4fbe4ff98 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:30.628035088 +0000 UTC m=+66.563000227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-58q2l" (UID: "0722ceea-9544-4e73-84f2-aae4fbe4ff98") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:22.728843 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:22.728812    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:22.728981 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:22.728853    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:22.729020 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:22.728985    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle podName:5b1b97a5-adb1-4a5f-89a9-785ea463af4e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:30.728963172 +0000 UTC m=+66.663928306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle") pod "router-default-77578d5fdb-w9ttn" (UID: "5b1b97a5-adb1-4a5f-89a9-785ea463af4e") : configmap references non-existent config key: service-ca.crt
Sep 18 13:45:22.729074 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:22.729019    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:45:22.729074 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:22.729062    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs podName:5b1b97a5-adb1-4a5f-89a9-785ea463af4e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:30.729050009 +0000 UTC m=+66.664015132 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs") pod "router-default-77578d5fdb-w9ttn" (UID: "5b1b97a5-adb1-4a5f-89a9-785ea463af4e") : secret "router-metrics-certs-default" not found
Sep 18 13:45:23.022368 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:23.022331    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nm5v8_7f3f5b5d-0e90-4535-b2e9-d994388cbc51/serve-healthcheck-canary/0.log"
Sep 18 13:45:23.536650 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:23.536615    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:23.537025 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:23.536760    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:23.537025 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:23.536773    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:23.537025 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:23.536816    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls podName:1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:27.536802247 +0000 UTC m=+63.471767370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzrnp" (UID: "1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:23.537025 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:23.536842    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:45:23.537025 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:23.536892    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm podName:1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:27.536880213 +0000 UTC m=+63.471845336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzrnp" (UID: "1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e") : configmap "kube-rbac-proxy" not found
Sep 18 13:45:24.982452 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:24.982421    2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:24.982869 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:24.982790    2572 scope.go:117] "RemoveContainer" containerID="8ce93ce6d8ab1b8f447fabce46a05a7d714786c4d934b24ce06b68676b43d073"
Sep 18 13:45:24.982950 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:24.982932    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-lj48l_openshift-console-operator(24a56b4d-b52e-4af5-b13e-5e9d26a998e8)\"" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l" podUID="24a56b4d-b52e-4af5-b13e-5e9d26a998e8"
Sep 18 13:45:25.378100 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:25.378030    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tg7h"
Sep 18 13:45:27.570454 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:27.570417    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:27.570908 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:27.570505    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:27.570908 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:27.570566    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:45:27.570908 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:27.570630    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:27.570908 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:27.570647    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm podName:1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:35.570626637 +0000 UTC m=+71.505591780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzrnp" (UID: "1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e") : configmap "kube-rbac-proxy" not found
Sep 18 13:45:27.570908 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:27.570695    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls podName:1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:35.570678692 +0000 UTC m=+71.505643831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzrnp" (UID: "1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:29.451720 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:29.451686    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:29.452099 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:29.452085    2572 scope.go:117] "RemoveContainer" containerID="8ce93ce6d8ab1b8f447fabce46a05a7d714786c4d934b24ce06b68676b43d073"
Sep 18 13:45:29.452252 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:29.452223    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-lj48l_openshift-console-operator(24a56b4d-b52e-4af5-b13e-5e9d26a998e8)\"" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l" podUID="24a56b4d-b52e-4af5-b13e-5e9d26a998e8"
Sep 18 13:45:30.593465 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:30.593426    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:30.595821 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:30.595800    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls\") pod \"image-registry-77d48d976-6gts2\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") " pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:30.683204 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:30.683174    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zf5xv\""
Sep 18 13:45:30.691113 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:30.691092    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:30.694592 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:30.694574    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-58q2l\" (UID: \"0722ceea-9544-4e73-84f2-aae4fbe4ff98\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:30.694726 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:30.694710    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:30.694801 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:30.694789    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls podName:0722ceea-9544-4e73-84f2-aae4fbe4ff98 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:46.694770144 +0000 UTC m=+82.629735270 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-58q2l" (UID: "0722ceea-9544-4e73-84f2-aae4fbe4ff98") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:45:30.795184 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:30.795144    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:30.795352 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:30.795210    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:30.795352 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:30.795342    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle podName:5b1b97a5-adb1-4a5f-89a9-785ea463af4e nodeName:}" failed. No retries permitted until 2026-09-18 13:45:46.795319418 +0000 UTC m=+82.730284545 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle") pod "router-default-77578d5fdb-w9ttn" (UID: "5b1b97a5-adb1-4a5f-89a9-785ea463af4e") : configmap references non-existent config key: service-ca.crt
Sep 18 13:45:30.797488 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:30.797456    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-metrics-certs\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:30.963600 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:30.963572    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77d48d976-6gts2"]
Sep 18 13:45:30.965861 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:45:30.965831    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3669f1_1d97_46e9_b533_a4dbbb942be0.slice/crio-e2bdbd1fcbd8d8a8c8a73331d14f36e4419a83181f4d9033b152430b81aeeeb1 WatchSource:0}: Error finding container e2bdbd1fcbd8d8a8c8a73331d14f36e4419a83181f4d9033b152430b81aeeeb1: Status 404 returned error can't find the container with id e2bdbd1fcbd8d8a8c8a73331d14f36e4419a83181f4d9033b152430b81aeeeb1
Sep 18 13:45:31.490521 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:31.490486    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77d48d976-6gts2" event={"ID":"cb3669f1-1d97-46e9-b533-a4dbbb942be0","Type":"ContainerStarted","Data":"488b9cbf5285cfc6a3bf809e5a081de1be7d4053a5b739424c99be4694a1bfe8"}
Sep 18 13:45:31.490669 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:31.490553    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77d48d976-6gts2" event={"ID":"cb3669f1-1d97-46e9-b533-a4dbbb942be0","Type":"ContainerStarted","Data":"e2bdbd1fcbd8d8a8c8a73331d14f36e4419a83181f4d9033b152430b81aeeeb1"}
Sep 18 13:45:31.490669 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:31.490608    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:31.513000 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:31.512954    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77d48d976-6gts2" podStartSLOduration=17.5129404 podStartE2EDuration="17.5129404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:31.511669306 +0000 UTC m=+67.446634451" watchObservedRunningTime="2026-09-18 13:45:31.5129404 +0000 UTC m=+67.447905579"
Sep 18 13:45:31.905752 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:31.905659    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret\") pod \"global-pull-secret-syncer-stclq\" (UID: \"cbb05c7c-6add-4a50-82c5-ca67901571f7\") " pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:45:31.907891 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:31.907874    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 13:45:31.918681 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:31.918660    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb05c7c-6add-4a50-82c5-ca67901571f7-original-pull-secret\") pod \"global-pull-secret-syncer-stclq\" (UID: \"cbb05c7c-6add-4a50-82c5-ca67901571f7\") " pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:45:32.006267 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:32.006235    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:32.006267 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:32.006269    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs\") pod \"network-metrics-daemon-d2v6t\" (UID: \"40156dc5-57f0-409f-a03f-2f0a80718d1d\") " pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:45:32.008430 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:32.008410    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/233eb0fd-25e7-4116-88c8-c2aad00a3fac-metrics-tls\") pod \"dns-default-qmrw7\" (UID: \"233eb0fd-25e7-4116-88c8-c2aad00a3fac\") " pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:32.008570 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:32.008415    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 13:45:32.017366 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:32.017341    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 13:45:32.017475 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:45:32.017408    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs podName:40156dc5-57f0-409f-a03f-2f0a80718d1d nodeName:}" failed. No retries permitted until 2026-09-18 13:46:36.017394086 +0000 UTC m=+131.952359208 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs") pod "network-metrics-daemon-d2v6t" (UID: "40156dc5-57f0-409f-a03f-2f0a80718d1d") : secret "metrics-daemon-secret" not found
Sep 18 13:45:32.100184 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:32.100155    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stclq"
Sep 18 13:45:32.107487 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:32.107466    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxqw\" (UniqueName: \"kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw\") pod \"network-check-target-wstlx\" (UID: \"07d3a346-ad98-4e7a-9ebc-77bfe00b06be\") " pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:45:32.109817 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:32.109800    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxqw\" (UniqueName: \"kubernetes.io/projected/07d3a346-ad98-4e7a-9ebc-77bfe00b06be-kube-api-access-dfxqw\") pod \"network-check-target-wstlx\" (UID: \"07d3a346-ad98-4e7a-9ebc-77bfe00b06be\") " pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:45:32.288487 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:32.288459    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jlrx5\""
Sep 18 13:45:32.296429 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:32.296409    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:32.308821 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:32.308799    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7x5xj\""
Sep 18 13:45:32.316995 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:32.316967    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:45:32.388765 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:32.388723    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-stclq"]
Sep 18 13:45:32.391672 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:45:32.391590    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb05c7c_6add_4a50_82c5_ca67901571f7.slice/crio-3ddc675fbfd182e79ff56df6968358157437a6c406e4fad0417db053e316c375 WatchSource:0}: Error finding container 3ddc675fbfd182e79ff56df6968358157437a6c406e4fad0417db053e316c375: Status 404 returned error can't find the container with id 3ddc675fbfd182e79ff56df6968358157437a6c406e4fad0417db053e316c375
Sep 18 13:45:32.498521 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:32.498477    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-stclq" event={"ID":"cbb05c7c-6add-4a50-82c5-ca67901571f7","Type":"ContainerStarted","Data":"3ddc675fbfd182e79ff56df6968358157437a6c406e4fad0417db053e316c375"}
Sep 18 13:45:32.581377 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:32.581351    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qmrw7"]
Sep 18 13:45:32.583971 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:45:32.583943    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233eb0fd_25e7_4116_88c8_c2aad00a3fac.slice/crio-47ddbeedc467029f6e35d4765aa85bb88d6633744606297b693a321c3842ee6c WatchSource:0}: Error finding container 47ddbeedc467029f6e35d4765aa85bb88d6633744606297b693a321c3842ee6c: Status 404 returned error can't find the container with id 47ddbeedc467029f6e35d4765aa85bb88d6633744606297b693a321c3842ee6c
Sep 18 13:45:32.601336 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:32.601286    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wstlx"]
Sep 18 13:45:32.603685 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:45:32.603642    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d3a346_ad98_4e7a_9ebc_77bfe00b06be.slice/crio-bad254d8d1c9fcca7a7a54a0bd5dfa448ed9c508c4ca9502ffdc899b4f0a5501 WatchSource:0}: Error finding container bad254d8d1c9fcca7a7a54a0bd5dfa448ed9c508c4ca9502ffdc899b4f0a5501: Status 404 returned error can't find the container with id bad254d8d1c9fcca7a7a54a0bd5dfa448ed9c508c4ca9502ffdc899b4f0a5501
Sep 18 13:45:33.505248 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:33.505211    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wstlx" event={"ID":"07d3a346-ad98-4e7a-9ebc-77bfe00b06be","Type":"ContainerStarted","Data":"020cf359ffb790091df78e4295e4ae9eb8617b0a5b6c48505130c5bd103ae0b0"}
Sep 18 13:45:33.506000 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:33.505274    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wstlx" event={"ID":"07d3a346-ad98-4e7a-9ebc-77bfe00b06be","Type":"ContainerStarted","Data":"bad254d8d1c9fcca7a7a54a0bd5dfa448ed9c508c4ca9502ffdc899b4f0a5501"}
Sep 18 13:45:33.506000 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:33.505492    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:45:33.507272 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:33.507245    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qmrw7" event={"ID":"233eb0fd-25e7-4116-88c8-c2aad00a3fac","Type":"ContainerStarted","Data":"47ddbeedc467029f6e35d4765aa85bb88d6633744606297b693a321c3842ee6c"}
Sep 18 13:45:33.522459 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:33.522417    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wstlx" podStartSLOduration=67.52240234 podStartE2EDuration="1m7.52240234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:33.521151825 +0000 UTC m=+69.456116973" watchObservedRunningTime="2026-09-18 13:45:33.52240234 +0000 UTC m=+69.457367488"
Sep 18 13:45:35.520585 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:35.520552    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qmrw7" event={"ID":"233eb0fd-25e7-4116-88c8-c2aad00a3fac","Type":"ContainerStarted","Data":"8e9b9d89d915574fc8273abc4b7a6f0f8561d4af3863f332e99be62c6a7b716a"}
Sep 18 13:45:35.521041 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:35.520614    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qmrw7" event={"ID":"233eb0fd-25e7-4116-88c8-c2aad00a3fac","Type":"ContainerStarted","Data":"f5a8427743ae46ef302b0b940a4b64f19044e9dfc9d7442bd2363be2df22c2af"}
Sep 18 13:45:35.521041 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:35.520685    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:35.538714 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:35.538668    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qmrw7" podStartSLOduration=33.429505162 podStartE2EDuration="35.538654965s" totalImagesPullingTime="2.109149803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:35.537558973 +0000 UTC m=+71.472524119" watchObservedRunningTime="2026-09-18 13:45:35.538654965 +0000 UTC m=+71.473620109"
Sep 18 13:45:35.639734 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:35.639695    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:35.639996 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:35.639956    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:35.640323 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:35.640298    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:35.642500 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:35.642480    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzrnp\" (UID: \"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e\") " pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:35.932218 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:35.932146    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8k78s\""
Sep 18 13:45:35.939880 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:35.939852    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bzrnp"
Sep 18 13:45:36.525976 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:36.525939    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-stclq" event={"ID":"cbb05c7c-6add-4a50-82c5-ca67901571f7","Type":"ContainerStarted","Data":"0706ba2345f30c8bcf93123fac6fa42d0bb0da185e587ecbcfff3f86eeac94b4"}
Sep 18 13:45:36.543006 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:36.542949    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-stclq" podStartSLOduration=66.560169386 podStartE2EDuration="1m10.542930462s" totalImagesPullingTime="3.982761076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:36.542785886 +0000 UTC m=+72.477751032" watchObservedRunningTime="2026-09-18 13:45:36.542930462 +0000 UTC m=+72.477895608"
Sep 18 13:45:36.614267 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:36.614099    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bzrnp"]
Sep 18 13:45:36.616753 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:45:36.616726    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc1f02a_2e4b_43e7_8a7b_a3721a9be63e.slice/crio-91830b8f8d1b4b46247947f4526e8d63c09a12af39dd879b196de24fe825771e WatchSource:0}: Error finding container 91830b8f8d1b4b46247947f4526e8d63c09a12af39dd879b196de24fe825771e: Status 404 returned error can't find the container with id 91830b8f8d1b4b46247947f4526e8d63c09a12af39dd879b196de24fe825771e
Sep 18 13:45:37.531432 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:37.531400    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzrnp" event={"ID":"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e","Type":"ContainerStarted","Data":"a22eb9db52cbd7457d707e5636c945151bd9a35846f432a8f5299f39b33521db"}
Sep 18 13:45:37.531778 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:37.531459    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzrnp" event={"ID":"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e","Type":"ContainerStarted","Data":"4761942790cbb5e6f255fa75b5545e39c5b4f6ed8d5954d0696fa97c207b8ca8"}
Sep 18 13:45:37.531778 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:37.531481    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzrnp" event={"ID":"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e","Type":"ContainerStarted","Data":"91830b8f8d1b4b46247947f4526e8d63c09a12af39dd879b196de24fe825771e"}
Sep 18 13:45:39.540598 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:39.540565    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzrnp" event={"ID":"1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e","Type":"ContainerStarted","Data":"2684fb6797feb1505046b3fb06c12c11e9d86e1e14d847a7d13a4272feebdcc8"}
Sep 18 13:45:39.561449 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:39.561408    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bzrnp" podStartSLOduration=18.653129308 podStartE2EDuration="20.561395126s" totalImagesPullingTime="1.908265818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:45:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:39.560613661 +0000 UTC m=+75.495578807" watchObservedRunningTime="2026-09-18 13:45:39.561395126 +0000 UTC m=+75.496360271"
Sep 18 13:45:44.191697 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.191663    2572 scope.go:117] "RemoveContainer" containerID="8ce93ce6d8ab1b8f447fabce46a05a7d714786c4d934b24ce06b68676b43d073"
Sep 18 13:45:44.502431 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.502347    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf"]
Sep 18 13:45:44.505493 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.505474    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf"
Sep 18 13:45:44.512762 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.512743    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 13:45:44.514894 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.514876    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 13:45:44.514983 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.514879    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 13:45:44.515637 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.515623    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 13:45:44.523989 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.523969    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf"]
Sep 18 13:45:44.549810 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.549783    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77d48d976-6gts2"]
Sep 18 13:45:44.558657 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.558641    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 13:45:44.558748 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.558711    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l" event={"ID":"24a56b4d-b52e-4af5-b13e-5e9d26a998e8","Type":"ContainerStarted","Data":"4ce4308962fd2cc546eef6e57b506bc7fc80bca9c2612d48b1499a83fdf174a0"}
Sep 18 13:45:44.558997 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.558977    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:44.601925 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.601899    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-ddf994ddf-lbkh4"]
Sep 18 13:45:44.603686 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.603658    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/71eba09e-c791-4c6d-96d7-456e5ea34bbf-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f4d67cdd9-p27vf\" (UID: \"71eba09e-c791-4c6d-96d7-456e5ea34bbf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf"
Sep 18 13:45:44.603792 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.603705    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp898\" (UniqueName: \"kubernetes.io/projected/71eba09e-c791-4c6d-96d7-456e5ea34bbf-kube-api-access-hp898\") pod \"klusterlet-addon-workmgr-6f4d67cdd9-p27vf\" (UID: \"71eba09e-c791-4c6d-96d7-456e5ea34bbf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf"
Sep 18 13:45:44.603850 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.603810    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71eba09e-c791-4c6d-96d7-456e5ea34bbf-tmp\") pod \"klusterlet-addon-workmgr-6f4d67cdd9-p27vf\" (UID: \"71eba09e-c791-4c6d-96d7-456e5ea34bbf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf"
Sep 18 13:45:44.605099 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.605083    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.618208 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.618186    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-ddf994ddf-lbkh4"]
Sep 18 13:45:44.704141 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.704117    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp898\" (UniqueName: \"kubernetes.io/projected/71eba09e-c791-4c6d-96d7-456e5ea34bbf-kube-api-access-hp898\") pod \"klusterlet-addon-workmgr-6f4d67cdd9-p27vf\" (UID: \"71eba09e-c791-4c6d-96d7-456e5ea34bbf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf"
Sep 18 13:45:44.704242 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.704166    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4357fd99-f1f1-4070-8b74-59f84c4a483a-registry-certificates\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.704318 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.704298    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71eba09e-c791-4c6d-96d7-456e5ea34bbf-tmp\") pod \"klusterlet-addon-workmgr-6f4d67cdd9-p27vf\" (UID: \"71eba09e-c791-4c6d-96d7-456e5ea34bbf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf"
Sep 18 13:45:44.704361 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.704338    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4357fd99-f1f1-4070-8b74-59f84c4a483a-trusted-ca\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.704398 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.704367    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45mns\" (UniqueName: \"kubernetes.io/projected/4357fd99-f1f1-4070-8b74-59f84c4a483a-kube-api-access-45mns\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.704438 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.704399    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4357fd99-f1f1-4070-8b74-59f84c4a483a-image-registry-private-configuration\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.704438 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.704416    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4357fd99-f1f1-4070-8b74-59f84c4a483a-ca-trust-extracted\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.704518 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.704439    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4357fd99-f1f1-4070-8b74-59f84c4a483a-installation-pull-secrets\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.704518 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.704463    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/71eba09e-c791-4c6d-96d7-456e5ea34bbf-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f4d67cdd9-p27vf\" (UID: \"71eba09e-c791-4c6d-96d7-456e5ea34bbf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf"
Sep 18 13:45:44.704518 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.704486    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4357fd99-f1f1-4070-8b74-59f84c4a483a-registry-tls\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.704650 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.704635    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4357fd99-f1f1-4070-8b74-59f84c4a483a-bound-sa-token\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.704695 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.704671    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71eba09e-c791-4c6d-96d7-456e5ea34bbf-tmp\") pod \"klusterlet-addon-workmgr-6f4d67cdd9-p27vf\" (UID: \"71eba09e-c791-4c6d-96d7-456e5ea34bbf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf"
Sep 18 13:45:44.707031 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.707010    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/71eba09e-c791-4c6d-96d7-456e5ea34bbf-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f4d67cdd9-p27vf\" (UID: \"71eba09e-c791-4c6d-96d7-456e5ea34bbf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf"
Sep 18 13:45:44.722624 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.722601    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp898\" (UniqueName: \"kubernetes.io/projected/71eba09e-c791-4c6d-96d7-456e5ea34bbf-kube-api-access-hp898\") pod \"klusterlet-addon-workmgr-6f4d67cdd9-p27vf\" (UID: \"71eba09e-c791-4c6d-96d7-456e5ea34bbf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf"
Sep 18 13:45:44.790863 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.790797    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-lj48l"
Sep 18 13:45:44.805276 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.805250    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4357fd99-f1f1-4070-8b74-59f84c4a483a-registry-certificates\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.805388 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.805301    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4357fd99-f1f1-4070-8b74-59f84c4a483a-trusted-ca\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.805443 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.805418    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45mns\" (UniqueName: \"kubernetes.io/projected/4357fd99-f1f1-4070-8b74-59f84c4a483a-kube-api-access-45mns\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.805488 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.805454    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4357fd99-f1f1-4070-8b74-59f84c4a483a-image-registry-private-configuration\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.805488 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.805478    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4357fd99-f1f1-4070-8b74-59f84c4a483a-ca-trust-extracted\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.805593 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.805513    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4357fd99-f1f1-4070-8b74-59f84c4a483a-installation-pull-secrets\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.805593 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.805563    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4357fd99-f1f1-4070-8b74-59f84c4a483a-registry-tls\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.805685 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.805600    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4357fd99-f1f1-4070-8b74-59f84c4a483a-bound-sa-token\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.805932 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.805906    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4357fd99-f1f1-4070-8b74-59f84c4a483a-ca-trust-extracted\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.806154 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.806135    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4357fd99-f1f1-4070-8b74-59f84c4a483a-registry-certificates\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.806433 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.806411    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4357fd99-f1f1-4070-8b74-59f84c4a483a-trusted-ca\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.808649 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.808386    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4357fd99-f1f1-4070-8b74-59f84c4a483a-registry-tls\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.808649 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.808600    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4357fd99-f1f1-4070-8b74-59f84c4a483a-installation-pull-secrets\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.808799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.808685    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4357fd99-f1f1-4070-8b74-59f84c4a483a-image-registry-private-configuration\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.814238 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.814212    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf"
Sep 18 13:45:44.818988 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.818968    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4357fd99-f1f1-4070-8b74-59f84c4a483a-bound-sa-token\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.819058 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.819022    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45mns\" (UniqueName: \"kubernetes.io/projected/4357fd99-f1f1-4070-8b74-59f84c4a483a-kube-api-access-45mns\") pod \"image-registry-ddf994ddf-lbkh4\" (UID: \"4357fd99-f1f1-4070-8b74-59f84c4a483a\") " pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.913779 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.913752    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:44.991813 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.991786    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-vtlp8"]
Sep 18 13:45:44.995121 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.995101    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-vtlp8"
Sep 18 13:45:44.997886 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.997833    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 13:45:44.998734 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:44.998708    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-f649n\""
Sep 18 13:45:45.000391 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:45.000372    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 13:45:45.015986 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:45.015963    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-vtlp8"]
Sep 18 13:45:45.108471 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:45.108406    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k2n7\" (UniqueName: \"kubernetes.io/projected/be82ea7b-c939-4ee3-af33-ec978fe19ba4-kube-api-access-7k2n7\") pod \"downloads-6594f4c75-vtlp8\" (UID: \"be82ea7b-c939-4ee3-af33-ec978fe19ba4\") " pod="openshift-console/downloads-6594f4c75-vtlp8"
Sep 18 13:45:45.122415 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:45.122393    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf"]
Sep 18 13:45:45.124841 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:45:45.124820    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71eba09e_c791_4c6d_96d7_456e5ea34bbf.slice/crio-d580adb9a33eaf316426a27bd70c2bed3afd7bcfc3705319dc0c50ad1ed54f16 WatchSource:0}: Error finding container d580adb9a33eaf316426a27bd70c2bed3afd7bcfc3705319dc0c50ad1ed54f16: Status 404 returned error can't find the container with id d580adb9a33eaf316426a27bd70c2bed3afd7bcfc3705319dc0c50ad1ed54f16
Sep 18 13:45:45.199504 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:45.199481    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-ddf994ddf-lbkh4"]
Sep 18 13:45:45.201251 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:45:45.201233    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4357fd99_f1f1_4070_8b74_59f84c4a483a.slice/crio-b41e16e279c23f05a09f461f5ba47f6cb8069245beac247d9fb55d7f4ceebeb2 WatchSource:0}: Error finding container b41e16e279c23f05a09f461f5ba47f6cb8069245beac247d9fb55d7f4ceebeb2: Status 404 returned error can't find the container with id b41e16e279c23f05a09f461f5ba47f6cb8069245beac247d9fb55d7f4ceebeb2
Sep 18 13:45:45.209150 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:45.209129    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k2n7\" (UniqueName: \"kubernetes.io/projected/be82ea7b-c939-4ee3-af33-ec978fe19ba4-kube-api-access-7k2n7\") pod \"downloads-6594f4c75-vtlp8\" (UID: \"be82ea7b-c939-4ee3-af33-ec978fe19ba4\") " pod="openshift-console/downloads-6594f4c75-vtlp8"
Sep 18 13:45:45.219748 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:45.219725    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k2n7\" (UniqueName: \"kubernetes.io/projected/be82ea7b-c939-4ee3-af33-ec978fe19ba4-kube-api-access-7k2n7\") pod \"downloads-6594f4c75-vtlp8\" (UID: \"be82ea7b-c939-4ee3-af33-ec978fe19ba4\") " pod="openshift-console/downloads-6594f4c75-vtlp8"
Sep 18 13:45:45.303972 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:45.303945    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-vtlp8"
Sep 18 13:45:45.528117 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:45.528088    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qmrw7"
Sep 18 13:45:45.564465 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:45.564430    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4" event={"ID":"4357fd99-f1f1-4070-8b74-59f84c4a483a","Type":"ContainerStarted","Data":"614cd03f21b7547a5e14fe7a8b2823cbae141ff63d813a5aa99e2835ba683d07"}
Sep 18 13:45:45.564643 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:45.564488    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4" event={"ID":"4357fd99-f1f1-4070-8b74-59f84c4a483a","Type":"ContainerStarted","Data":"b41e16e279c23f05a09f461f5ba47f6cb8069245beac247d9fb55d7f4ceebeb2"}
Sep 18 13:45:45.564643 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:45.564518    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:45:45.565817 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:45.565790    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf" event={"ID":"71eba09e-c791-4c6d-96d7-456e5ea34bbf","Type":"ContainerStarted","Data":"d580adb9a33eaf316426a27bd70c2bed3afd7bcfc3705319dc0c50ad1ed54f16"}
Sep 18 13:45:45.577183 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:45.577164    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-vtlp8"]
Sep 18 13:45:45.579771 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:45:45.579735    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe82ea7b_c939_4ee3_af33_ec978fe19ba4.slice/crio-882370f164601428eee6e6a8a2e120a875e547cdc1aad350d73720a5417556de WatchSource:0}: Error finding container 882370f164601428eee6e6a8a2e120a875e547cdc1aad350d73720a5417556de: Status 404 returned error can't find the container with id 882370f164601428eee6e6a8a2e120a875e547cdc1aad350d73720a5417556de
Sep 18 13:45:45.598966 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:45.598923    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4" podStartSLOduration=1.598908007 podStartE2EDuration="1.598908007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:45.597772301 +0000 UTC m=+81.532737447" watchObservedRunningTime="2026-09-18 13:45:45.598908007 +0000 UTC m=+81.533873151"
Sep 18 13:45:46.573664 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:46.573595    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-vtlp8" event={"ID":"be82ea7b-c939-4ee3-af33-ec978fe19ba4","Type":"ContainerStarted","Data":"882370f164601428eee6e6a8a2e120a875e547cdc1aad350d73720a5417556de"}
Sep 18 13:45:46.726969 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:46.726856    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-58q2l\" (UID: \"0722ceea-9544-4e73-84f2-aae4fbe4ff98\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:46.729976 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:46.729922    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0722ceea-9544-4e73-84f2-aae4fbe4ff98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-58q2l\" (UID: \"0722ceea-9544-4e73-84f2-aae4fbe4ff98\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:46.827834 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:46.827749    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:46.828527 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:46.828460    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1b97a5-adb1-4a5f-89a9-785ea463af4e-service-ca-bundle\") pod \"router-default-77578d5fdb-w9ttn\" (UID: \"5b1b97a5-adb1-4a5f-89a9-785ea463af4e\") " pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:46.896778 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:46.896736    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wtcms\""
Sep 18 13:45:46.904454 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:46.904249    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"
Sep 18 13:45:47.077298 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:47.077096    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-njvq6\""
Sep 18 13:45:47.084743 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:47.084612    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:47.608490 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:47.608428    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77578d5fdb-w9ttn"]
Sep 18 13:45:47.611702 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:47.611675    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l"]
Sep 18 13:45:47.612987 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:45:47.612949    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1b97a5_adb1_4a5f_89a9_785ea463af4e.slice/crio-2433ba3e281220e720257b14faf0dd3d036e7bff17bb7f801eddbef74a170a59 WatchSource:0}: Error finding container 2433ba3e281220e720257b14faf0dd3d036e7bff17bb7f801eddbef74a170a59: Status 404 returned error can't find the container with id 2433ba3e281220e720257b14faf0dd3d036e7bff17bb7f801eddbef74a170a59
Sep 18 13:45:47.615853 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:45:47.615828    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0722ceea_9544_4e73_84f2_aae4fbe4ff98.slice/crio-30dac422e16757a28792fcf2ccd1b5c4b04ddbb3b2714781174f8ef158a24331 WatchSource:0}: Error finding container 30dac422e16757a28792fcf2ccd1b5c4b04ddbb3b2714781174f8ef158a24331: Status 404 returned error can't find the container with id 30dac422e16757a28792fcf2ccd1b5c4b04ddbb3b2714781174f8ef158a24331
Sep 18 13:45:48.583781 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:48.583744    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77578d5fdb-w9ttn" event={"ID":"5b1b97a5-adb1-4a5f-89a9-785ea463af4e","Type":"ContainerStarted","Data":"29acbc7d63e1cc4e2cf21d02441594d4a177e1bf8ddb576295c7cd257231c50b"}
Sep 18 13:45:48.583955 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:48.583811    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77578d5fdb-w9ttn" event={"ID":"5b1b97a5-adb1-4a5f-89a9-785ea463af4e","Type":"ContainerStarted","Data":"2433ba3e281220e720257b14faf0dd3d036e7bff17bb7f801eddbef74a170a59"}
Sep 18 13:45:48.585197 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:48.585167    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l" event={"ID":"0722ceea-9544-4e73-84f2-aae4fbe4ff98","Type":"ContainerStarted","Data":"30dac422e16757a28792fcf2ccd1b5c4b04ddbb3b2714781174f8ef158a24331"}
Sep 18 13:45:48.606894 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:48.606799    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-77578d5fdb-w9ttn" podStartSLOduration=34.606779928 podStartE2EDuration="34.606779928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:48.603982854 +0000 UTC m=+84.538947999" watchObservedRunningTime="2026-09-18 13:45:48.606779928 +0000 UTC m=+84.541745068"
Sep 18 13:45:49.085453 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:49.085416    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:49.088267 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:49.088245    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:49.592365 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:49.592048    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf" event={"ID":"71eba09e-c791-4c6d-96d7-456e5ea34bbf","Type":"ContainerStarted","Data":"1c6d4da63eac644a7eb14a143878c04b3baed61ea4014a85fa86aafdaeb967fd"}
Sep 18 13:45:49.592365 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:49.592262    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:49.592365 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:49.592353    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf"
Sep 18 13:45:49.594178 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:49.594139    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-77578d5fdb-w9ttn"
Sep 18 13:45:49.594429 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:49.594405    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf"
Sep 18 13:45:49.617182 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:49.617134    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d67cdd9-p27vf" podStartSLOduration=1.6206327109999998 podStartE2EDuration="5.617118656s" totalImagesPullingTime="3.996485945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:49.616423398 +0000 UTC m=+85.551388545" watchObservedRunningTime="2026-09-18 13:45:49.617118656 +0000 UTC m=+85.552083800"
Sep 18 13:45:51.602270 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:51.602231    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l" event={"ID":"0722ceea-9544-4e73-84f2-aae4fbe4ff98","Type":"ContainerStarted","Data":"5250cf4ddaf791ea4657cd9ff65f430a52138b9b83724707b09aa6cd6c7974b5"}
Sep 18 13:45:51.623847 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:51.623786    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-58q2l" podStartSLOduration=34.667386976 podStartE2EDuration="37.623766059s" totalImagesPullingTime="2.956379083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:51.62180592 +0000 UTC m=+87.556771065" watchObservedRunningTime="2026-09-18 13:45:51.623766059 +0000 UTC m=+87.558731205"
Sep 18 13:45:54.555882 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:54.555852    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:45:57.706101 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.706064    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n67xl"]
Sep 18 13:45:57.710667 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.710644    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.714252 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.714219    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 13:45:57.714729 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.714693    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 13:45:57.715188 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.715171    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 13:45:57.717356 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.717290    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h4fd4\""
Sep 18 13:45:57.717626 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.717600    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 13:45:57.828485 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.828446    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-root\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.828685 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.828503    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.828685 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.828601    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-sys\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.828685 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.828662    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-metrics-client-ca\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.828837 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.828725    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.828837 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.828769    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-node-exporter-textfile\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.828837 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.828797    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-node-exporter-wtmp\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.828978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.828859    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4wgr\" (UniqueName: \"kubernetes.io/projected/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-kube-api-access-n4wgr\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.828978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.828905    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-node-exporter-tls\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.929688 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.929654    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-metrics-client-ca\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.929860 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.929697    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.929860 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.929741    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-node-exporter-textfile\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.929860 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.929767    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-node-exporter-wtmp\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.929860 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.929826    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4wgr\" (UniqueName: \"kubernetes.io/projected/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-kube-api-access-n4wgr\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.929860 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.929857    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-node-exporter-tls\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.930124 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.929910    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-root\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.930124 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.929942    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.930124 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.929979    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-sys\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.930124 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.930083    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-sys\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.930429 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.930399    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-metrics-client-ca\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.930870 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.930631    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.930870 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.930712    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-root\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.930870 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.930842    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-node-exporter-wtmp\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.931076 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.930971    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-node-exporter-textfile\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.933329 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.933309    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-node-exporter-tls\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.933439 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.933375    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:57.941193 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:57.941169    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4wgr\" (UniqueName: \"kubernetes.io/projected/9843c12f-6b4e-4cf1-b7c4-c042f83b05b6-kube-api-access-n4wgr\") pod \"node-exporter-n67xl\" (UID: \"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6\") " pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:45:58.022287 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:45:58.022212    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n67xl"
Sep 18 13:46:02.193714 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:46:02.193682    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9843c12f_6b4e_4cf1_b7c4_c042f83b05b6.slice/crio-6ab32997dfc94c9f17165ab3502bea79e6b876bd0b62d08ece128a2e7dc4ffb8 WatchSource:0}: Error finding container 6ab32997dfc94c9f17165ab3502bea79e6b876bd0b62d08ece128a2e7dc4ffb8: Status 404 returned error can't find the container with id 6ab32997dfc94c9f17165ab3502bea79e6b876bd0b62d08ece128a2e7dc4ffb8
Sep 18 13:46:02.648662 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:02.648569    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-vtlp8" event={"ID":"be82ea7b-c939-4ee3-af33-ec978fe19ba4","Type":"ContainerStarted","Data":"18ea7cdf36433ffac10b3f078749e988003b3452f70ee64baab306b8e165be9a"}
Sep 18 13:46:02.649215 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:02.649109    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-vtlp8"
Sep 18 13:46:02.650383 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:02.650326    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n67xl" event={"ID":"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6","Type":"ContainerStarted","Data":"6ab32997dfc94c9f17165ab3502bea79e6b876bd0b62d08ece128a2e7dc4ffb8"}
Sep 18 13:46:02.667643 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:02.667616    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-vtlp8"
Sep 18 13:46:02.672398 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:02.671802    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-vtlp8" podStartSLOduration=2.000788815 podStartE2EDuration="18.671781826s" totalImagesPullingTime="16.670993011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:46:02.671135159 +0000 UTC m=+98.606100307" watchObservedRunningTime="2026-09-18 13:46:02.671781826 +0000 UTC m=+98.606746973"
Sep 18 13:46:03.656983 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:03.656942    2572 generic.go:356] "Generic (PLEG): container finished" podID="9843c12f-6b4e-4cf1-b7c4-c042f83b05b6" containerID="7c7511629fe58691e2ef964a56b61b61ae9d0b8bc6cf3becf88428ed161e743f" exitCode=0
Sep 18 13:46:03.657551 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:03.657039    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n67xl" event={"ID":"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6","Type":"ContainerDied","Data":"7c7511629fe58691e2ef964a56b61b61ae9d0b8bc6cf3becf88428ed161e743f"}
Sep 18 13:46:04.516626 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:04.516596    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wstlx"
Sep 18 13:46:04.664356 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:04.664318    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n67xl" event={"ID":"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6","Type":"ContainerStarted","Data":"9d8fb89407d185fa4318e5edfa14323731a5f13c44070ed971c652bb21bc1b0d"}
Sep 18 13:46:04.664808 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:04.664384    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n67xl" event={"ID":"9843c12f-6b4e-4cf1-b7c4-c042f83b05b6","Type":"ContainerStarted","Data":"9b90f36978f75c3cbfa76e6b39c77c336833c3eaa711a436b7e5a093a8c984fb"}
Sep 18 13:46:04.684398 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:04.684344    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n67xl" podStartSLOduration=6.675223488 podStartE2EDuration="7.68432688s" totalImagesPullingTime="938.006916ms" totalInitContainerRuntime="71.096476ms" isStatefulPod=true podCreationTimestamp="2026-09-18 13:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:46:04.682801989 +0000 UTC m=+100.617767136" watchObservedRunningTime="2026-09-18 13:46:04.68432688 +0000 UTC m=+100.619292025"
Sep 18 13:46:05.500886 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.500849    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6494ff59dc-qhhqq"]
Sep 18 13:46:05.527217 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.527189    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6494ff59dc-qhhqq"]
Sep 18 13:46:05.527390 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.527306    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.530043 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.530011    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 13:46:05.530043 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.530029    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r6kn9\""
Sep 18 13:46:05.530833 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.530810    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 13:46:05.530958 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.530870    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 13:46:05.530958 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.530925    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 13:46:05.531093 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.530937    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 13:46:05.538266 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.538243    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 13:46:05.706907 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.706873    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-service-ca\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.707428 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.706960    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-oauth-config\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.707428 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.707006    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-oauth-serving-cert\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.707428 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.707049    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-serving-cert\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.707428 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.707075    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-config\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.707428 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.707096    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn44v\" (UniqueName: \"kubernetes.io/projected/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-kube-api-access-qn44v\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.707428 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.707120    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-trusted-ca-bundle\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.808461 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.808382    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-serving-cert\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.808461 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.808424    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-config\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.808461 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.808451    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn44v\" (UniqueName: \"kubernetes.io/projected/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-kube-api-access-qn44v\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.808779 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.808491    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-trusted-ca-bundle\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.808779 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.808574    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-service-ca\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.808779 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.808666    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-oauth-config\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.808779 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.808707    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-oauth-serving-cert\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.809251 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.809227    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-config\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.809491 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.809468    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-service-ca\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.809883 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.809855    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-oauth-serving-cert\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.810570 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.810524    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-trusted-ca-bundle\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.811906 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.811883    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-oauth-config\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.812193 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.812171    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-serving-cert\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.821721 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.821698    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn44v\" (UniqueName: \"kubernetes.io/projected/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-kube-api-access-qn44v\") pod \"console-6494ff59dc-qhhqq\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") " pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:05.839465 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:05.839443    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:06.145670 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:06.145425    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6494ff59dc-qhhqq"]
Sep 18 13:46:06.148593 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:46:06.148566    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8884a18_7e03_45b1_b3fb_2e2be2a170c2.slice/crio-c388709e328e5d85865865e501a929e600dfd59b2222e123862fc45d30ca64cb WatchSource:0}: Error finding container c388709e328e5d85865865e501a929e600dfd59b2222e123862fc45d30ca64cb: Status 404 returned error can't find the container with id c388709e328e5d85865865e501a929e600dfd59b2222e123862fc45d30ca64cb
Sep 18 13:46:06.578788 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:06.578758    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-ddf994ddf-lbkh4"
Sep 18 13:46:06.678076 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:06.678011    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6494ff59dc-qhhqq" event={"ID":"d8884a18-7e03-45b1-b3fb-2e2be2a170c2","Type":"ContainerStarted","Data":"c388709e328e5d85865865e501a929e600dfd59b2222e123862fc45d30ca64cb"}
Sep 18 13:46:09.569327 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:09.569231    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-77d48d976-6gts2" podUID="cb3669f1-1d97-46e9-b533-a4dbbb942be0" containerName="registry" containerID="cri-o://488b9cbf5285cfc6a3bf809e5a081de1be7d4053a5b739424c99be4694a1bfe8" gracePeriod=30
Sep 18 13:46:10.700128 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:10.700092    2572 generic.go:356] "Generic (PLEG): container finished" podID="cb3669f1-1d97-46e9-b533-a4dbbb942be0" containerID="488b9cbf5285cfc6a3bf809e5a081de1be7d4053a5b739424c99be4694a1bfe8" exitCode=0
Sep 18 13:46:10.700657 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:10.700148    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77d48d976-6gts2" event={"ID":"cb3669f1-1d97-46e9-b533-a4dbbb942be0","Type":"ContainerDied","Data":"488b9cbf5285cfc6a3bf809e5a081de1be7d4053a5b739424c99be4694a1bfe8"}
Sep 18 13:46:11.511629 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.511602    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:46:11.564246 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.564219    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6d2p\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-kube-api-access-z6d2p\") pod \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") "
Sep 18 13:46:11.564375 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.564257    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb3669f1-1d97-46e9-b533-a4dbbb942be0-installation-pull-secrets\") pod \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") "
Sep 18 13:46:11.564375 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.564276    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb3669f1-1d97-46e9-b533-a4dbbb942be0-trusted-ca\") pod \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") "
Sep 18 13:46:11.564375 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.564292    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-bound-sa-token\") pod \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") "
Sep 18 13:46:11.564375 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.564318    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb3669f1-1d97-46e9-b533-a4dbbb942be0-image-registry-private-configuration\") pod \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") "
Sep 18 13:46:11.564375 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.564350    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-certificates\") pod \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") "
Sep 18 13:46:11.564666 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.564394    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb3669f1-1d97-46e9-b533-a4dbbb942be0-ca-trust-extracted\") pod \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") "
Sep 18 13:46:11.564666 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.564426    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls\") pod \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\" (UID: \"cb3669f1-1d97-46e9-b533-a4dbbb942be0\") "
Sep 18 13:46:11.564841 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.564810    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3669f1-1d97-46e9-b533-a4dbbb942be0-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cb3669f1-1d97-46e9-b533-a4dbbb942be0" (UID: "cb3669f1-1d97-46e9-b533-a4dbbb942be0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:46:11.564918 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.564816    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cb3669f1-1d97-46e9-b533-a4dbbb942be0" (UID: "cb3669f1-1d97-46e9-b533-a4dbbb942be0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:46:11.567204 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.567177    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3669f1-1d97-46e9-b533-a4dbbb942be0-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cb3669f1-1d97-46e9-b533-a4dbbb942be0" (UID: "cb3669f1-1d97-46e9-b533-a4dbbb942be0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:46:11.567320 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.567262    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3669f1-1d97-46e9-b533-a4dbbb942be0-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cb3669f1-1d97-46e9-b533-a4dbbb942be0" (UID: "cb3669f1-1d97-46e9-b533-a4dbbb942be0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:46:11.567320 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.567292    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cb3669f1-1d97-46e9-b533-a4dbbb942be0" (UID: "cb3669f1-1d97-46e9-b533-a4dbbb942be0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:46:11.567433 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.567351    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cb3669f1-1d97-46e9-b533-a4dbbb942be0" (UID: "cb3669f1-1d97-46e9-b533-a4dbbb942be0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:46:11.567483 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.567442    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-kube-api-access-z6d2p" (OuterVolumeSpecName: "kube-api-access-z6d2p") pod "cb3669f1-1d97-46e9-b533-a4dbbb942be0" (UID: "cb3669f1-1d97-46e9-b533-a4dbbb942be0"). InnerVolumeSpecName "kube-api-access-z6d2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:46:11.573353 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.573331    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3669f1-1d97-46e9-b533-a4dbbb942be0-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cb3669f1-1d97-46e9-b533-a4dbbb942be0" (UID: "cb3669f1-1d97-46e9-b533-a4dbbb942be0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:46:11.665113 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.665091    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6d2p\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-kube-api-access-z6d2p\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:46:11.665113 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.665114    2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb3669f1-1d97-46e9-b533-a4dbbb942be0-installation-pull-secrets\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:46:11.665242 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.665124    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb3669f1-1d97-46e9-b533-a4dbbb942be0-trusted-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:46:11.665242 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.665133    2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-bound-sa-token\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:46:11.665242 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.665141    2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb3669f1-1d97-46e9-b533-a4dbbb942be0-image-registry-private-configuration\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:46:11.665242 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.665151    2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-certificates\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:46:11.665242 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.665159    2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb3669f1-1d97-46e9-b533-a4dbbb942be0-ca-trust-extracted\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:46:11.665242 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.665167    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb3669f1-1d97-46e9-b533-a4dbbb942be0-registry-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:46:11.705180 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.705165    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77d48d976-6gts2"
Sep 18 13:46:11.705180 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.705169    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77d48d976-6gts2" event={"ID":"cb3669f1-1d97-46e9-b533-a4dbbb942be0","Type":"ContainerDied","Data":"e2bdbd1fcbd8d8a8c8a73331d14f36e4419a83181f4d9033b152430b81aeeeb1"}
Sep 18 13:46:11.705568 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.705223    2572 scope.go:117] "RemoveContainer" containerID="488b9cbf5285cfc6a3bf809e5a081de1be7d4053a5b739424c99be4694a1bfe8"
Sep 18 13:46:11.724837 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.724813    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77d48d976-6gts2"]
Sep 18 13:46:11.728685 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:11.728666    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-77d48d976-6gts2"]
Sep 18 13:46:12.195364 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:12.195327    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb3669f1-1d97-46e9-b533-a4dbbb942be0" path="/var/lib/kubelet/pods/cb3669f1-1d97-46e9-b533-a4dbbb942be0/volumes"
Sep 18 13:46:12.710501 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:12.710460    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6494ff59dc-qhhqq" event={"ID":"d8884a18-7e03-45b1-b3fb-2e2be2a170c2","Type":"ContainerStarted","Data":"ccd6ecd0a3198114672ec92feb814bbac99acbad4646586f886b34a437e4f67d"}
Sep 18 13:46:12.727972 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:12.727927    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6494ff59dc-qhhqq" podStartSLOduration=2.001289035 podStartE2EDuration="7.727910591s" totalImagesPullingTime="5.726621556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:46:12.726807915 +0000 UTC m=+108.661773060" watchObservedRunningTime="2026-09-18 13:46:12.727910591 +0000 UTC m=+108.662875735"
Sep 18 13:46:15.840133 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:15.840094    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:15.840608 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:15.840193    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:15.844989 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:15.844969    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:16.729240 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:16.729214    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:46:30.247793 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:30.247756    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77578d5fdb-w9ttn_5b1b97a5-adb1-4a5f-89a9-785ea463af4e/router/0.log"
Sep 18 13:46:30.266041 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:30.266017    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nm5v8_7f3f5b5d-0e90-4535-b2e9-d994388cbc51/serve-healthcheck-canary/0.log"
Sep 18 13:46:34.785023 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:34.784995    2572 generic.go:356] "Generic (PLEG): container finished" podID="2cb3e59e-5c9c-486d-89dd-99474a7edade" containerID="08faeddb5f9fab25a52724f7744a6ba8467af6a44060ba9991f524394f678b8b" exitCode=0
Sep 18 13:46:34.785349 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:34.785050    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz" event={"ID":"2cb3e59e-5c9c-486d-89dd-99474a7edade","Type":"ContainerDied","Data":"08faeddb5f9fab25a52724f7744a6ba8467af6a44060ba9991f524394f678b8b"}
Sep 18 13:46:34.785349 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:34.785330    2572 scope.go:117] "RemoveContainer" containerID="08faeddb5f9fab25a52724f7744a6ba8467af6a44060ba9991f524394f678b8b"
Sep 18 13:46:35.790548 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:35.790506    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4wcgz" event={"ID":"2cb3e59e-5c9c-486d-89dd-99474a7edade","Type":"ContainerStarted","Data":"c435d211b746c0264b9eea91031b9776dcef13db16ec375f8b318599a0bd07ce"}
Sep 18 13:46:36.061200 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:36.061125    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs\") pod \"network-metrics-daemon-d2v6t\" (UID: \"40156dc5-57f0-409f-a03f-2f0a80718d1d\") " pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:46:36.063230 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:36.063210    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40156dc5-57f0-409f-a03f-2f0a80718d1d-metrics-certs\") pod \"network-metrics-daemon-d2v6t\" (UID: \"40156dc5-57f0-409f-a03f-2f0a80718d1d\") " pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:46:36.213604 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:36.213581    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cttxw\""
Sep 18 13:46:36.221046 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:36.221026    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2v6t"
Sep 18 13:46:36.536583 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:36.536527    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d2v6t"]
Sep 18 13:46:36.539228 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:46:36.539203    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40156dc5_57f0_409f_a03f_2f0a80718d1d.slice/crio-6151dfd4ff7e404fb7330f2fddd3de38e14dcb1fa52822ccdcb4b0c817575c1b WatchSource:0}: Error finding container 6151dfd4ff7e404fb7330f2fddd3de38e14dcb1fa52822ccdcb4b0c817575c1b: Status 404 returned error can't find the container with id 6151dfd4ff7e404fb7330f2fddd3de38e14dcb1fa52822ccdcb4b0c817575c1b
Sep 18 13:46:36.795172 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:36.795139    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d2v6t" event={"ID":"40156dc5-57f0-409f-a03f-2f0a80718d1d","Type":"ContainerStarted","Data":"6151dfd4ff7e404fb7330f2fddd3de38e14dcb1fa52822ccdcb4b0c817575c1b"}
Sep 18 13:46:38.803508 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:38.803476    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d2v6t" event={"ID":"40156dc5-57f0-409f-a03f-2f0a80718d1d","Type":"ContainerStarted","Data":"c0efa41ce36db382359cbf882d235bf2bde521b76183b6a8935b00931124d2eb"}
Sep 18 13:46:38.803987 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:38.803528    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d2v6t" event={"ID":"40156dc5-57f0-409f-a03f-2f0a80718d1d","Type":"ContainerStarted","Data":"4fe46c205e0def8812a986885f2b34aaf9ad40075460b6c0d9b08900caa98532"}
Sep 18 13:46:38.820768 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:46:38.820719    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d2v6t" podStartSLOduration=131.41629718 podStartE2EDuration="2m12.820706183s" totalImagesPullingTime="1.404409003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:46:38.819581207 +0000 UTC m=+134.754546352" watchObservedRunningTime="2026-09-18 13:46:38.820706183 +0000 UTC m=+134.755671328"
Sep 18 13:47:22.116709 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.116633    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7dbcfb5c5-np254"]
Sep 18 13:47:22.117143 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.116886    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb3669f1-1d97-46e9-b533-a4dbbb942be0" containerName="registry"
Sep 18 13:47:22.117143 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.116896    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb3669f1-1d97-46e9-b533-a4dbbb942be0" containerName="registry"
Sep 18 13:47:22.117143 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.116947    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb3669f1-1d97-46e9-b533-a4dbbb942be0" containerName="registry"
Sep 18 13:47:22.119087 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.119067    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.121777 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.121745    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 13:47:22.122499 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.121917    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-d7c2h\""
Sep 18 13:47:22.122499 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.122143    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 13:47:22.122499 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.122200    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 13:47:22.122499 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.122146    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 13:47:22.122499 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.122360    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 13:47:22.128244 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.128216    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 13:47:22.132761 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.132738    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7dbcfb5c5-np254"]
Sep 18 13:47:22.211780 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.211754    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stgnc\" (UniqueName: \"kubernetes.io/projected/0bf1f531-64a6-4adc-92e0-dca725db2aa8-kube-api-access-stgnc\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.211887 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.211795    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0bf1f531-64a6-4adc-92e0-dca725db2aa8-federate-client-tls\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.211887 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.211820    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0bf1f531-64a6-4adc-92e0-dca725db2aa8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.211887 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.211877    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bf1f531-64a6-4adc-92e0-dca725db2aa8-serving-certs-ca-bundle\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.211985 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.211907    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0bf1f531-64a6-4adc-92e0-dca725db2aa8-metrics-client-ca\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.211985 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.211929    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0bf1f531-64a6-4adc-92e0-dca725db2aa8-telemeter-client-tls\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.211985 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.211966    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0bf1f531-64a6-4adc-92e0-dca725db2aa8-secret-telemeter-client\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.212080 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.211994    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bf1f531-64a6-4adc-92e0-dca725db2aa8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.313046 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.313012    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bf1f531-64a6-4adc-92e0-dca725db2aa8-serving-certs-ca-bundle\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.313185 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.313055    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0bf1f531-64a6-4adc-92e0-dca725db2aa8-metrics-client-ca\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.313185 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.313176    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0bf1f531-64a6-4adc-92e0-dca725db2aa8-telemeter-client-tls\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.313286 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.313208    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0bf1f531-64a6-4adc-92e0-dca725db2aa8-secret-telemeter-client\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.313286 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.313233    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bf1f531-64a6-4adc-92e0-dca725db2aa8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.313387 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.313305    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stgnc\" (UniqueName: \"kubernetes.io/projected/0bf1f531-64a6-4adc-92e0-dca725db2aa8-kube-api-access-stgnc\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.313387 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.313349    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0bf1f531-64a6-4adc-92e0-dca725db2aa8-federate-client-tls\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.313495 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.313391    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0bf1f531-64a6-4adc-92e0-dca725db2aa8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.313970 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.313948    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bf1f531-64a6-4adc-92e0-dca725db2aa8-serving-certs-ca-bundle\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.314282 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.314260    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0bf1f531-64a6-4adc-92e0-dca725db2aa8-metrics-client-ca\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.314755 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.314728    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bf1f531-64a6-4adc-92e0-dca725db2aa8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.315976 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.315949    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0bf1f531-64a6-4adc-92e0-dca725db2aa8-federate-client-tls\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.316137 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.316115    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0bf1f531-64a6-4adc-92e0-dca725db2aa8-telemeter-client-tls\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.316285 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.316264    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0bf1f531-64a6-4adc-92e0-dca725db2aa8-secret-telemeter-client\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.316350 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.316229    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0bf1f531-64a6-4adc-92e0-dca725db2aa8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.321734 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.321713    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stgnc\" (UniqueName: \"kubernetes.io/projected/0bf1f531-64a6-4adc-92e0-dca725db2aa8-kube-api-access-stgnc\") pod \"telemeter-client-7dbcfb5c5-np254\" (UID: \"0bf1f531-64a6-4adc-92e0-dca725db2aa8\") " pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.431959 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.431884    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254"
Sep 18 13:47:22.704337 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.704298    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7dbcfb5c5-np254"]
Sep 18 13:47:22.706387 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:47:22.706360    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf1f531_64a6_4adc_92e0_dca725db2aa8.slice/crio-bb78a5ccc65bd29be8308755219bdcbd497e9ada32f53589cb675f10d9f32286 WatchSource:0}: Error finding container bb78a5ccc65bd29be8308755219bdcbd497e9ada32f53589cb675f10d9f32286: Status 404 returned error can't find the container with id bb78a5ccc65bd29be8308755219bdcbd497e9ada32f53589cb675f10d9f32286
Sep 18 13:47:22.961292 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:22.961207    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254" event={"ID":"0bf1f531-64a6-4adc-92e0-dca725db2aa8","Type":"ContainerStarted","Data":"bb78a5ccc65bd29be8308755219bdcbd497e9ada32f53589cb675f10d9f32286"}
Sep 18 13:47:24.970919 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:24.970880    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254" event={"ID":"0bf1f531-64a6-4adc-92e0-dca725db2aa8","Type":"ContainerStarted","Data":"3f17314cb05dae03f6ad501c88cda294a8870166b9f735760988975de618f381"}
Sep 18 13:47:25.976745 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:25.976707    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254" event={"ID":"0bf1f531-64a6-4adc-92e0-dca725db2aa8","Type":"ContainerStarted","Data":"79c82107cde06b0841ce55bdb115a50b10d75361af0b758db3fb45295f913068"}
Sep 18 13:47:25.977119 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:25.976775    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254" event={"ID":"0bf1f531-64a6-4adc-92e0-dca725db2aa8","Type":"ContainerStarted","Data":"c4a557354e97506eba41af7590e5568961e467380b402f382d5ba0cc81a146d9"}
Sep 18 13:47:25.999974 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:25.999923    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7dbcfb5c5-np254" podStartSLOduration=1.493941464 podStartE2EDuration="3.999903791s" totalImagesPullingTime="2.505962327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:47:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:47:25.999589828 +0000 UTC m=+181.934554973" watchObservedRunningTime="2026-09-18 13:47:25.999903791 +0000 UTC m=+181.934868961"
Sep 18 13:47:26.090763 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.090732    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66f5568799-qskng"]
Sep 18 13:47:26.092803 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.092784    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.105062 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.105035    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66f5568799-qskng"]
Sep 18 13:47:26.141160 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.141115    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-oauth-config\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.141160 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.141158    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-config\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.141371 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.141219    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-trusted-ca-bundle\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.141371 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.141256    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-serving-cert\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.141371 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.141283    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh6fs\" (UniqueName: \"kubernetes.io/projected/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-kube-api-access-mh6fs\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.141371 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.141311    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-service-ca\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.141371 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.141339    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-oauth-serving-cert\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.242434 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.242371    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-serving-cert\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.242434 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.242401    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh6fs\" (UniqueName: \"kubernetes.io/projected/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-kube-api-access-mh6fs\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.242434 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.242419    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-service-ca\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.242693 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.242439    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-oauth-serving-cert\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.242693 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.242570    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-oauth-config\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.242693 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.242607    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-config\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.242693 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.242689    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-trusted-ca-bundle\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.243227 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.243203    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-oauth-serving-cert\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.243322 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.243213    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-service-ca\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.243388 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.243378    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-trusted-ca-bundle\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.243455 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.243386    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-config\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.244853 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.244825    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-serving-cert\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.244943 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.244895    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-oauth-config\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.251265 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.251245    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh6fs\" (UniqueName: \"kubernetes.io/projected/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-kube-api-access-mh6fs\") pod \"console-66f5568799-qskng\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") " pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.403445 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.403397    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:26.677516 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.677473    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66f5568799-qskng"]
Sep 18 13:47:26.680026 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:47:26.680002    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d871d43_9e41_4fc4_a2b0_0a0c0f71bb13.slice/crio-fdaa9493e3e94c3d4dd469e40393c8d446089afe1e4f3990d1606c2f40c6c00a WatchSource:0}: Error finding container fdaa9493e3e94c3d4dd469e40393c8d446089afe1e4f3990d1606c2f40c6c00a: Status 404 returned error can't find the container with id fdaa9493e3e94c3d4dd469e40393c8d446089afe1e4f3990d1606c2f40c6c00a
Sep 18 13:47:26.982687 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.982639    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f5568799-qskng" event={"ID":"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13","Type":"ContainerStarted","Data":"c40ad2568261a6f131f23c2c52a0f4ece587b6f2db765f423228f52fe2019c8a"}
Sep 18 13:47:26.983164 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:26.982703    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f5568799-qskng" event={"ID":"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13","Type":"ContainerStarted","Data":"fdaa9493e3e94c3d4dd469e40393c8d446089afe1e4f3990d1606c2f40c6c00a"}
Sep 18 13:47:27.000761 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:27.000689    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66f5568799-qskng" podStartSLOduration=1.000671946 podStartE2EDuration="1.000671946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:47:27.000443979 +0000 UTC m=+182.935409141" watchObservedRunningTime="2026-09-18 13:47:27.000671946 +0000 UTC m=+182.935637130"
Sep 18 13:47:36.404548 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:36.404490    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:36.404990 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:36.404589    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:36.409273 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:36.409248    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:37.021705 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:37.021674    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:47:37.070750 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:47:37.070718    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6494ff59dc-qhhqq"]
Sep 18 13:48:02.093184 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.093123    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6494ff59dc-qhhqq" podUID="d8884a18-7e03-45b1-b3fb-2e2be2a170c2" containerName="console" containerID="cri-o://ccd6ecd0a3198114672ec92feb814bbac99acbad4646586f886b34a437e4f67d" gracePeriod=15
Sep 18 13:48:02.337066 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.337043    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:48:02.431341 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.431261    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-oauth-config\") pod \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") "
Sep 18 13:48:02.431341 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.431293    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn44v\" (UniqueName: \"kubernetes.io/projected/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-kube-api-access-qn44v\") pod \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") "
Sep 18 13:48:02.431341 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.431316    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-oauth-serving-cert\") pod \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") "
Sep 18 13:48:02.431341 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.431333    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-config\") pod \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") "
Sep 18 13:48:02.431676 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.431352    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-serving-cert\") pod \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") "
Sep 18 13:48:02.431676 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.431372    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-service-ca\") pod \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") "
Sep 18 13:48:02.431676 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.431404    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-trusted-ca-bundle\") pod \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\" (UID: \"d8884a18-7e03-45b1-b3fb-2e2be2a170c2\") "
Sep 18 13:48:02.431831 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.431747    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-config" (OuterVolumeSpecName: "console-config") pod "d8884a18-7e03-45b1-b3fb-2e2be2a170c2" (UID: "d8884a18-7e03-45b1-b3fb-2e2be2a170c2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:48:02.432116 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.431948    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d8884a18-7e03-45b1-b3fb-2e2be2a170c2" (UID: "d8884a18-7e03-45b1-b3fb-2e2be2a170c2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:48:02.432116 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.432076    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-service-ca" (OuterVolumeSpecName: "service-ca") pod "d8884a18-7e03-45b1-b3fb-2e2be2a170c2" (UID: "d8884a18-7e03-45b1-b3fb-2e2be2a170c2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:48:02.432116 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.432049    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d8884a18-7e03-45b1-b3fb-2e2be2a170c2" (UID: "d8884a18-7e03-45b1-b3fb-2e2be2a170c2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:48:02.433663 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.433641    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d8884a18-7e03-45b1-b3fb-2e2be2a170c2" (UID: "d8884a18-7e03-45b1-b3fb-2e2be2a170c2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:48:02.433753 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.433659    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d8884a18-7e03-45b1-b3fb-2e2be2a170c2" (UID: "d8884a18-7e03-45b1-b3fb-2e2be2a170c2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:48:02.433753 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.433734    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-kube-api-access-qn44v" (OuterVolumeSpecName: "kube-api-access-qn44v") pod "d8884a18-7e03-45b1-b3fb-2e2be2a170c2" (UID: "d8884a18-7e03-45b1-b3fb-2e2be2a170c2"). InnerVolumeSpecName "kube-api-access-qn44v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:48:02.532368 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.532348    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-trusted-ca-bundle\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:48:02.532368 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.532368    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-oauth-config\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:48:02.532498 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.532378    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn44v\" (UniqueName: \"kubernetes.io/projected/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-kube-api-access-qn44v\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:48:02.532498 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.532387    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-oauth-serving-cert\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:48:02.532498 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.532396    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-config\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:48:02.532498 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.532405    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-console-serving-cert\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:48:02.532498 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:02.532413    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8884a18-7e03-45b1-b3fb-2e2be2a170c2-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:48:03.113225 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:03.113190    2572 generic.go:356] "Generic (PLEG): container finished" podID="d8884a18-7e03-45b1-b3fb-2e2be2a170c2" containerID="ccd6ecd0a3198114672ec92feb814bbac99acbad4646586f886b34a437e4f67d" exitCode=0
Sep 18 13:48:03.113671 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:03.113232    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6494ff59dc-qhhqq"
Sep 18 13:48:03.113671 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:03.113270    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6494ff59dc-qhhqq" event={"ID":"d8884a18-7e03-45b1-b3fb-2e2be2a170c2","Type":"ContainerDied","Data":"ccd6ecd0a3198114672ec92feb814bbac99acbad4646586f886b34a437e4f67d"}
Sep 18 13:48:03.113671 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:03.113317    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6494ff59dc-qhhqq" event={"ID":"d8884a18-7e03-45b1-b3fb-2e2be2a170c2","Type":"ContainerDied","Data":"c388709e328e5d85865865e501a929e600dfd59b2222e123862fc45d30ca64cb"}
Sep 18 13:48:03.113671 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:03.113337    2572 scope.go:117] "RemoveContainer" containerID="ccd6ecd0a3198114672ec92feb814bbac99acbad4646586f886b34a437e4f67d"
Sep 18 13:48:03.121978 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:03.121963    2572 scope.go:117] "RemoveContainer" containerID="ccd6ecd0a3198114672ec92feb814bbac99acbad4646586f886b34a437e4f67d"
Sep 18 13:48:03.122241 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:48:03.122220    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd6ecd0a3198114672ec92feb814bbac99acbad4646586f886b34a437e4f67d\": container with ID starting with ccd6ecd0a3198114672ec92feb814bbac99acbad4646586f886b34a437e4f67d not found: ID does not exist" containerID="ccd6ecd0a3198114672ec92feb814bbac99acbad4646586f886b34a437e4f67d"
Sep 18 13:48:03.122311 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:03.122255    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd6ecd0a3198114672ec92feb814bbac99acbad4646586f886b34a437e4f67d"} err="failed to get container status \"ccd6ecd0a3198114672ec92feb814bbac99acbad4646586f886b34a437e4f67d\": rpc error: code = NotFound desc = could not find container \"ccd6ecd0a3198114672ec92feb814bbac99acbad4646586f886b34a437e4f67d\": container with ID starting with ccd6ecd0a3198114672ec92feb814bbac99acbad4646586f886b34a437e4f67d not found: ID does not exist"
Sep 18 13:48:03.132792 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:03.132769    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6494ff59dc-qhhqq"]
Sep 18 13:48:03.136827 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:03.136806    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6494ff59dc-qhhqq"]
Sep 18 13:48:04.195371 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:04.195338    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8884a18-7e03-45b1-b3fb-2e2be2a170c2" path="/var/lib/kubelet/pods/d8884a18-7e03-45b1-b3fb-2e2be2a170c2/volumes"
Sep 18 13:48:45.033342 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.033272    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-db48ddbff-2c7w4"]
Sep 18 13:48:45.033804 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.033551    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8884a18-7e03-45b1-b3fb-2e2be2a170c2" containerName="console"
Sep 18 13:48:45.033804 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.033563    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8884a18-7e03-45b1-b3fb-2e2be2a170c2" containerName="console"
Sep 18 13:48:45.033804 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.033629    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8884a18-7e03-45b1-b3fb-2e2be2a170c2" containerName="console"
Sep 18 13:48:45.036368 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.036353    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.047972 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.047953    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-db48ddbff-2c7w4"]
Sep 18 13:48:45.130872 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.130848    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-serving-cert\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.130972 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.130899    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-config\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.130972 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.130958    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-oauth-config\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.131046 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.130990    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-service-ca\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.131046 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.131006    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrdsg\" (UniqueName: \"kubernetes.io/projected/ec86a21e-2355-4be8-a6bc-4aad15bbf654-kube-api-access-qrdsg\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.131108 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.131051    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-trusted-ca-bundle\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.131108 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.131068    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-oauth-serving-cert\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.231926 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.231900    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-serving-cert\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.232017 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.231930    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-config\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.232017 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.231959    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-oauth-config\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.232017 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.231977    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-service-ca\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.232017 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.231999    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrdsg\" (UniqueName: \"kubernetes.io/projected/ec86a21e-2355-4be8-a6bc-4aad15bbf654-kube-api-access-qrdsg\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.232175 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.232045    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-trusted-ca-bundle\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.232175 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.232071    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-oauth-serving-cert\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.232723 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.232697    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-config\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.232882 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.232767    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-oauth-serving-cert\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.232882 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.232871    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-trusted-ca-bundle\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.232882 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.232791    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-service-ca\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.234507 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.234488    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-oauth-config\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.234604 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.234580    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-serving-cert\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.240717 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.240699    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrdsg\" (UniqueName: \"kubernetes.io/projected/ec86a21e-2355-4be8-a6bc-4aad15bbf654-kube-api-access-qrdsg\") pod \"console-db48ddbff-2c7w4\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") " pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.345139 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.345071    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:45.627272 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:45.627244    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-db48ddbff-2c7w4"]
Sep 18 13:48:45.628815 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:48:45.628789    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec86a21e_2355_4be8_a6bc_4aad15bbf654.slice/crio-9c243659bf34ded5488ddf948f7faf2cff724a61b14cb77f9c47e8d4589da048 WatchSource:0}: Error finding container 9c243659bf34ded5488ddf948f7faf2cff724a61b14cb77f9c47e8d4589da048: Status 404 returned error can't find the container with id 9c243659bf34ded5488ddf948f7faf2cff724a61b14cb77f9c47e8d4589da048
Sep 18 13:48:46.267774 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:46.267743    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db48ddbff-2c7w4" event={"ID":"ec86a21e-2355-4be8-a6bc-4aad15bbf654","Type":"ContainerStarted","Data":"07a85f71775e36873e5225f202107c1a383df9c8c59e579eb6f3edecedb75249"}
Sep 18 13:48:46.268139 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:46.267792    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db48ddbff-2c7w4" event={"ID":"ec86a21e-2355-4be8-a6bc-4aad15bbf654","Type":"ContainerStarted","Data":"9c243659bf34ded5488ddf948f7faf2cff724a61b14cb77f9c47e8d4589da048"}
Sep 18 13:48:46.288923 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:46.288874    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-db48ddbff-2c7w4" podStartSLOduration=1.288860973 podStartE2EDuration="1.288860973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:48:46.287965314 +0000 UTC m=+262.222930460" watchObservedRunningTime="2026-09-18 13:48:46.288860973 +0000 UTC m=+262.223826117"
Sep 18 13:48:55.345682 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:55.345649    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:55.346130 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:55.345704    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:55.350509 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:55.350488    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:56.311830 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:56.311794    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:48:56.361922 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:48:56.361884    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66f5568799-qskng"]
Sep 18 13:49:21.386290 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.386245    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66f5568799-qskng" podUID="7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13" containerName="console" containerID="cri-o://c40ad2568261a6f131f23c2c52a0f4ece587b6f2db765f423228f52fe2019c8a" gracePeriod=15
Sep 18 13:49:21.620057 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.620035    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:49:21.713099 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.713061    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-config\") pod \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") "
Sep 18 13:49:21.713099 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.713116    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-service-ca\") pod \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") "
Sep 18 13:49:21.713314 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.713135    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-oauth-serving-cert\") pod \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") "
Sep 18 13:49:21.713314 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.713166    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-serving-cert\") pod \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") "
Sep 18 13:49:21.713314 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.713187    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh6fs\" (UniqueName: \"kubernetes.io/projected/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-kube-api-access-mh6fs\") pod \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") "
Sep 18 13:49:21.713314 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.713214    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-oauth-config\") pod \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") "
Sep 18 13:49:21.713314 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.713227    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-trusted-ca-bundle\") pod \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\" (UID: \"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13\") "
Sep 18 13:49:21.713602 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.713438    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-config" (OuterVolumeSpecName: "console-config") pod "7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13" (UID: "7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:49:21.713602 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.713515    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-service-ca" (OuterVolumeSpecName: "service-ca") pod "7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13" (UID: "7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:49:21.713716 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.713685    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13" (UID: "7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:49:21.713781 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.713751    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13" (UID: "7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:49:21.715381 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.715358    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13" (UID: "7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:49:21.715460 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.715380    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13" (UID: "7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:49:21.715495 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.715472    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-kube-api-access-mh6fs" (OuterVolumeSpecName: "kube-api-access-mh6fs") pod "7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13" (UID: "7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13"). InnerVolumeSpecName "kube-api-access-mh6fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:49:21.814374 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.814345    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-oauth-config\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:49:21.814374 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.814369    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-trusted-ca-bundle\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:49:21.814522 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.814383    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-config\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:49:21.814522 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.814395    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:49:21.814522 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.814407    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-oauth-serving-cert\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:49:21.814522 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.814418    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-console-serving-cert\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:49:21.814522 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:21.814448    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh6fs\" (UniqueName: \"kubernetes.io/projected/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13-kube-api-access-mh6fs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:49:22.401879 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:22.401845    2572 generic.go:356] "Generic (PLEG): container finished" podID="7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13" containerID="c40ad2568261a6f131f23c2c52a0f4ece587b6f2db765f423228f52fe2019c8a" exitCode=0
Sep 18 13:49:22.402257 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:22.401891    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f5568799-qskng"
Sep 18 13:49:22.402257 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:22.401930    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f5568799-qskng" event={"ID":"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13","Type":"ContainerDied","Data":"c40ad2568261a6f131f23c2c52a0f4ece587b6f2db765f423228f52fe2019c8a"}
Sep 18 13:49:22.402257 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:22.401983    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f5568799-qskng" event={"ID":"7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13","Type":"ContainerDied","Data":"fdaa9493e3e94c3d4dd469e40393c8d446089afe1e4f3990d1606c2f40c6c00a"}
Sep 18 13:49:22.402257 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:22.402003    2572 scope.go:117] "RemoveContainer" containerID="c40ad2568261a6f131f23c2c52a0f4ece587b6f2db765f423228f52fe2019c8a"
Sep 18 13:49:22.409790 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:22.409769    2572 scope.go:117] "RemoveContainer" containerID="c40ad2568261a6f131f23c2c52a0f4ece587b6f2db765f423228f52fe2019c8a"
Sep 18 13:49:22.410056 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:49:22.410038    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c40ad2568261a6f131f23c2c52a0f4ece587b6f2db765f423228f52fe2019c8a\": container with ID starting with c40ad2568261a6f131f23c2c52a0f4ece587b6f2db765f423228f52fe2019c8a not found: ID does not exist" containerID="c40ad2568261a6f131f23c2c52a0f4ece587b6f2db765f423228f52fe2019c8a"
Sep 18 13:49:22.410122 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:22.410067    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c40ad2568261a6f131f23c2c52a0f4ece587b6f2db765f423228f52fe2019c8a"} err="failed to get container status \"c40ad2568261a6f131f23c2c52a0f4ece587b6f2db765f423228f52fe2019c8a\": rpc error: code = NotFound desc = could not find container \"c40ad2568261a6f131f23c2c52a0f4ece587b6f2db765f423228f52fe2019c8a\": container with ID starting with c40ad2568261a6f131f23c2c52a0f4ece587b6f2db765f423228f52fe2019c8a not found: ID does not exist"
Sep 18 13:49:22.438899 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:22.438874    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66f5568799-qskng"]
Sep 18 13:49:22.442443 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:22.442422    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66f5568799-qskng"]
Sep 18 13:49:24.195315 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:24.195281    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13" path="/var/lib/kubelet/pods/7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13/volumes"
Sep 18 13:49:26.124953 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:26.124921    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 13:49:26.125425 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:26.125085    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 13:49:26.135134 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:26.135109    2572 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 13:49:27.711118 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.711087    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq"]
Sep 18 13:49:27.713561 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.711481    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13" containerName="console"
Sep 18 13:49:27.713561 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.711498    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13" containerName="console"
Sep 18 13:49:27.713561 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.711590    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d871d43-9e41-4fc4-a2b0-0a0c0f71bb13" containerName="console"
Sep 18 13:49:27.714424 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.714406    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq"
Sep 18 13:49:27.716673 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.716649    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 13:49:27.717251 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.717232    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4m6sb\""
Sep 18 13:49:27.717313 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.717275    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 13:49:27.723766 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.723744    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq"]
Sep 18 13:49:27.861813 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.861770    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq\" (UID: \"5db2ccd0-df6f-4e24-bff6-a0e6c396d119\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq"
Sep 18 13:49:27.861998 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.861845    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq\" (UID: \"5db2ccd0-df6f-4e24-bff6-a0e6c396d119\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq"
Sep 18 13:49:27.861998 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.861884    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgwrh\" (UniqueName: \"kubernetes.io/projected/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-kube-api-access-wgwrh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq\" (UID: \"5db2ccd0-df6f-4e24-bff6-a0e6c396d119\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq"
Sep 18 13:49:27.962786 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.962691    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgwrh\" (UniqueName: \"kubernetes.io/projected/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-kube-api-access-wgwrh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq\" (UID: \"5db2ccd0-df6f-4e24-bff6-a0e6c396d119\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq"
Sep 18 13:49:27.962935 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.962859    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq\" (UID: \"5db2ccd0-df6f-4e24-bff6-a0e6c396d119\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq"
Sep 18 13:49:27.962935 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.962903    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq\" (UID: \"5db2ccd0-df6f-4e24-bff6-a0e6c396d119\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq"
Sep 18 13:49:27.963221 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.963204    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq\" (UID: \"5db2ccd0-df6f-4e24-bff6-a0e6c396d119\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq"
Sep 18 13:49:27.963273 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.963228    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq\" (UID: \"5db2ccd0-df6f-4e24-bff6-a0e6c396d119\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq"
Sep 18 13:49:27.972003 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:27.971985    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgwrh\" (UniqueName: \"kubernetes.io/projected/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-kube-api-access-wgwrh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq\" (UID: \"5db2ccd0-df6f-4e24-bff6-a0e6c396d119\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq"
Sep 18 13:49:28.023800 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:28.023776    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq"
Sep 18 13:49:28.293412 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:28.293390    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq"]
Sep 18 13:49:28.295419 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:49:28.295389    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db2ccd0_df6f_4e24_bff6_a0e6c396d119.slice/crio-f350288dfc1ed936207594c80cf66bdd82948a155c7ac5268d4827401b77b05e WatchSource:0}: Error finding container f350288dfc1ed936207594c80cf66bdd82948a155c7ac5268d4827401b77b05e: Status 404 returned error can't find the container with id f350288dfc1ed936207594c80cf66bdd82948a155c7ac5268d4827401b77b05e
Sep 18 13:49:28.297414 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:28.297395    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:49:28.428019 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:28.427985    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq" event={"ID":"5db2ccd0-df6f-4e24-bff6-a0e6c396d119","Type":"ContainerStarted","Data":"f350288dfc1ed936207594c80cf66bdd82948a155c7ac5268d4827401b77b05e"}
Sep 18 13:49:34.453992 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:34.453949    2572 generic.go:356] "Generic (PLEG): container finished" podID="5db2ccd0-df6f-4e24-bff6-a0e6c396d119" containerID="b8fb64dd4fd1ad8da9904b79491623f0f22d4c2549c9f24cd15c143055f37cce" exitCode=0
Sep 18 13:49:34.454431 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:34.454031    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq" event={"ID":"5db2ccd0-df6f-4e24-bff6-a0e6c396d119","Type":"ContainerDied","Data":"b8fb64dd4fd1ad8da9904b79491623f0f22d4c2549c9f24cd15c143055f37cce"}
Sep 18 13:49:37.467940 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:37.467909    2572 generic.go:356] "Generic (PLEG): container finished" podID="5db2ccd0-df6f-4e24-bff6-a0e6c396d119" containerID="47519a7b49672db1583abfd1d2bbacd64a50bb1eddd53513117051550a91ca0a" exitCode=0
Sep 18 13:49:37.468340 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:37.467999    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq" event={"ID":"5db2ccd0-df6f-4e24-bff6-a0e6c396d119","Type":"ContainerDied","Data":"47519a7b49672db1583abfd1d2bbacd64a50bb1eddd53513117051550a91ca0a"}
Sep 18 13:49:44.501078 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:44.501044    2572 generic.go:356] "Generic (PLEG): container finished" podID="5db2ccd0-df6f-4e24-bff6-a0e6c396d119" containerID="2cbfa627e2ee387e87a65a112c6370a8844ba19733774c5eea1191851d85e6a0" exitCode=0
Sep 18 13:49:44.501459 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:44.501083    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq" event={"ID":"5db2ccd0-df6f-4e24-bff6-a0e6c396d119","Type":"ContainerDied","Data":"2cbfa627e2ee387e87a65a112c6370a8844ba19733774c5eea1191851d85e6a0"}
Sep 18 13:49:45.631266 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:45.631242    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq"
Sep 18 13:49:45.713507 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:45.713469    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgwrh\" (UniqueName: \"kubernetes.io/projected/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-kube-api-access-wgwrh\") pod \"5db2ccd0-df6f-4e24-bff6-a0e6c396d119\" (UID: \"5db2ccd0-df6f-4e24-bff6-a0e6c396d119\") "
Sep 18 13:49:45.713664 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:45.713555    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-bundle\") pod \"5db2ccd0-df6f-4e24-bff6-a0e6c396d119\" (UID: \"5db2ccd0-df6f-4e24-bff6-a0e6c396d119\") "
Sep 18 13:49:45.713664 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:45.713592    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-util\") pod \"5db2ccd0-df6f-4e24-bff6-a0e6c396d119\" (UID: \"5db2ccd0-df6f-4e24-bff6-a0e6c396d119\") "
Sep 18 13:49:45.714220 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:45.714191    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-bundle" (OuterVolumeSpecName: "bundle") pod "5db2ccd0-df6f-4e24-bff6-a0e6c396d119" (UID: "5db2ccd0-df6f-4e24-bff6-a0e6c396d119"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:49:45.715674 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:45.715652    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-kube-api-access-wgwrh" (OuterVolumeSpecName: "kube-api-access-wgwrh") pod "5db2ccd0-df6f-4e24-bff6-a0e6c396d119" (UID: "5db2ccd0-df6f-4e24-bff6-a0e6c396d119"). InnerVolumeSpecName "kube-api-access-wgwrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:49:45.719478 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:45.719452    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-util" (OuterVolumeSpecName: "util") pod "5db2ccd0-df6f-4e24-bff6-a0e6c396d119" (UID: "5db2ccd0-df6f-4e24-bff6-a0e6c396d119"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:49:45.814191 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:45.814109    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgwrh\" (UniqueName: \"kubernetes.io/projected/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-kube-api-access-wgwrh\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:49:45.814191 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:45.814139    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-bundle\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:49:45.814191 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:45.814152    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5db2ccd0-df6f-4e24-bff6-a0e6c396d119-util\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:49:46.509336 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:46.509243    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq" event={"ID":"5db2ccd0-df6f-4e24-bff6-a0e6c396d119","Type":"ContainerDied","Data":"f350288dfc1ed936207594c80cf66bdd82948a155c7ac5268d4827401b77b05e"}
Sep 18 13:49:46.509336 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:46.509284    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lb2zq"
Sep 18 13:49:46.509336 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:46.509300    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f350288dfc1ed936207594c80cf66bdd82948a155c7ac5268d4827401b77b05e"
Sep 18 13:49:50.837457 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:50.837423    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qfmnn"]
Sep 18 13:49:50.837918 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:50.837734    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db2ccd0-df6f-4e24-bff6-a0e6c396d119" containerName="util"
Sep 18 13:49:50.837918 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:50.837746    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db2ccd0-df6f-4e24-bff6-a0e6c396d119" containerName="util"
Sep 18 13:49:50.837918 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:50.837758    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db2ccd0-df6f-4e24-bff6-a0e6c396d119" containerName="pull"
Sep 18 13:49:50.837918 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:50.837764    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db2ccd0-df6f-4e24-bff6-a0e6c396d119" containerName="pull"
Sep 18 13:49:50.837918 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:50.837775    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db2ccd0-df6f-4e24-bff6-a0e6c396d119" containerName="extract"
Sep 18 13:49:50.837918 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:50.837780    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db2ccd0-df6f-4e24-bff6-a0e6c396d119" containerName="extract"
Sep 18 13:49:50.837918 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:50.837826    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db2ccd0-df6f-4e24-bff6-a0e6c396d119" containerName="extract"
Sep 18 13:49:50.869153 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:50.869127    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qfmnn"]
Sep 18 13:49:50.869298 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:50.869217    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qfmnn"
Sep 18 13:49:50.876975 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:50.876950    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 18 13:49:50.877722 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:50.877700    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:49:50.877924 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:50.877905    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-r79cq\""
Sep 18 13:49:50.951019 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:50.950985    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ca36e2e-36cd-4997-b057-e5149afcd2d0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qfmnn\" (UID: \"7ca36e2e-36cd-4997-b057-e5149afcd2d0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qfmnn"
Sep 18 13:49:50.951176 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:50.951077    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2s6g\" (UniqueName: \"kubernetes.io/projected/7ca36e2e-36cd-4997-b057-e5149afcd2d0-kube-api-access-j2s6g\") pod \"cert-manager-operator-controller-manager-55c444bf86-qfmnn\" (UID: \"7ca36e2e-36cd-4997-b057-e5149afcd2d0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qfmnn"
Sep 18 13:49:51.052022 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:51.051984    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2s6g\" (UniqueName: \"kubernetes.io/projected/7ca36e2e-36cd-4997-b057-e5149afcd2d0-kube-api-access-j2s6g\") pod \"cert-manager-operator-controller-manager-55c444bf86-qfmnn\" (UID: \"7ca36e2e-36cd-4997-b057-e5149afcd2d0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qfmnn"
Sep 18 13:49:51.052207 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:51.052039    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ca36e2e-36cd-4997-b057-e5149afcd2d0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qfmnn\" (UID: \"7ca36e2e-36cd-4997-b057-e5149afcd2d0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qfmnn"
Sep 18 13:49:51.052462 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:51.052440    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ca36e2e-36cd-4997-b057-e5149afcd2d0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qfmnn\" (UID: \"7ca36e2e-36cd-4997-b057-e5149afcd2d0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qfmnn"
Sep 18 13:49:51.066731 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:51.066702    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2s6g\" (UniqueName: \"kubernetes.io/projected/7ca36e2e-36cd-4997-b057-e5149afcd2d0-kube-api-access-j2s6g\") pod \"cert-manager-operator-controller-manager-55c444bf86-qfmnn\" (UID: \"7ca36e2e-36cd-4997-b057-e5149afcd2d0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qfmnn"
Sep 18 13:49:51.180868 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:51.180789    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qfmnn"
Sep 18 13:49:51.471553 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:51.471511    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qfmnn"]
Sep 18 13:49:51.473656 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:49:51.473618    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca36e2e_36cd_4997_b057_e5149afcd2d0.slice/crio-03d55c6d7f284a76965e494ef7884bbd71309b4123b2177774c05e8d960972ea WatchSource:0}: Error finding container 03d55c6d7f284a76965e494ef7884bbd71309b4123b2177774c05e8d960972ea: Status 404 returned error can't find the container with id 03d55c6d7f284a76965e494ef7884bbd71309b4123b2177774c05e8d960972ea
Sep 18 13:49:51.530259 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:51.530225    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qfmnn" event={"ID":"7ca36e2e-36cd-4997-b057-e5149afcd2d0","Type":"ContainerStarted","Data":"03d55c6d7f284a76965e494ef7884bbd71309b4123b2177774c05e8d960972ea"}
Sep 18 13:49:53.540591 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:53.540556    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qfmnn" event={"ID":"7ca36e2e-36cd-4997-b057-e5149afcd2d0","Type":"ContainerStarted","Data":"9fadeb77c77c5bf6ea08f40e11522519d9a2dab963b5056ab8a961e85d5020c3"}
Sep 18 13:49:53.564228 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:53.564182    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qfmnn" podStartSLOduration=1.954520967 podStartE2EDuration="3.564169554s" totalImagesPullingTime="1.609648587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:49:53.563153084 +0000 UTC m=+329.498118229" watchObservedRunningTime="2026-09-18 13:49:53.564169554 +0000 UTC m=+329.499134698"
Sep 18 13:49:54.889388 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:54.889350    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978"]
Sep 18 13:49:54.891524 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:54.891507    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978"
Sep 18 13:49:54.894405 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:54.894374    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 13:49:54.894405 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:54.894397    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 13:49:54.894878 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:54.894863    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4m6sb\""
Sep 18 13:49:54.910375 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:54.910353    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978"]
Sep 18 13:49:54.987575 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:54.987545    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978\" (UID: \"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978"
Sep 18 13:49:54.987745 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:54.987590    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj7v9\" (UniqueName: \"kubernetes.io/projected/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-kube-api-access-rj7v9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978\" (UID: \"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978"
Sep 18 13:49:54.987745 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:54.987612    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978\" (UID: \"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978"
Sep 18 13:49:55.088061 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:55.088017    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978\" (UID: \"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978"
Sep 18 13:49:55.088061 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:55.088070    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj7v9\" (UniqueName: \"kubernetes.io/projected/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-kube-api-access-rj7v9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978\" (UID: \"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978"
Sep 18 13:49:55.088311 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:55.088094    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978\" (UID: \"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978"
Sep 18 13:49:55.088402 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:55.088380    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978\" (UID: \"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978"
Sep 18 13:49:55.088402 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:55.088398    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978\" (UID: \"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978"
Sep 18 13:49:55.115449 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:55.115430    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj7v9\" (UniqueName: \"kubernetes.io/projected/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-kube-api-access-rj7v9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978\" (UID: \"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978"
Sep 18 13:49:55.200637 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:55.200608    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978"
Sep 18 13:49:55.468091 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:55.468064    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978"]
Sep 18 13:49:55.470628 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:49:55.470599    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e5743c_b2c1_4fa6_ba62_2055ccc71c1b.slice/crio-1a37e13291a4a153305b0445a27d93e65bf5a37276d4af6d907c3b5e0d6098ba WatchSource:0}: Error finding container 1a37e13291a4a153305b0445a27d93e65bf5a37276d4af6d907c3b5e0d6098ba: Status 404 returned error can't find the container with id 1a37e13291a4a153305b0445a27d93e65bf5a37276d4af6d907c3b5e0d6098ba
Sep 18 13:49:55.550374 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:55.550336    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978" event={"ID":"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b","Type":"ContainerStarted","Data":"06a837ebaf634faee47460d5ecfc731f1552da0b2efeeec11bb5c21a8f87416e"}
Sep 18 13:49:55.550512 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:55.550387    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978" event={"ID":"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b","Type":"ContainerStarted","Data":"1a37e13291a4a153305b0445a27d93e65bf5a37276d4af6d907c3b5e0d6098ba"}
Sep 18 13:49:56.555312 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:56.555279    2572 generic.go:356] "Generic (PLEG): container finished" podID="c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b" containerID="06a837ebaf634faee47460d5ecfc731f1552da0b2efeeec11bb5c21a8f87416e" exitCode=0
Sep 18 13:49:56.555697 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:56.555366    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978" event={"ID":"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b","Type":"ContainerDied","Data":"06a837ebaf634faee47460d5ecfc731f1552da0b2efeeec11bb5c21a8f87416e"}
Sep 18 13:49:59.570670 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:59.570634    2572 generic.go:356] "Generic (PLEG): container finished" podID="c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b" containerID="a0f25e89db955d9494118e8a71044971ed77e3768d487b8e9535baf6409a01c8" exitCode=0
Sep 18 13:49:59.571045 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:49:59.570720    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978" event={"ID":"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b","Type":"ContainerDied","Data":"a0f25e89db955d9494118e8a71044971ed77e3768d487b8e9535baf6409a01c8"}
Sep 18 13:50:00.576946 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:00.576913    2572 generic.go:356] "Generic (PLEG): container finished" podID="c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b" containerID="c4aca677f8249cd1f65b7a4b32ee0037c6aa71c62f06682798dca7e2f72b4dde" exitCode=0
Sep 18 13:50:00.577307 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:00.576992    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978" event={"ID":"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b","Type":"ContainerDied","Data":"c4aca677f8249cd1f65b7a4b32ee0037c6aa71c62f06682798dca7e2f72b4dde"}
Sep 18 13:50:01.703827 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:01.703809    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978"
Sep 18 13:50:01.847514 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:01.847432    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-bundle\") pod \"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b\" (UID: \"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b\") "
Sep 18 13:50:01.847514 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:01.847496    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rj7v9\" (UniqueName: \"kubernetes.io/projected/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-kube-api-access-rj7v9\") pod \"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b\" (UID: \"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b\") "
Sep 18 13:50:01.847759 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:01.847553    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-util\") pod \"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b\" (UID: \"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b\") "
Sep 18 13:50:01.847925 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:01.847899    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-bundle" (OuterVolumeSpecName: "bundle") pod "c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b" (UID: "c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:50:01.849605 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:01.849580    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-kube-api-access-rj7v9" (OuterVolumeSpecName: "kube-api-access-rj7v9") pod "c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b" (UID: "c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b"). InnerVolumeSpecName "kube-api-access-rj7v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:50:01.852124 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:01.852106    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-util" (OuterVolumeSpecName: "util") pod "c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b" (UID: "c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:50:01.948614 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:01.948567    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-bundle\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:50:01.948614 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:01.948608    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj7v9\" (UniqueName: \"kubernetes.io/projected/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-kube-api-access-rj7v9\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:50:01.948614 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:01.948619    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b-util\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:50:02.587075 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:02.586986    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978" event={"ID":"c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b","Type":"ContainerDied","Data":"1a37e13291a4a153305b0445a27d93e65bf5a37276d4af6d907c3b5e0d6098ba"}
Sep 18 13:50:02.587075 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:02.587044    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a37e13291a4a153305b0445a27d93e65bf5a37276d4af6d907c3b5e0d6098ba"
Sep 18 13:50:02.587075 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:02.587000    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzn978"
Sep 18 13:50:04.478036 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.477998    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k8dhx"]
Sep 18 13:50:04.478405 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.478264    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b" containerName="extract"
Sep 18 13:50:04.478405 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.478275    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b" containerName="extract"
Sep 18 13:50:04.478405 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.478286    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b" containerName="util"
Sep 18 13:50:04.478405 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.478292    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b" containerName="util"
Sep 18 13:50:04.478405 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.478307    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b" containerName="pull"
Sep 18 13:50:04.478405 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.478312    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b" containerName="pull"
Sep 18 13:50:04.478405 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.478354    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e5743c-b2c1-4fa6-ba62-2055ccc71c1b" containerName="extract"
Sep 18 13:50:04.481397 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.481381    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-k8dhx"
Sep 18 13:50:04.486223 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.486200    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 13:50:04.486419 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.486401    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2kk2g\""
Sep 18 13:50:04.486793 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.486776    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 13:50:04.517513 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.517487    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k8dhx"]
Sep 18 13:50:04.569993 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.569959    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbdjf\" (UniqueName: \"kubernetes.io/projected/e406700d-8557-4b74-b5c5-a0fe3e018383-kube-api-access-vbdjf\") pod \"cert-manager-55b4d954d5-k8dhx\" (UID: \"e406700d-8557-4b74-b5c5-a0fe3e018383\") " pod="cert-manager/cert-manager-55b4d954d5-k8dhx"
Sep 18 13:50:04.570147 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.570013    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e406700d-8557-4b74-b5c5-a0fe3e018383-bound-sa-token\") pod \"cert-manager-55b4d954d5-k8dhx\" (UID: \"e406700d-8557-4b74-b5c5-a0fe3e018383\") " pod="cert-manager/cert-manager-55b4d954d5-k8dhx"
Sep 18 13:50:04.670974 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.670935    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e406700d-8557-4b74-b5c5-a0fe3e018383-bound-sa-token\") pod \"cert-manager-55b4d954d5-k8dhx\" (UID: \"e406700d-8557-4b74-b5c5-a0fe3e018383\") " pod="cert-manager/cert-manager-55b4d954d5-k8dhx"
Sep 18 13:50:04.671123 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.671014    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbdjf\" (UniqueName: \"kubernetes.io/projected/e406700d-8557-4b74-b5c5-a0fe3e018383-kube-api-access-vbdjf\") pod \"cert-manager-55b4d954d5-k8dhx\" (UID: \"e406700d-8557-4b74-b5c5-a0fe3e018383\") " pod="cert-manager/cert-manager-55b4d954d5-k8dhx"
Sep 18 13:50:04.684735 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.684711    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbdjf\" (UniqueName: \"kubernetes.io/projected/e406700d-8557-4b74-b5c5-a0fe3e018383-kube-api-access-vbdjf\") pod \"cert-manager-55b4d954d5-k8dhx\" (UID: \"e406700d-8557-4b74-b5c5-a0fe3e018383\") " pod="cert-manager/cert-manager-55b4d954d5-k8dhx"
Sep 18 13:50:04.686216 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.686195    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e406700d-8557-4b74-b5c5-a0fe3e018383-bound-sa-token\") pod \"cert-manager-55b4d954d5-k8dhx\" (UID: \"e406700d-8557-4b74-b5c5-a0fe3e018383\") " pod="cert-manager/cert-manager-55b4d954d5-k8dhx"
Sep 18 13:50:04.808468 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:04.808395    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-k8dhx"
Sep 18 13:50:05.101186 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:05.101159    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k8dhx"]
Sep 18 13:50:05.103584 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:50:05.103556    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode406700d_8557_4b74_b5c5_a0fe3e018383.slice/crio-736cb28c49defd7589280bdcdde189001986cef6fc8c1e2a86667377eb980da0 WatchSource:0}: Error finding container 736cb28c49defd7589280bdcdde189001986cef6fc8c1e2a86667377eb980da0: Status 404 returned error can't find the container with id 736cb28c49defd7589280bdcdde189001986cef6fc8c1e2a86667377eb980da0
Sep 18 13:50:05.602122 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:05.602084    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-k8dhx" event={"ID":"e406700d-8557-4b74-b5c5-a0fe3e018383","Type":"ContainerStarted","Data":"736cb28c49defd7589280bdcdde189001986cef6fc8c1e2a86667377eb980da0"}
Sep 18 13:50:08.617033 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:08.616960    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-k8dhx" event={"ID":"e406700d-8557-4b74-b5c5-a0fe3e018383","Type":"ContainerStarted","Data":"0e6acd69f98297c053be2f0ed7e73e930d66c700bc6eeac4c0cd692aa067e609"}
Sep 18 13:50:08.648179 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:08.648133    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-k8dhx" podStartSLOduration=2.140338389 podStartE2EDuration="4.648119231s" totalImagesPullingTime="2.507780842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:50:08.647849556 +0000 UTC m=+344.582814713" watchObservedRunningTime="2026-09-18 13:50:08.648119231 +0000 UTC m=+344.583084369"
Sep 18 13:50:18.519460 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:18.519428    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg"]
Sep 18 13:50:18.521617 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:18.521599    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg"
Sep 18 13:50:18.523745 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:18.523725    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 13:50:18.523924 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:18.523905    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4m6sb\""
Sep 18 13:50:18.523981 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:18.523955    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 13:50:18.530747 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:18.530724    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg"]
Sep 18 13:50:18.694799 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:18.694765    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tsdc\" (UniqueName: \"kubernetes.io/projected/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-kube-api-access-2tsdc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg\" (UID: \"9e09dcc2-b2e9-4584-b4b2-429a18fb8730\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg"
Sep 18 13:50:18.694965 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:18.694818    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg\" (UID: \"9e09dcc2-b2e9-4584-b4b2-429a18fb8730\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg"
Sep 18 13:50:18.694965 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:18.694839    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg\" (UID: \"9e09dcc2-b2e9-4584-b4b2-429a18fb8730\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg"
Sep 18 13:50:18.795734 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:18.795655    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tsdc\" (UniqueName: \"kubernetes.io/projected/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-kube-api-access-2tsdc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg\" (UID: \"9e09dcc2-b2e9-4584-b4b2-429a18fb8730\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg"
Sep 18 13:50:18.795734 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:18.795714    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg\" (UID: \"9e09dcc2-b2e9-4584-b4b2-429a18fb8730\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg"
Sep 18 13:50:18.795943 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:18.795745    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg\" (UID: \"9e09dcc2-b2e9-4584-b4b2-429a18fb8730\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg"
Sep 18 13:50:18.796141 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:18.796122    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg\" (UID: \"9e09dcc2-b2e9-4584-b4b2-429a18fb8730\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg"
Sep 18 13:50:18.796218 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:18.796179    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg\" (UID: \"9e09dcc2-b2e9-4584-b4b2-429a18fb8730\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg"
Sep 18 13:50:18.804579 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:18.804556    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tsdc\" (UniqueName: \"kubernetes.io/projected/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-kube-api-access-2tsdc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg\" (UID: \"9e09dcc2-b2e9-4584-b4b2-429a18fb8730\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg"
Sep 18 13:50:18.830815 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:18.830791    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg"
Sep 18 13:50:19.100443 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:19.100372    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg"]
Sep 18 13:50:19.104405 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:50:19.104377    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e09dcc2_b2e9_4584_b4b2_429a18fb8730.slice/crio-1eab3c9b2d9f7313f9dde0fded344a9d6a19760aca314bd52e7a602e3598f0ba WatchSource:0}: Error finding container 1eab3c9b2d9f7313f9dde0fded344a9d6a19760aca314bd52e7a602e3598f0ba: Status 404 returned error can't find the container with id 1eab3c9b2d9f7313f9dde0fded344a9d6a19760aca314bd52e7a602e3598f0ba
Sep 18 13:50:19.669468 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:19.669436    2572 generic.go:356] "Generic (PLEG): container finished" podID="9e09dcc2-b2e9-4584-b4b2-429a18fb8730" containerID="5645622e91bd818849dfa259c2f264128dd515b94e9b822899f1fdddbd22329a" exitCode=0
Sep 18 13:50:19.669944 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:19.669525    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg" event={"ID":"9e09dcc2-b2e9-4584-b4b2-429a18fb8730","Type":"ContainerDied","Data":"5645622e91bd818849dfa259c2f264128dd515b94e9b822899f1fdddbd22329a"}
Sep 18 13:50:19.669944 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:19.669617    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg" event={"ID":"9e09dcc2-b2e9-4584-b4b2-429a18fb8730","Type":"ContainerStarted","Data":"1eab3c9b2d9f7313f9dde0fded344a9d6a19760aca314bd52e7a602e3598f0ba"}
Sep 18 13:50:20.675458 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:20.675368    2572 generic.go:356] "Generic (PLEG): container finished" podID="9e09dcc2-b2e9-4584-b4b2-429a18fb8730" containerID="7d7cd4083491550d7d3ac73d5bfb76b9c4def7552ba2beaca1b405c174b3ec5c" exitCode=0
Sep 18 13:50:20.675833 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:20.675450    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg" event={"ID":"9e09dcc2-b2e9-4584-b4b2-429a18fb8730","Type":"ContainerDied","Data":"7d7cd4083491550d7d3ac73d5bfb76b9c4def7552ba2beaca1b405c174b3ec5c"}
Sep 18 13:50:21.681497 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:21.681454    2572 generic.go:356] "Generic (PLEG): container finished" podID="9e09dcc2-b2e9-4584-b4b2-429a18fb8730" containerID="7513c92dc1f3ff763a2fa230628dc0786ea4e2fb28ba618085e52b2775c3d8c2" exitCode=0
Sep 18 13:50:21.681880 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:21.681525    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg" event={"ID":"9e09dcc2-b2e9-4584-b4b2-429a18fb8730","Type":"ContainerDied","Data":"7513c92dc1f3ff763a2fa230628dc0786ea4e2fb28ba618085e52b2775c3d8c2"}
Sep 18 13:50:22.807483 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:22.807459    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg"
Sep 18 13:50:22.828381 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:22.828359    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tsdc\" (UniqueName: \"kubernetes.io/projected/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-kube-api-access-2tsdc\") pod \"9e09dcc2-b2e9-4584-b4b2-429a18fb8730\" (UID: \"9e09dcc2-b2e9-4584-b4b2-429a18fb8730\") "
Sep 18 13:50:22.828508 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:22.828399    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-util\") pod \"9e09dcc2-b2e9-4584-b4b2-429a18fb8730\" (UID: \"9e09dcc2-b2e9-4584-b4b2-429a18fb8730\") "
Sep 18 13:50:22.828508 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:22.828447    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-bundle\") pod \"9e09dcc2-b2e9-4584-b4b2-429a18fb8730\" (UID: \"9e09dcc2-b2e9-4584-b4b2-429a18fb8730\") "
Sep 18 13:50:22.829381 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:22.829353    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-bundle" (OuterVolumeSpecName: "bundle") pod "9e09dcc2-b2e9-4584-b4b2-429a18fb8730" (UID: "9e09dcc2-b2e9-4584-b4b2-429a18fb8730"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:50:22.830413 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:22.830378    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-kube-api-access-2tsdc" (OuterVolumeSpecName: "kube-api-access-2tsdc") pod "9e09dcc2-b2e9-4584-b4b2-429a18fb8730" (UID: "9e09dcc2-b2e9-4584-b4b2-429a18fb8730"). InnerVolumeSpecName "kube-api-access-2tsdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:50:22.834310 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:22.834286    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-util" (OuterVolumeSpecName: "util") pod "9e09dcc2-b2e9-4584-b4b2-429a18fb8730" (UID: "9e09dcc2-b2e9-4584-b4b2-429a18fb8730"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:50:22.929829 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:22.929789    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tsdc\" (UniqueName: \"kubernetes.io/projected/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-kube-api-access-2tsdc\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:50:22.929829 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:22.929819    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-util\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:50:22.929829 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:22.929829    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e09dcc2-b2e9-4584-b4b2-429a18fb8730-bundle\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:50:23.695008 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:23.694972    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg" event={"ID":"9e09dcc2-b2e9-4584-b4b2-429a18fb8730","Type":"ContainerDied","Data":"1eab3c9b2d9f7313f9dde0fded344a9d6a19760aca314bd52e7a602e3598f0ba"}
Sep 18 13:50:23.695183 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:23.695023    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eab3c9b2d9f7313f9dde0fded344a9d6a19760aca314bd52e7a602e3598f0ba"
Sep 18 13:50:23.695183 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:23.695029    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2wbkg"
Sep 18 13:50:32.525425 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.525389    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5"]
Sep 18 13:50:32.525920 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.525694    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e09dcc2-b2e9-4584-b4b2-429a18fb8730" containerName="util"
Sep 18 13:50:32.525920 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.525706    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e09dcc2-b2e9-4584-b4b2-429a18fb8730" containerName="util"
Sep 18 13:50:32.525920 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.525727    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e09dcc2-b2e9-4584-b4b2-429a18fb8730" containerName="extract"
Sep 18 13:50:32.525920 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.525733    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e09dcc2-b2e9-4584-b4b2-429a18fb8730" containerName="extract"
Sep 18 13:50:32.525920 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.525741    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e09dcc2-b2e9-4584-b4b2-429a18fb8730" containerName="pull"
Sep 18 13:50:32.525920 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.525746    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e09dcc2-b2e9-4584-b4b2-429a18fb8730" containerName="pull"
Sep 18 13:50:32.525920 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.525795    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e09dcc2-b2e9-4584-b4b2-429a18fb8730" containerName="extract"
Sep 18 13:50:32.533360 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.533343    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5"
Sep 18 13:50:32.535643 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.535614    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 13:50:32.535784 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.535702    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4m6sb\""
Sep 18 13:50:32.535784 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.535710    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 13:50:32.540235 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.539888    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5"]
Sep 18 13:50:32.601640 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.601611    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsg56\" (UniqueName: \"kubernetes.io/projected/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-kube-api-access-nsg56\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5\" (UID: \"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5"
Sep 18 13:50:32.601758 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.601691    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5\" (UID: \"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5"
Sep 18 13:50:32.601804 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.601768    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5\" (UID: \"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5"
Sep 18 13:50:32.702065 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.702040    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5\" (UID: \"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5"
Sep 18 13:50:32.702166 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.702080    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5\" (UID: \"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5"
Sep 18 13:50:32.702166 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.702115    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsg56\" (UniqueName: \"kubernetes.io/projected/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-kube-api-access-nsg56\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5\" (UID: \"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5"
Sep 18 13:50:32.702442 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.702426    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5\" (UID: \"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5"
Sep 18 13:50:32.702507 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.702489    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5\" (UID: \"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5"
Sep 18 13:50:32.710779 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.710755    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsg56\" (UniqueName: \"kubernetes.io/projected/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-kube-api-access-nsg56\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5\" (UID: \"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5"
Sep 18 13:50:32.842842 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:32.842782    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5"
Sep 18 13:50:33.136076 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:33.136040    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5"]
Sep 18 13:50:33.137335 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:50:33.137310    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe1c7a3_5d8f_4846_bbf6_b8f3bf400ef1.slice/crio-e2961c490d058889768eb103694b6aabfc11fc24b2fef6582f04cba12daf2f8b WatchSource:0}: Error finding container e2961c490d058889768eb103694b6aabfc11fc24b2fef6582f04cba12daf2f8b: Status 404 returned error can't find the container with id e2961c490d058889768eb103694b6aabfc11fc24b2fef6582f04cba12daf2f8b
Sep 18 13:50:33.738365 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:33.738335    2572 generic.go:356] "Generic (PLEG): container finished" podID="0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1" containerID="bff0d3f81022d44f188a0e8520fcf9ee95ef6648600bdcebe1ba47cdff271d00" exitCode=0
Sep 18 13:50:33.738762 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:33.738428    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5" event={"ID":"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1","Type":"ContainerDied","Data":"bff0d3f81022d44f188a0e8520fcf9ee95ef6648600bdcebe1ba47cdff271d00"}
Sep 18 13:50:33.738762 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:33.738491    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5" event={"ID":"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1","Type":"ContainerStarted","Data":"e2961c490d058889768eb103694b6aabfc11fc24b2fef6582f04cba12daf2f8b"}
Sep 18 13:50:34.665691 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.665609    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"]
Sep 18 13:50:34.668865 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.668850    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:34.672286 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.672266    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 13:50:34.672573 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.672557    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-g9q4g\""
Sep 18 13:50:34.676644 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.676626    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 13:50:34.676744 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.676673    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 13:50:34.676744 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.676720    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 13:50:34.700903 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.700881    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"]
Sep 18 13:50:34.716641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.716614    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/66150446-024c-4fd4-994e-fa7eeb9c9f5a-manifests\") pod \"opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz\" (UID: \"66150446-024c-4fd4-994e-fa7eeb9c9f5a\") " pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:34.716755 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.716651    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdfrp\" (UniqueName: \"kubernetes.io/projected/66150446-024c-4fd4-994e-fa7eeb9c9f5a-kube-api-access-qdfrp\") pod \"opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz\" (UID: \"66150446-024c-4fd4-994e-fa7eeb9c9f5a\") " pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:34.716755 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.716689    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66150446-024c-4fd4-994e-fa7eeb9c9f5a-webhook-cert\") pod \"opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz\" (UID: \"66150446-024c-4fd4-994e-fa7eeb9c9f5a\") " pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:34.716755 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.716737    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66150446-024c-4fd4-994e-fa7eeb9c9f5a-tmp\") pod \"opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz\" (UID: \"66150446-024c-4fd4-994e-fa7eeb9c9f5a\") " pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:34.716884 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.716770    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/66150446-024c-4fd4-994e-fa7eeb9c9f5a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz\" (UID: \"66150446-024c-4fd4-994e-fa7eeb9c9f5a\") " pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:34.744959 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.744932    2572 generic.go:356] "Generic (PLEG): container finished" podID="0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1" containerID="017c762d855a1df6556761761ac109bd26f45193f3c1193cec1cbee2c97728a7" exitCode=0
Sep 18 13:50:34.745268 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.744998    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5" event={"ID":"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1","Type":"ContainerDied","Data":"017c762d855a1df6556761761ac109bd26f45193f3c1193cec1cbee2c97728a7"}
Sep 18 13:50:34.817435 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.817408    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/66150446-024c-4fd4-994e-fa7eeb9c9f5a-manifests\") pod \"opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz\" (UID: \"66150446-024c-4fd4-994e-fa7eeb9c9f5a\") " pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:34.817592 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.817440    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdfrp\" (UniqueName: \"kubernetes.io/projected/66150446-024c-4fd4-994e-fa7eeb9c9f5a-kube-api-access-qdfrp\") pod \"opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz\" (UID: \"66150446-024c-4fd4-994e-fa7eeb9c9f5a\") " pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:34.817592 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.817494    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66150446-024c-4fd4-994e-fa7eeb9c9f5a-webhook-cert\") pod \"opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz\" (UID: \"66150446-024c-4fd4-994e-fa7eeb9c9f5a\") " pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:34.817592 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.817583    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66150446-024c-4fd4-994e-fa7eeb9c9f5a-tmp\") pod \"opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz\" (UID: \"66150446-024c-4fd4-994e-fa7eeb9c9f5a\") " pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:34.817781 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.817618    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/66150446-024c-4fd4-994e-fa7eeb9c9f5a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz\" (UID: \"66150446-024c-4fd4-994e-fa7eeb9c9f5a\") " pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:34.817781 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.817766    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/66150446-024c-4fd4-994e-fa7eeb9c9f5a-manifests\") pod \"opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz\" (UID: \"66150446-024c-4fd4-994e-fa7eeb9c9f5a\") " pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:34.819719 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.819693    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66150446-024c-4fd4-994e-fa7eeb9c9f5a-tmp\") pod \"opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz\" (UID: \"66150446-024c-4fd4-994e-fa7eeb9c9f5a\") " pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:34.819852 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.819774    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66150446-024c-4fd4-994e-fa7eeb9c9f5a-webhook-cert\") pod \"opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz\" (UID: \"66150446-024c-4fd4-994e-fa7eeb9c9f5a\") " pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:34.819943 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.819925    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/66150446-024c-4fd4-994e-fa7eeb9c9f5a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz\" (UID: \"66150446-024c-4fd4-994e-fa7eeb9c9f5a\") " pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:34.831550 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.831514    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdfrp\" (UniqueName: \"kubernetes.io/projected/66150446-024c-4fd4-994e-fa7eeb9c9f5a-kube-api-access-qdfrp\") pod \"opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz\" (UID: \"66150446-024c-4fd4-994e-fa7eeb9c9f5a\") " pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:34.979425 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:34.979395    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:35.254606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:35.254578    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"]
Sep 18 13:50:35.255683 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:50:35.255656    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66150446_024c_4fd4_994e_fa7eeb9c9f5a.slice/crio-fc7d90f1cb8123c612fb950b5f6c26596b119316d1df533accfdb3266d26f712 WatchSource:0}: Error finding container fc7d90f1cb8123c612fb950b5f6c26596b119316d1df533accfdb3266d26f712: Status 404 returned error can't find the container with id fc7d90f1cb8123c612fb950b5f6c26596b119316d1df533accfdb3266d26f712
Sep 18 13:50:35.752231 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:35.752190    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz" event={"ID":"66150446-024c-4fd4-994e-fa7eeb9c9f5a","Type":"ContainerStarted","Data":"fc7d90f1cb8123c612fb950b5f6c26596b119316d1df533accfdb3266d26f712"}
Sep 18 13:50:35.754712 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:35.754684    2572 generic.go:356] "Generic (PLEG): container finished" podID="0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1" containerID="d2edde3295824a99fcf583c802535949fcea0d9b716ea8547f423673e5f808fd" exitCode=0
Sep 18 13:50:35.754836 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:35.754727    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5" event={"ID":"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1","Type":"ContainerDied","Data":"d2edde3295824a99fcf583c802535949fcea0d9b716ea8547f423673e5f808fd"}
Sep 18 13:50:37.217085 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:37.217056    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5"
Sep 18 13:50:37.239318 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:37.239295    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsg56\" (UniqueName: \"kubernetes.io/projected/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-kube-api-access-nsg56\") pod \"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1\" (UID: \"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1\") "
Sep 18 13:50:37.239455 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:37.239352    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-bundle\") pod \"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1\" (UID: \"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1\") "
Sep 18 13:50:37.239455 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:37.239379    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-util\") pod \"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1\" (UID: \"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1\") "
Sep 18 13:50:37.240374 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:37.240349    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-bundle" (OuterVolumeSpecName: "bundle") pod "0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1" (UID: "0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:50:37.242140 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:37.242103    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-kube-api-access-nsg56" (OuterVolumeSpecName: "kube-api-access-nsg56") pod "0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1" (UID: "0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1"). InnerVolumeSpecName "kube-api-access-nsg56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:50:37.245838 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:37.245813    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-util" (OuterVolumeSpecName: "util") pod "0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1" (UID: "0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:50:37.340669 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:37.340634    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsg56\" (UniqueName: \"kubernetes.io/projected/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-kube-api-access-nsg56\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:50:37.340813 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:37.340690    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-bundle\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:50:37.340813 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:37.340706    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1-util\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:50:37.767899 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:37.767855    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5" event={"ID":"0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1","Type":"ContainerDied","Data":"e2961c490d058889768eb103694b6aabfc11fc24b2fef6582f04cba12daf2f8b"}
Sep 18 13:50:37.768079 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:37.767904    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b029bt5"
Sep 18 13:50:37.768079 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:37.767914    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2961c490d058889768eb103694b6aabfc11fc24b2fef6582f04cba12daf2f8b"
Sep 18 13:50:38.775176 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:38.775140    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz" event={"ID":"66150446-024c-4fd4-994e-fa7eeb9c9f5a","Type":"ContainerStarted","Data":"657d1344f8e103d4077e8c186a4eff78a19542e2d7d162591cec61882cadd7d6"}
Sep 18 13:50:39.781590 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:39.781558    2572 generic.go:356] "Generic (PLEG): container finished" podID="66150446-024c-4fd4-994e-fa7eeb9c9f5a" containerID="657d1344f8e103d4077e8c186a4eff78a19542e2d7d162591cec61882cadd7d6" exitCode=0
Sep 18 13:50:39.781972 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:39.781629    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz" event={"ID":"66150446-024c-4fd4-994e-fa7eeb9c9f5a","Type":"ContainerDied","Data":"657d1344f8e103d4077e8c186a4eff78a19542e2d7d162591cec61882cadd7d6"}
Sep 18 13:50:40.787793 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:40.787755    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz" event={"ID":"66150446-024c-4fd4-994e-fa7eeb9c9f5a","Type":"ContainerStarted","Data":"7683ae8f9ba432021ed2df274dbaf891ddc4ce47f2070a11dcbbd79666d1f07a"}
Sep 18 13:50:40.788209 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:40.787907    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:40.815199 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:40.815151    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz" podStartSLOduration=2.584756983 podStartE2EDuration="6.815130817s" totalImagesPullingTime="3.22483399s" totalInitContainerRuntime="1.005539844s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:50:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:50:40.813713153 +0000 UTC m=+376.748678300" watchObservedRunningTime="2026-09-18 13:50:40.815130817 +0000 UTC m=+376.750095962"
Sep 18 13:50:51.794928 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:51.794892    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz"
Sep 18 13:50:53.914564 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:53.914519    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt"]
Sep 18 13:50:53.914935 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:53.914815    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1" containerName="pull"
Sep 18 13:50:53.914935 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:53.914826    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1" containerName="pull"
Sep 18 13:50:53.914935 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:53.914838    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1" containerName="util"
Sep 18 13:50:53.914935 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:53.914844    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1" containerName="util"
Sep 18 13:50:53.914935 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:53.914859    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1" containerName="extract"
Sep 18 13:50:53.914935 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:53.914865    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1" containerName="extract"
Sep 18 13:50:53.914935 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:53.914913    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fe1c7a3-5d8f-4846-bbf6-b8f3bf400ef1" containerName="extract"
Sep 18 13:50:53.922795 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:53.922769    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt"
Sep 18 13:50:53.931502 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:53.931476    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 13:50:53.931657 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:53.931483    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 13:50:53.950474 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:53.950450    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt"]
Sep 18 13:50:53.976894 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:53.976865    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4144f489-44f8-4933-8d38-6c0fea51a78f-tmp\") pod \"kube-auth-proxy-849fd8447d-n2lrt\" (UID: \"4144f489-44f8-4933-8d38-6c0fea51a78f\") " pod="openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt"
Sep 18 13:50:53.977063 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:53.976908    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq9k8\" (UniqueName: \"kubernetes.io/projected/4144f489-44f8-4933-8d38-6c0fea51a78f-kube-api-access-bq9k8\") pod \"kube-auth-proxy-849fd8447d-n2lrt\" (UID: \"4144f489-44f8-4933-8d38-6c0fea51a78f\") " pod="openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt"
Sep 18 13:50:53.977063 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:53.976981    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4144f489-44f8-4933-8d38-6c0fea51a78f-tls-certs\") pod \"kube-auth-proxy-849fd8447d-n2lrt\" (UID: \"4144f489-44f8-4933-8d38-6c0fea51a78f\") " pod="openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt"
Sep 18 13:50:54.077736 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:54.077705    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4144f489-44f8-4933-8d38-6c0fea51a78f-tmp\") pod \"kube-auth-proxy-849fd8447d-n2lrt\" (UID: \"4144f489-44f8-4933-8d38-6c0fea51a78f\") " pod="openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt"
Sep 18 13:50:54.077888 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:54.077741    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9k8\" (UniqueName: \"kubernetes.io/projected/4144f489-44f8-4933-8d38-6c0fea51a78f-kube-api-access-bq9k8\") pod \"kube-auth-proxy-849fd8447d-n2lrt\" (UID: \"4144f489-44f8-4933-8d38-6c0fea51a78f\") " pod="openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt"
Sep 18 13:50:54.077888 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:54.077766    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4144f489-44f8-4933-8d38-6c0fea51a78f-tls-certs\") pod \"kube-auth-proxy-849fd8447d-n2lrt\" (UID: \"4144f489-44f8-4933-8d38-6c0fea51a78f\") " pod="openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt"
Sep 18 13:50:54.077888 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:50:54.077879    2572 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 18 13:50:54.077988 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:50:54.077964    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4144f489-44f8-4933-8d38-6c0fea51a78f-tls-certs podName:4144f489-44f8-4933-8d38-6c0fea51a78f nodeName:}" failed. No retries permitted until 2026-09-18 13:50:54.577944903 +0000 UTC m=+390.512910026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4144f489-44f8-4933-8d38-6c0fea51a78f-tls-certs") pod "kube-auth-proxy-849fd8447d-n2lrt" (UID: "4144f489-44f8-4933-8d38-6c0fea51a78f") : secret "kube-auth-proxy-tls" not found
Sep 18 13:50:54.079939 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:54.079908    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4144f489-44f8-4933-8d38-6c0fea51a78f-tmp\") pod \"kube-auth-proxy-849fd8447d-n2lrt\" (UID: \"4144f489-44f8-4933-8d38-6c0fea51a78f\") " pod="openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt"
Sep 18 13:50:54.112353 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:54.112319    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9k8\" (UniqueName: \"kubernetes.io/projected/4144f489-44f8-4933-8d38-6c0fea51a78f-kube-api-access-bq9k8\") pod \"kube-auth-proxy-849fd8447d-n2lrt\" (UID: \"4144f489-44f8-4933-8d38-6c0fea51a78f\") " pod="openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt"
Sep 18 13:50:54.581132 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:54.581097    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4144f489-44f8-4933-8d38-6c0fea51a78f-tls-certs\") pod \"kube-auth-proxy-849fd8447d-n2lrt\" (UID: \"4144f489-44f8-4933-8d38-6c0fea51a78f\") " pod="openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt"
Sep 18 13:50:54.583600 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:54.583570    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4144f489-44f8-4933-8d38-6c0fea51a78f-tls-certs\") pod \"kube-auth-proxy-849fd8447d-n2lrt\" (UID: \"4144f489-44f8-4933-8d38-6c0fea51a78f\") " pod="openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt"
Sep 18 13:50:54.832458 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:54.832386    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt"
Sep 18 13:50:55.115592 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.115567    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt"]
Sep 18 13:50:55.117794 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:50:55.117765    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4144f489_44f8_4933_8d38_6c0fea51a78f.slice/crio-5c7149488d1557f10f9e34149020ca2babbad1e9968149c8bc27555c34c64b79 WatchSource:0}: Error finding container 5c7149488d1557f10f9e34149020ca2babbad1e9968149c8bc27555c34c64b79: Status 404 returned error can't find the container with id 5c7149488d1557f10f9e34149020ca2babbad1e9968149c8bc27555c34c64b79
Sep 18 13:50:55.744137 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.744098    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"]
Sep 18 13:50:55.747525 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.747501    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.751029 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.750994    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 18 13:50:55.751141 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.751001    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 13:50:55.751671 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.751652    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 18 13:50:55.752093 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.752071    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 18 13:50:55.752308 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.752288    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9h58x\""
Sep 18 13:50:55.779421 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.779397    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"]
Sep 18 13:50:55.861008 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.860937    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt" event={"ID":"4144f489-44f8-4933-8d38-6c0fea51a78f","Type":"ContainerStarted","Data":"5c7149488d1557f10f9e34149020ca2babbad1e9968149c8bc27555c34c64b79"}
Sep 18 13:50:55.892644 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.892609    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.892789 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.892659    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.892789 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.892692    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.892920 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.892806    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.892920 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.892876    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.892920 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.892902    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh4g8\" (UniqueName: \"kubernetes.io/projected/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-kube-api-access-rh4g8\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.893078 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.892940    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.993668 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.993510    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.993668 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.993605    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.993668 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.993637    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh4g8\" (UniqueName: \"kubernetes.io/projected/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-kube-api-access-rh4g8\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.993668 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.993672    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.994010 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.993726    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.994010 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.993760    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.994010 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.993792    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.994284 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.994223    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.997078 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.997052    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.997204 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.997120    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.997204 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.997120    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:55.997678 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:55.997657    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:56.003268 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:56.003234    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh4g8\" (UniqueName: \"kubernetes.io/projected/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-kube-api-access-rh4g8\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:56.003754 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:56.003724    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wm95m\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:56.059963 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:56.059935    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:56.373368 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:56.373296    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"]
Sep 18 13:50:56.923304 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:50:56.923272    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b82cbc3_2770_4ebc_8ac4_a9ce04a10a3e.slice/crio-b18813d75c2ff939a962150b26f587fa2baf54787eab4c92465efa6b6bc4b531 WatchSource:0}: Error finding container b18813d75c2ff939a962150b26f587fa2baf54787eab4c92465efa6b6bc4b531: Status 404 returned error can't find the container with id b18813d75c2ff939a962150b26f587fa2baf54787eab4c92465efa6b6bc4b531
Sep 18 13:50:57.876896 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:57.876571    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt" event={"ID":"4144f489-44f8-4933-8d38-6c0fea51a78f","Type":"ContainerStarted","Data":"f67e56db6ffe1715f75af7eecbe785786afdf6c9e8dc72b7579882fc94a29920"}
Sep 18 13:50:57.878100 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:57.878072    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m" event={"ID":"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e","Type":"ContainerStarted","Data":"b18813d75c2ff939a962150b26f587fa2baf54787eab4c92465efa6b6bc4b531"}
Sep 18 13:50:57.896054 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:57.896001    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-849fd8447d-n2lrt" podStartSLOduration=2.418402018 podStartE2EDuration="4.895984605s" totalImagesPullingTime="2.477582587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:50:57.894356282 +0000 UTC m=+393.829321428" watchObservedRunningTime="2026-09-18 13:50:57.895984605 +0000 UTC m=+393.830949769"
Sep 18 13:50:59.377549 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:59.377491    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:50:59.377904 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:59.377602    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:50:59.890228 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:59.890191    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m" event={"ID":"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e","Type":"ContainerStarted","Data":"dfcde10e9ed3dcd70dc895cfc11bd8f68b716938fd66ba4b9645ce0341e27b19"}
Sep 18 13:50:59.890572 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:59.890523    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:50:59.916079 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:50:59.916017    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m" podStartSLOduration=2.464064797 podStartE2EDuration="4.915996226s" totalImagesPullingTime="2.451931429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:50:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:50:59.91286154 +0000 UTC m=+395.847826682" watchObservedRunningTime="2026-09-18 13:50:59.915996226 +0000 UTC m=+395.850961372"
Sep 18 13:51:00.896481 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:00.896454    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:51:10.450262 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:10.450225    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-db48ddbff-2c7w4"]
Sep 18 13:51:12.241822 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.241789    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"]
Sep 18 13:51:12.244895 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.244871    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.258337 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.258315    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"]
Sep 18 13:51:12.335833 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.335804    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c908b76c-afff-4af9-81bb-be1c7f8df54e-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.335959 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.335903    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c908b76c-afff-4af9-81bb-be1c7f8df54e-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.336014 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.335953    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c908b76c-afff-4af9-81bb-be1c7f8df54e-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.336089 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.336064    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs474\" (UniqueName: \"kubernetes.io/projected/c908b76c-afff-4af9-81bb-be1c7f8df54e-kube-api-access-vs474\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.336144 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.336120    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c908b76c-afff-4af9-81bb-be1c7f8df54e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.336197 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.336183    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c908b76c-afff-4af9-81bb-be1c7f8df54e-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.336289 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.336270    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c908b76c-afff-4af9-81bb-be1c7f8df54e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.436953 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.436915    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c908b76c-afff-4af9-81bb-be1c7f8df54e-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.437149 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.436960    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs474\" (UniqueName: \"kubernetes.io/projected/c908b76c-afff-4af9-81bb-be1c7f8df54e-kube-api-access-vs474\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.437149 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.436984    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c908b76c-afff-4af9-81bb-be1c7f8df54e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.437149 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.437014    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c908b76c-afff-4af9-81bb-be1c7f8df54e-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.437149 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.437043    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c908b76c-afff-4af9-81bb-be1c7f8df54e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.437149 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.437079    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c908b76c-afff-4af9-81bb-be1c7f8df54e-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.437149 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.437117    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c908b76c-afff-4af9-81bb-be1c7f8df54e-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.437933 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.437905    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c908b76c-afff-4af9-81bb-be1c7f8df54e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.439394 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.439375    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c908b76c-afff-4af9-81bb-be1c7f8df54e-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.439463 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.439441    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c908b76c-afff-4af9-81bb-be1c7f8df54e-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.439828 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.439801    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c908b76c-afff-4af9-81bb-be1c7f8df54e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.439912 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.439831    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c908b76c-afff-4af9-81bb-be1c7f8df54e-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.445889 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.445868    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs474\" (UniqueName: \"kubernetes.io/projected/c908b76c-afff-4af9-81bb-be1c7f8df54e-kube-api-access-vs474\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.446118 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.446095    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c908b76c-afff-4af9-81bb-be1c7f8df54e-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-mrcfl\" (UID: \"c908b76c-afff-4af9-81bb-be1c7f8df54e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.554163 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.554075    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.839957 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.839928    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"]
Sep 18 13:51:12.840730 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:51:12.840704    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc908b76c_afff_4af9_81bb_be1c7f8df54e.slice/crio-97d3b84d6bd30f63c1a97f6e5ef904cffdec5b9f8c2dc2442354ad6e26b06b73 WatchSource:0}: Error finding container 97d3b84d6bd30f63c1a97f6e5ef904cffdec5b9f8c2dc2442354ad6e26b06b73: Status 404 returned error can't find the container with id 97d3b84d6bd30f63c1a97f6e5ef904cffdec5b9f8c2dc2442354ad6e26b06b73
Sep 18 13:51:12.842807 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.842773    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:51:12.842879 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.842832    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 13:51:12.950954 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.950913    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl" event={"ID":"c908b76c-afff-4af9-81bb-be1c7f8df54e","Type":"ContainerStarted","Data":"9bfff6ced9cdc9578545507978920cf33b557854f8e6c41d5f3567d9e36049c3"}
Sep 18 13:51:12.950954 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.950962    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl" event={"ID":"c908b76c-afff-4af9-81bb-be1c7f8df54e","Type":"ContainerStarted","Data":"97d3b84d6bd30f63c1a97f6e5ef904cffdec5b9f8c2dc2442354ad6e26b06b73"}
Sep 18 13:51:12.951202 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.951000    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:12.972176 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:12.972121    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl" podStartSLOduration=0.972102127 podStartE2EDuration="972.102127ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:51:12.970597299 +0000 UTC m=+408.905562445" watchObservedRunningTime="2026-09-18 13:51:12.972102127 +0000 UTC m=+408.907067272"
Sep 18 13:51:13.958082 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:13.958055    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mrcfl"
Sep 18 13:51:14.018036 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.018004    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"]
Sep 18 13:51:14.018205 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.018185    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m" podUID="7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e" containerName="discovery" containerID="cri-o://dfcde10e9ed3dcd70dc895cfc11bd8f68b716938fd66ba4b9645ce0341e27b19" gracePeriod=30
Sep 18 13:51:14.252396 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.252371    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:51:14.352981 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.352952    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-csr-ca-configmap\") pod \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") "
Sep 18 13:51:14.353146 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.353016    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-local-certs\") pod \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") "
Sep 18 13:51:14.353146 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.353041    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh4g8\" (UniqueName: \"kubernetes.io/projected/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-kube-api-access-rh4g8\") pod \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") "
Sep 18 13:51:14.353146 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.353064    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-token\") pod \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") "
Sep 18 13:51:14.353146 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.353104    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-csr-dns-cert\") pod \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") "
Sep 18 13:51:14.353352 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.353153    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-kubeconfig\") pod \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") "
Sep 18 13:51:14.353352 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.353217    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-cacerts\") pod \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\" (UID: \"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e\") "
Sep 18 13:51:14.353444 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.353371    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e" (UID: "7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:14.353604 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.353577    2572 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-csr-ca-configmap\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:51:14.355566 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.355502    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-local-certs" (OuterVolumeSpecName: "local-certs") pod "7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e" (UID: "7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:14.355566 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.355504    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-kube-api-access-rh4g8" (OuterVolumeSpecName: "kube-api-access-rh4g8") pod "7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e" (UID: "7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e"). InnerVolumeSpecName "kube-api-access-rh4g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:14.355724 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.355598    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e" (UID: "7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:51:14.355724 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.355615    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e" (UID: "7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:14.355724 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.355632    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e" (UID: "7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:51:14.355899 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.355875    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-cacerts" (OuterVolumeSpecName: "cacerts") pod "7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e" (UID: "7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:51:14.454943 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.454910    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh4g8\" (UniqueName: \"kubernetes.io/projected/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-kube-api-access-rh4g8\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:51:14.454943 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.454940    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-token\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:51:14.455128 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.454953    2572 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-csr-dns-cert\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:51:14.455128 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.454964    2572 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-istio-kubeconfig\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:51:14.455128 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.454973    2572 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-cacerts\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:51:14.455128 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.454982    2572 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e-local-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:51:14.962690 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.962653    2572 generic.go:356] "Generic (PLEG): container finished" podID="7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e" containerID="dfcde10e9ed3dcd70dc895cfc11bd8f68b716938fd66ba4b9645ce0341e27b19" exitCode=0
Sep 18 13:51:14.963103 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.962714    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"
Sep 18 13:51:14.963103 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.962723    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m" event={"ID":"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e","Type":"ContainerDied","Data":"dfcde10e9ed3dcd70dc895cfc11bd8f68b716938fd66ba4b9645ce0341e27b19"}
Sep 18 13:51:14.963103 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.962777    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m" event={"ID":"7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e","Type":"ContainerDied","Data":"b18813d75c2ff939a962150b26f587fa2baf54787eab4c92465efa6b6bc4b531"}
Sep 18 13:51:14.963103 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.962803    2572 scope.go:117] "RemoveContainer" containerID="dfcde10e9ed3dcd70dc895cfc11bd8f68b716938fd66ba4b9645ce0341e27b19"
Sep 18 13:51:14.972289 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.972275    2572 scope.go:117] "RemoveContainer" containerID="dfcde10e9ed3dcd70dc895cfc11bd8f68b716938fd66ba4b9645ce0341e27b19"
Sep 18 13:51:14.972560 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:51:14.972522    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfcde10e9ed3dcd70dc895cfc11bd8f68b716938fd66ba4b9645ce0341e27b19\": container with ID starting with dfcde10e9ed3dcd70dc895cfc11bd8f68b716938fd66ba4b9645ce0341e27b19 not found: ID does not exist" containerID="dfcde10e9ed3dcd70dc895cfc11bd8f68b716938fd66ba4b9645ce0341e27b19"
Sep 18 13:51:14.972634 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.972567    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfcde10e9ed3dcd70dc895cfc11bd8f68b716938fd66ba4b9645ce0341e27b19"} err="failed to get container status \"dfcde10e9ed3dcd70dc895cfc11bd8f68b716938fd66ba4b9645ce0341e27b19\": rpc error: code = NotFound desc = could not find container \"dfcde10e9ed3dcd70dc895cfc11bd8f68b716938fd66ba4b9645ce0341e27b19\": container with ID starting with dfcde10e9ed3dcd70dc895cfc11bd8f68b716938fd66ba4b9645ce0341e27b19 not found: ID does not exist"
Sep 18 13:51:14.992960 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.992938    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"]
Sep 18 13:51:14.997574 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:14.997552    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wm95m"]
Sep 18 13:51:15.669076 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:15.669044    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl"]
Sep 18 13:51:15.669366 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:15.669354    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e" containerName="discovery"
Sep 18 13:51:15.669416 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:15.669367    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e" containerName="discovery"
Sep 18 13:51:15.669416 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:15.669414    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e" containerName="discovery"
Sep 18 13:51:15.673652 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:15.673633    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl"
Sep 18 13:51:15.676542 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:15.676506    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-s99ds\""
Sep 18 13:51:15.676669 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:15.676556    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 18 13:51:15.685431 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:15.685406    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl"]
Sep 18 13:51:15.768847 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:15.768817    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfrs7\" (UniqueName: \"kubernetes.io/projected/1d645000-9b99-4314-bcb1-bad1a334ec2b-kube-api-access-pfrs7\") pod \"llmisvc-controller-manager-777c6bd7d5-qktrl\" (UID: \"1d645000-9b99-4314-bcb1-bad1a334ec2b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl"
Sep 18 13:51:15.768979 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:15.768898    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d645000-9b99-4314-bcb1-bad1a334ec2b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qktrl\" (UID: \"1d645000-9b99-4314-bcb1-bad1a334ec2b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl"
Sep 18 13:51:15.869522 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:15.869488    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfrs7\" (UniqueName: \"kubernetes.io/projected/1d645000-9b99-4314-bcb1-bad1a334ec2b-kube-api-access-pfrs7\") pod \"llmisvc-controller-manager-777c6bd7d5-qktrl\" (UID: \"1d645000-9b99-4314-bcb1-bad1a334ec2b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl"
Sep 18 13:51:15.869706 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:15.869560    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d645000-9b99-4314-bcb1-bad1a334ec2b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qktrl\" (UID: \"1d645000-9b99-4314-bcb1-bad1a334ec2b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl"
Sep 18 13:51:15.869706 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:51:15.869644    2572 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 18 13:51:15.869706 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:51:15.869696    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d645000-9b99-4314-bcb1-bad1a334ec2b-cert podName:1d645000-9b99-4314-bcb1-bad1a334ec2b nodeName:}" failed. No retries permitted until 2026-09-18 13:51:16.369680617 +0000 UTC m=+412.304645739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1d645000-9b99-4314-bcb1-bad1a334ec2b-cert") pod "llmisvc-controller-manager-777c6bd7d5-qktrl" (UID: "1d645000-9b99-4314-bcb1-bad1a334ec2b") : secret "llmisvc-webhook-server-cert" not found
Sep 18 13:51:15.889773 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:15.889750    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfrs7\" (UniqueName: \"kubernetes.io/projected/1d645000-9b99-4314-bcb1-bad1a334ec2b-kube-api-access-pfrs7\") pod \"llmisvc-controller-manager-777c6bd7d5-qktrl\" (UID: \"1d645000-9b99-4314-bcb1-bad1a334ec2b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl"
Sep 18 13:51:16.196888 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:16.196860    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e" path="/var/lib/kubelet/pods/7b82cbc3-2770-4ebc-8ac4-a9ce04a10a3e/volumes"
Sep 18 13:51:16.372724 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:16.372694    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d645000-9b99-4314-bcb1-bad1a334ec2b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qktrl\" (UID: \"1d645000-9b99-4314-bcb1-bad1a334ec2b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl"
Sep 18 13:51:16.375080 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:16.375060    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d645000-9b99-4314-bcb1-bad1a334ec2b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qktrl\" (UID: \"1d645000-9b99-4314-bcb1-bad1a334ec2b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl"
Sep 18 13:51:16.585038 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:16.584967    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl"
Sep 18 13:51:16.858041 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:16.858016    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl"]
Sep 18 13:51:16.859333 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:51:16.859307    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1d645000_9b99_4314_bcb1_bad1a334ec2b.slice/crio-7d6d9beaca56d1c773ee6bce452422ef72bb32074ddb2087452b8470a7a45b06 WatchSource:0}: Error finding container 7d6d9beaca56d1c773ee6bce452422ef72bb32074ddb2087452b8470a7a45b06: Status 404 returned error can't find the container with id 7d6d9beaca56d1c773ee6bce452422ef72bb32074ddb2087452b8470a7a45b06
Sep 18 13:51:16.976236 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:16.976199    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl" event={"ID":"1d645000-9b99-4314-bcb1-bad1a334ec2b","Type":"ContainerStarted","Data":"7d6d9beaca56d1c773ee6bce452422ef72bb32074ddb2087452b8470a7a45b06"}
Sep 18 13:51:19.995380 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:19.995334    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl" event={"ID":"1d645000-9b99-4314-bcb1-bad1a334ec2b","Type":"ContainerStarted","Data":"67a3768731d8d8a47de223bded15182a6d0cee74063a7e4aaba1d3f72a0f4548"}
Sep 18 13:51:19.995823 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:19.995398    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl"
Sep 18 13:51:20.016712 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:20.016656    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl" podStartSLOduration=2.5419709900000003 podStartE2EDuration="5.016641918s" totalImagesPullingTime="2.474670928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:51:20.015156214 +0000 UTC m=+415.950121374" watchObservedRunningTime="2026-09-18 13:51:20.016641918 +0000 UTC m=+415.951607464"
Sep 18 13:51:35.469339 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.469277    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-db48ddbff-2c7w4" podUID="ec86a21e-2355-4be8-a6bc-4aad15bbf654" containerName="console" containerID="cri-o://07a85f71775e36873e5225f202107c1a383df9c8c59e579eb6f3edecedb75249" gracePeriod=15
Sep 18 13:51:35.710465 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.710441    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:51:35.817066 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.816993    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-serving-cert\") pod \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") "
Sep 18 13:51:35.817066 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.817036    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-oauth-serving-cert\") pod \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") "
Sep 18 13:51:35.817066 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.817054    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-service-ca\") pod \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") "
Sep 18 13:51:35.817331 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.817089    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-config\") pod \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") "
Sep 18 13:51:35.817331 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.817116    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-trusted-ca-bundle\") pod \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") "
Sep 18 13:51:35.817331 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.817146    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrdsg\" (UniqueName: \"kubernetes.io/projected/ec86a21e-2355-4be8-a6bc-4aad15bbf654-kube-api-access-qrdsg\") pod \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") "
Sep 18 13:51:35.817331 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.817166    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-oauth-config\") pod \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\" (UID: \"ec86a21e-2355-4be8-a6bc-4aad15bbf654\") "
Sep 18 13:51:35.817571 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.817468    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-service-ca" (OuterVolumeSpecName: "service-ca") pod "ec86a21e-2355-4be8-a6bc-4aad15bbf654" (UID: "ec86a21e-2355-4be8-a6bc-4aad15bbf654"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:35.817571 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.817473    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ec86a21e-2355-4be8-a6bc-4aad15bbf654" (UID: "ec86a21e-2355-4be8-a6bc-4aad15bbf654"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:35.817690 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.817652    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-config" (OuterVolumeSpecName: "console-config") pod "ec86a21e-2355-4be8-a6bc-4aad15bbf654" (UID: "ec86a21e-2355-4be8-a6bc-4aad15bbf654"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:35.817862 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.817837    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ec86a21e-2355-4be8-a6bc-4aad15bbf654" (UID: "ec86a21e-2355-4be8-a6bc-4aad15bbf654"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:35.819445 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.819424    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ec86a21e-2355-4be8-a6bc-4aad15bbf654" (UID: "ec86a21e-2355-4be8-a6bc-4aad15bbf654"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:51:35.819560 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.819476    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ec86a21e-2355-4be8-a6bc-4aad15bbf654" (UID: "ec86a21e-2355-4be8-a6bc-4aad15bbf654"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:51:35.819560 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.819494    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec86a21e-2355-4be8-a6bc-4aad15bbf654-kube-api-access-qrdsg" (OuterVolumeSpecName: "kube-api-access-qrdsg") pod "ec86a21e-2355-4be8-a6bc-4aad15bbf654" (UID: "ec86a21e-2355-4be8-a6bc-4aad15bbf654"). InnerVolumeSpecName "kube-api-access-qrdsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:35.918724 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.918695    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-trusted-ca-bundle\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:51:35.918724 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.918726    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrdsg\" (UniqueName: \"kubernetes.io/projected/ec86a21e-2355-4be8-a6bc-4aad15bbf654-kube-api-access-qrdsg\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:51:35.918913 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.918739    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-oauth-config\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:51:35.918913 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.918748    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-serving-cert\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:51:35.918913 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.918758    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-oauth-serving-cert\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:51:35.918913 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.918769    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:51:35.918913 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:35.918776    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec86a21e-2355-4be8-a6bc-4aad15bbf654-console-config\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:51:36.068368 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:36.068286    2572 generic.go:356] "Generic (PLEG): container finished" podID="ec86a21e-2355-4be8-a6bc-4aad15bbf654" containerID="07a85f71775e36873e5225f202107c1a383df9c8c59e579eb6f3edecedb75249" exitCode=0
Sep 18 13:51:36.068368 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:36.068328    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db48ddbff-2c7w4"
Sep 18 13:51:36.068553 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:36.068363    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db48ddbff-2c7w4" event={"ID":"ec86a21e-2355-4be8-a6bc-4aad15bbf654","Type":"ContainerDied","Data":"07a85f71775e36873e5225f202107c1a383df9c8c59e579eb6f3edecedb75249"}
Sep 18 13:51:36.068553 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:36.068413    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db48ddbff-2c7w4" event={"ID":"ec86a21e-2355-4be8-a6bc-4aad15bbf654","Type":"ContainerDied","Data":"9c243659bf34ded5488ddf948f7faf2cff724a61b14cb77f9c47e8d4589da048"}
Sep 18 13:51:36.068553 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:36.068436    2572 scope.go:117] "RemoveContainer" containerID="07a85f71775e36873e5225f202107c1a383df9c8c59e579eb6f3edecedb75249"
Sep 18 13:51:36.077438 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:36.077429    2572 scope.go:117] "RemoveContainer" containerID="07a85f71775e36873e5225f202107c1a383df9c8c59e579eb6f3edecedb75249"
Sep 18 13:51:36.077731 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:51:36.077710    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a85f71775e36873e5225f202107c1a383df9c8c59e579eb6f3edecedb75249\": container with ID starting with 07a85f71775e36873e5225f202107c1a383df9c8c59e579eb6f3edecedb75249 not found: ID does not exist" containerID="07a85f71775e36873e5225f202107c1a383df9c8c59e579eb6f3edecedb75249"
Sep 18 13:51:36.077825 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:36.077736    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a85f71775e36873e5225f202107c1a383df9c8c59e579eb6f3edecedb75249"} err="failed to get container status \"07a85f71775e36873e5225f202107c1a383df9c8c59e579eb6f3edecedb75249\": rpc error: code = NotFound desc = could not find container \"07a85f71775e36873e5225f202107c1a383df9c8c59e579eb6f3edecedb75249\": container with ID starting with 07a85f71775e36873e5225f202107c1a383df9c8c59e579eb6f3edecedb75249 not found: ID does not exist"
Sep 18 13:51:36.090931 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:36.090908    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-db48ddbff-2c7w4"]
Sep 18 13:51:36.095754 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:36.095731    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-db48ddbff-2c7w4"]
Sep 18 13:51:36.196560 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:36.196520    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec86a21e-2355-4be8-a6bc-4aad15bbf654" path="/var/lib/kubelet/pods/ec86a21e-2355-4be8-a6bc-4aad15bbf654/volumes"
Sep 18 13:51:51.001741 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:51:51.001655    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qktrl"
Sep 18 13:52:09.999786 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:09.999746    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft"]
Sep 18 13:52:10.000226 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.000086    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec86a21e-2355-4be8-a6bc-4aad15bbf654" containerName="console"
Sep 18 13:52:10.000226 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.000097    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec86a21e-2355-4be8-a6bc-4aad15bbf654" containerName="console"
Sep 18 13:52:10.000226 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.000146    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec86a21e-2355-4be8-a6bc-4aad15bbf654" containerName="console"
Sep 18 13:52:10.003251 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.003235    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft"
Sep 18 13:52:10.005851 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.005824    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 13:52:10.006415 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.006397    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4m6sb\""
Sep 18 13:52:10.006468 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.006438    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 13:52:10.012200 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.012175    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft"]
Sep 18 13:52:10.093927 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.093901    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn"]
Sep 18 13:52:10.095505 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.095483    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c326102-e2e1-42de-8d41-328756f86d48-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft\" (UID: \"1c326102-e2e1-42de-8d41-328756f86d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft"
Sep 18 13:52:10.095627 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.095516    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c326102-e2e1-42de-8d41-328756f86d48-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft\" (UID: \"1c326102-e2e1-42de-8d41-328756f86d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft"
Sep 18 13:52:10.095708 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.095680    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpbrn\" (UniqueName: \"kubernetes.io/projected/1c326102-e2e1-42de-8d41-328756f86d48-kube-api-access-gpbrn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft\" (UID: \"1c326102-e2e1-42de-8d41-328756f86d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft"
Sep 18 13:52:10.097075 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.097060    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn"
Sep 18 13:52:10.108542 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.108508    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn"]
Sep 18 13:52:10.196194 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.196171    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01669a56-1af6-4d6d-8bca-e7179f5a84cc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn\" (UID: \"01669a56-1af6-4d6d-8bca-e7179f5a84cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn"
Sep 18 13:52:10.196325 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.196201    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpbrn\" (UniqueName: \"kubernetes.io/projected/1c326102-e2e1-42de-8d41-328756f86d48-kube-api-access-gpbrn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft\" (UID: \"1c326102-e2e1-42de-8d41-328756f86d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft"
Sep 18 13:52:10.196325 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.196230    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9cx6\" (UniqueName: \"kubernetes.io/projected/01669a56-1af6-4d6d-8bca-e7179f5a84cc-kube-api-access-p9cx6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn\" (UID: \"01669a56-1af6-4d6d-8bca-e7179f5a84cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn"
Sep 18 13:52:10.196439 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.196339    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c326102-e2e1-42de-8d41-328756f86d48-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft\" (UID: \"1c326102-e2e1-42de-8d41-328756f86d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft"
Sep 18 13:52:10.196439 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.196370    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c326102-e2e1-42de-8d41-328756f86d48-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft\" (UID: \"1c326102-e2e1-42de-8d41-328756f86d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft"
Sep 18 13:52:10.196439 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.196429    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01669a56-1af6-4d6d-8bca-e7179f5a84cc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn\" (UID: \"01669a56-1af6-4d6d-8bca-e7179f5a84cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn"
Sep 18 13:52:10.196700 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.196685    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c326102-e2e1-42de-8d41-328756f86d48-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft\" (UID: \"1c326102-e2e1-42de-8d41-328756f86d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft"
Sep 18 13:52:10.196779 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.196761    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c326102-e2e1-42de-8d41-328756f86d48-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft\" (UID: \"1c326102-e2e1-42de-8d41-328756f86d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft"
Sep 18 13:52:10.207398 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.207379    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpbrn\" (UniqueName: \"kubernetes.io/projected/1c326102-e2e1-42de-8d41-328756f86d48-kube-api-access-gpbrn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft\" (UID: \"1c326102-e2e1-42de-8d41-328756f86d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft"
Sep 18 13:52:10.212727 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.212706    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj"]
Sep 18 13:52:10.216199 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.216183    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj"
Sep 18 13:52:10.224176 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.224150    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj"]
Sep 18 13:52:10.297607 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.297527    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9cx6\" (UniqueName: \"kubernetes.io/projected/01669a56-1af6-4d6d-8bca-e7179f5a84cc-kube-api-access-p9cx6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn\" (UID: \"01669a56-1af6-4d6d-8bca-e7179f5a84cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn"
Sep 18 13:52:10.297710 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.297645    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj\" (UID: \"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj"
Sep 18 13:52:10.297785 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.297773    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01669a56-1af6-4d6d-8bca-e7179f5a84cc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn\" (UID: \"01669a56-1af6-4d6d-8bca-e7179f5a84cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn"
Sep 18 13:52:10.297834 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.297823    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01669a56-1af6-4d6d-8bca-e7179f5a84cc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn\" (UID: \"01669a56-1af6-4d6d-8bca-e7179f5a84cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn"
Sep 18 13:52:10.297869 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.297843    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbbl2\" (UniqueName: \"kubernetes.io/projected/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-kube-api-access-gbbl2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj\" (UID: \"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj"
Sep 18 13:52:10.297913 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.297868    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj\" (UID: \"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj"
Sep 18 13:52:10.298133 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.298113    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01669a56-1af6-4d6d-8bca-e7179f5a84cc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn\" (UID: \"01669a56-1af6-4d6d-8bca-e7179f5a84cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn"
Sep 18 13:52:10.298214 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.298168    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01669a56-1af6-4d6d-8bca-e7179f5a84cc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn\" (UID: \"01669a56-1af6-4d6d-8bca-e7179f5a84cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn"
Sep 18 13:52:10.304226 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.304201    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs"]
Sep 18 13:52:10.307528 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.307513    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs"
Sep 18 13:52:10.311380 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.311356    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9cx6\" (UniqueName: \"kubernetes.io/projected/01669a56-1af6-4d6d-8bca-e7179f5a84cc-kube-api-access-p9cx6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn\" (UID: \"01669a56-1af6-4d6d-8bca-e7179f5a84cc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn"
Sep 18 13:52:10.313186 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.313170    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft"
Sep 18 13:52:10.323197 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.323175    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs"]
Sep 18 13:52:10.399024 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.398994    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa0ca067-924f-485f-980f-b5f14d6b6b2b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs\" (UID: \"fa0ca067-924f-485f-980f-b5f14d6b6b2b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs"
Sep 18 13:52:10.399182 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.399054    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj\" (UID: \"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj"
Sep 18 13:52:10.399182 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.399100    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rxq7\" (UniqueName: \"kubernetes.io/projected/fa0ca067-924f-485f-980f-b5f14d6b6b2b-kube-api-access-6rxq7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs\" (UID: \"fa0ca067-924f-485f-980f-b5f14d6b6b2b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs"
Sep 18 13:52:10.399182 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.399130    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa0ca067-924f-485f-980f-b5f14d6b6b2b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs\" (UID: \"fa0ca067-924f-485f-980f-b5f14d6b6b2b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs"
Sep 18 13:52:10.399347 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.399201    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbbl2\" (UniqueName: \"kubernetes.io/projected/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-kube-api-access-gbbl2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj\" (UID: \"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj"
Sep 18 13:52:10.399347 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.399238    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj\" (UID: \"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj"
Sep 18 13:52:10.399458 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.399422    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj\" (UID: \"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj"
Sep 18 13:52:10.399587 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.399568    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj\" (UID: \"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj"
Sep 18 13:52:10.405431 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.405415    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn"
Sep 18 13:52:10.407981 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.407960    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbbl2\" (UniqueName: \"kubernetes.io/projected/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-kube-api-access-gbbl2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj\" (UID: \"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj"
Sep 18 13:52:10.500157 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.500121    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rxq7\" (UniqueName: \"kubernetes.io/projected/fa0ca067-924f-485f-980f-b5f14d6b6b2b-kube-api-access-6rxq7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs\" (UID: \"fa0ca067-924f-485f-980f-b5f14d6b6b2b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs"
Sep 18 13:52:10.500157 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.500159    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa0ca067-924f-485f-980f-b5f14d6b6b2b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs\" (UID: \"fa0ca067-924f-485f-980f-b5f14d6b6b2b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs"
Sep 18 13:52:10.500386 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.500365    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa0ca067-924f-485f-980f-b5f14d6b6b2b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs\" (UID: \"fa0ca067-924f-485f-980f-b5f14d6b6b2b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs"
Sep 18 13:52:10.500513 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.500492    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa0ca067-924f-485f-980f-b5f14d6b6b2b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs\" (UID: \"fa0ca067-924f-485f-980f-b5f14d6b6b2b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs"
Sep 18 13:52:10.500661 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.500646    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa0ca067-924f-485f-980f-b5f14d6b6b2b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs\" (UID: \"fa0ca067-924f-485f-980f-b5f14d6b6b2b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs"
Sep 18 13:52:10.509112 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.509086    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rxq7\" (UniqueName: \"kubernetes.io/projected/fa0ca067-924f-485f-980f-b5f14d6b6b2b-kube-api-access-6rxq7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs\" (UID: \"fa0ca067-924f-485f-980f-b5f14d6b6b2b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs"
Sep 18 13:52:10.525985 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.525967    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj"
Sep 18 13:52:10.588552 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.588501    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft"]
Sep 18 13:52:10.590123 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:52:10.590090    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c326102_e2e1_42de_8d41_328756f86d48.slice/crio-71e50e643e4dcc610e3f75c5054c75d1590623176989307c2f8122a1db7b904b WatchSource:0}: Error finding container 71e50e643e4dcc610e3f75c5054c75d1590623176989307c2f8122a1db7b904b: Status 404 returned error can't find the container with id 71e50e643e4dcc610e3f75c5054c75d1590623176989307c2f8122a1db7b904b
Sep 18 13:52:10.623431 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.623410    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs"
Sep 18 13:52:10.683430 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.683149    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn"]
Sep 18 13:52:10.796240 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.796213    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj"]
Sep 18 13:52:10.798861 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:52:10.798839    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde32c7c1_ce5b_420e_a5bc_20d4d67f01ba.slice/crio-0f5de6f3255fc65764f25cb773e58e1e58bdc20ae2766444d17b0a50c919ee8f WatchSource:0}: Error finding container 0f5de6f3255fc65764f25cb773e58e1e58bdc20ae2766444d17b0a50c919ee8f: Status 404 returned error can't find the container with id 0f5de6f3255fc65764f25cb773e58e1e58bdc20ae2766444d17b0a50c919ee8f
Sep 18 13:52:10.898700 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:10.898678    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs"]
Sep 18 13:52:10.900438 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:52:10.900406    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0ca067_924f_485f_980f_b5f14d6b6b2b.slice/crio-885abdd9e508d93be40cae3c02cd383af6a36a8fe9f95381dcba963be863277b WatchSource:0}: Error finding container 885abdd9e508d93be40cae3c02cd383af6a36a8fe9f95381dcba963be863277b: Status 404 returned error can't find the container with id 885abdd9e508d93be40cae3c02cd383af6a36a8fe9f95381dcba963be863277b
Sep 18 13:52:11.222591 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:11.222561    2572 generic.go:356] "Generic (PLEG): container finished" podID="fa0ca067-924f-485f-980f-b5f14d6b6b2b" containerID="3855815299ec53d7d205905b34707768f147228569ed43c74284b8e94f55d147" exitCode=0
Sep 18 13:52:11.222945 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:11.222635    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs" event={"ID":"fa0ca067-924f-485f-980f-b5f14d6b6b2b","Type":"ContainerDied","Data":"3855815299ec53d7d205905b34707768f147228569ed43c74284b8e94f55d147"}
Sep 18 13:52:11.222945 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:11.222692    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs" event={"ID":"fa0ca067-924f-485f-980f-b5f14d6b6b2b","Type":"ContainerStarted","Data":"885abdd9e508d93be40cae3c02cd383af6a36a8fe9f95381dcba963be863277b"}
Sep 18 13:52:11.224152 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:11.224133    2572 generic.go:356] "Generic (PLEG): container finished" podID="01669a56-1af6-4d6d-8bca-e7179f5a84cc" containerID="8f5a7c699473e50df2f4476c9d9638251c032de4257c744a951763e13aaaf502" exitCode=0
Sep 18 13:52:11.224218 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:11.224208    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn" event={"ID":"01669a56-1af6-4d6d-8bca-e7179f5a84cc","Type":"ContainerDied","Data":"8f5a7c699473e50df2f4476c9d9638251c032de4257c744a951763e13aaaf502"}
Sep 18 13:52:11.224287 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:11.224263    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn" event={"ID":"01669a56-1af6-4d6d-8bca-e7179f5a84cc","Type":"ContainerStarted","Data":"77f27af638d98260c5740827ba1fe46694c95d63e98caa2951b39e3ca0cbcaeb"}
Sep 18 13:52:11.225816 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:11.225798    2572 generic.go:356] "Generic (PLEG): container finished" podID="de32c7c1-ce5b-420e-a5bc-20d4d67f01ba" containerID="84f26591e0c328aaec54fdc3c93050ffc130e74f7fc159076e29f168181f56c0" exitCode=0
Sep 18 13:52:11.225929 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:11.225874    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj" event={"ID":"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba","Type":"ContainerDied","Data":"84f26591e0c328aaec54fdc3c93050ffc130e74f7fc159076e29f168181f56c0"}
Sep 18 13:52:11.225929 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:11.225918    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj" event={"ID":"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba","Type":"ContainerStarted","Data":"0f5de6f3255fc65764f25cb773e58e1e58bdc20ae2766444d17b0a50c919ee8f"}
Sep 18 13:52:11.227724 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:11.227703    2572 generic.go:356] "Generic (PLEG): container finished" podID="1c326102-e2e1-42de-8d41-328756f86d48" containerID="5d244206acdfffd79391aa52bb6bc9fd217a1c99cb07211b25c129df07f0bf8a" exitCode=0
Sep 18 13:52:11.227809 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:11.227749    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft" event={"ID":"1c326102-e2e1-42de-8d41-328756f86d48","Type":"ContainerDied","Data":"5d244206acdfffd79391aa52bb6bc9fd217a1c99cb07211b25c129df07f0bf8a"}
Sep 18 13:52:11.227809 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:11.227784    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft" event={"ID":"1c326102-e2e1-42de-8d41-328756f86d48","Type":"ContainerStarted","Data":"71e50e643e4dcc610e3f75c5054c75d1590623176989307c2f8122a1db7b904b"}
Sep 18 13:52:12.234784 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:12.234752    2572 generic.go:356] "Generic (PLEG): container finished" podID="fa0ca067-924f-485f-980f-b5f14d6b6b2b" containerID="f02812187ec23c68b21ad6fa495e9cbdd42c9da4ff3408ade009388f95c9375e" exitCode=0
Sep 18 13:52:12.235134 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:12.234796    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs" event={"ID":"fa0ca067-924f-485f-980f-b5f14d6b6b2b","Type":"ContainerDied","Data":"f02812187ec23c68b21ad6fa495e9cbdd42c9da4ff3408ade009388f95c9375e"}
Sep 18 13:52:12.236879 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:12.236860    2572 generic.go:356] "Generic (PLEG): container finished" podID="01669a56-1af6-4d6d-8bca-e7179f5a84cc" containerID="a8c0d4afd1f70a83afda6a4ec443938b881a86c0c1ad08e8631f3f07542daa24" exitCode=0
Sep 18 13:52:12.236958 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:12.236939    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn" event={"ID":"01669a56-1af6-4d6d-8bca-e7179f5a84cc","Type":"ContainerDied","Data":"a8c0d4afd1f70a83afda6a4ec443938b881a86c0c1ad08e8631f3f07542daa24"}
Sep 18 13:52:12.238849 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:12.238832    2572 generic.go:356] "Generic (PLEG): container finished" podID="de32c7c1-ce5b-420e-a5bc-20d4d67f01ba" containerID="5e48aecdd4aacf76d54e19d5c0051af09a66c0cf738364d8f1103a43f84ea9a8" exitCode=0
Sep 18 13:52:12.238924 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:12.238884    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj" event={"ID":"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba","Type":"ContainerDied","Data":"5e48aecdd4aacf76d54e19d5c0051af09a66c0cf738364d8f1103a43f84ea9a8"}
Sep 18 13:52:12.240806 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:12.240785    2572 generic.go:356] "Generic (PLEG): container finished" podID="1c326102-e2e1-42de-8d41-328756f86d48" containerID="d131cefaaae88267b1fb92d35a7f62fe9c336c6bc215e9deef19437f897d14e5" exitCode=0
Sep 18 13:52:12.240899 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:12.240853    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft" event={"ID":"1c326102-e2e1-42de-8d41-328756f86d48","Type":"ContainerDied","Data":"d131cefaaae88267b1fb92d35a7f62fe9c336c6bc215e9deef19437f897d14e5"}
Sep 18 13:52:13.248320 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:13.248282    2572 generic.go:356] "Generic (PLEG): container finished" podID="fa0ca067-924f-485f-980f-b5f14d6b6b2b" containerID="c1ab00ade0561c163e550bfc2e1d4a584b5397a9e4755ba27ce8e8918b64757d" exitCode=0
Sep 18 13:52:13.248784 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:13.248353    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs" event={"ID":"fa0ca067-924f-485f-980f-b5f14d6b6b2b","Type":"ContainerDied","Data":"c1ab00ade0561c163e550bfc2e1d4a584b5397a9e4755ba27ce8e8918b64757d"}
Sep 18 13:52:13.250433 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:13.250402    2572 generic.go:356] "Generic (PLEG): container finished" podID="01669a56-1af6-4d6d-8bca-e7179f5a84cc" containerID="da3fff222c99cd8567188d5b94c01e0919c256802173a9518c455d495ec84ce0" exitCode=0
Sep 18 13:52:13.250571 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:13.250480    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn" event={"ID":"01669a56-1af6-4d6d-8bca-e7179f5a84cc","Type":"ContainerDied","Data":"da3fff222c99cd8567188d5b94c01e0919c256802173a9518c455d495ec84ce0"}
Sep 18 13:52:13.252617 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:13.252572    2572 generic.go:356] "Generic (PLEG): container finished" podID="de32c7c1-ce5b-420e-a5bc-20d4d67f01ba" containerID="1b2ae895bdafc25a1e10364b4e1e96a165a7942db9aa2367d7ec5ec80e27490e" exitCode=0
Sep 18 13:52:13.252739 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:13.252647    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj" event={"ID":"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba","Type":"ContainerDied","Data":"1b2ae895bdafc25a1e10364b4e1e96a165a7942db9aa2367d7ec5ec80e27490e"}
Sep 18 13:52:13.254591 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:13.254569    2572 generic.go:356] "Generic (PLEG): container finished" podID="1c326102-e2e1-42de-8d41-328756f86d48" containerID="cb36526d7474e812a0ff90d30ceddc5ba322975e7017b49d8da513a7945952b7" exitCode=0
Sep 18 13:52:13.254694 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:13.254645    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft" event={"ID":"1c326102-e2e1-42de-8d41-328756f86d48","Type":"ContainerDied","Data":"cb36526d7474e812a0ff90d30ceddc5ba322975e7017b49d8da513a7945952b7"}
Sep 18 13:52:14.401831 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.401807    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj"
Sep 18 13:52:14.434037 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.434013    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft"
Sep 18 13:52:14.484181 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.484153    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs"
Sep 18 13:52:14.487062 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.487042    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn"
Sep 18 13:52:14.535703 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.535634    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpbrn\" (UniqueName: \"kubernetes.io/projected/1c326102-e2e1-42de-8d41-328756f86d48-kube-api-access-gpbrn\") pod \"1c326102-e2e1-42de-8d41-328756f86d48\" (UID: \"1c326102-e2e1-42de-8d41-328756f86d48\") "
Sep 18 13:52:14.535703 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.535676    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-bundle\") pod \"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba\" (UID: \"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba\") "
Sep 18 13:52:14.535906 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.535711    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c326102-e2e1-42de-8d41-328756f86d48-bundle\") pod \"1c326102-e2e1-42de-8d41-328756f86d48\" (UID: \"1c326102-e2e1-42de-8d41-328756f86d48\") "
Sep 18 13:52:14.535906 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.535782    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbbl2\" (UniqueName: \"kubernetes.io/projected/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-kube-api-access-gbbl2\") pod \"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba\" (UID: \"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba\") "
Sep 18 13:52:14.535906 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.535820    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c326102-e2e1-42de-8d41-328756f86d48-util\") pod \"1c326102-e2e1-42de-8d41-328756f86d48\" (UID: \"1c326102-e2e1-42de-8d41-328756f86d48\") "
Sep 18 13:52:14.535906 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.535850    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-util\") pod \"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba\" (UID: \"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba\") "
Sep 18 13:52:14.536362 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.536333    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c326102-e2e1-42de-8d41-328756f86d48-bundle" (OuterVolumeSpecName: "bundle") pod "1c326102-e2e1-42de-8d41-328756f86d48" (UID: "1c326102-e2e1-42de-8d41-328756f86d48"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:14.536517 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.536485    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-bundle" (OuterVolumeSpecName: "bundle") pod "de32c7c1-ce5b-420e-a5bc-20d4d67f01ba" (UID: "de32c7c1-ce5b-420e-a5bc-20d4d67f01ba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:14.537896 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.537873    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c326102-e2e1-42de-8d41-328756f86d48-kube-api-access-gpbrn" (OuterVolumeSpecName: "kube-api-access-gpbrn") pod "1c326102-e2e1-42de-8d41-328756f86d48" (UID: "1c326102-e2e1-42de-8d41-328756f86d48"). InnerVolumeSpecName "kube-api-access-gpbrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:52:14.538086 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.538064    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-kube-api-access-gbbl2" (OuterVolumeSpecName: "kube-api-access-gbbl2") pod "de32c7c1-ce5b-420e-a5bc-20d4d67f01ba" (UID: "de32c7c1-ce5b-420e-a5bc-20d4d67f01ba"). InnerVolumeSpecName "kube-api-access-gbbl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:52:14.543953 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.543928    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-util" (OuterVolumeSpecName: "util") pod "de32c7c1-ce5b-420e-a5bc-20d4d67f01ba" (UID: "de32c7c1-ce5b-420e-a5bc-20d4d67f01ba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:14.544551 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.544518    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c326102-e2e1-42de-8d41-328756f86d48-util" (OuterVolumeSpecName: "util") pod "1c326102-e2e1-42de-8d41-328756f86d48" (UID: "1c326102-e2e1-42de-8d41-328756f86d48"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:14.636917 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.636896    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa0ca067-924f-485f-980f-b5f14d6b6b2b-bundle\") pod \"fa0ca067-924f-485f-980f-b5f14d6b6b2b\" (UID: \"fa0ca067-924f-485f-980f-b5f14d6b6b2b\") "
Sep 18 13:52:14.637010 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.636941    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9cx6\" (UniqueName: \"kubernetes.io/projected/01669a56-1af6-4d6d-8bca-e7179f5a84cc-kube-api-access-p9cx6\") pod \"01669a56-1af6-4d6d-8bca-e7179f5a84cc\" (UID: \"01669a56-1af6-4d6d-8bca-e7179f5a84cc\") "
Sep 18 13:52:14.637010 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.636965    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rxq7\" (UniqueName: \"kubernetes.io/projected/fa0ca067-924f-485f-980f-b5f14d6b6b2b-kube-api-access-6rxq7\") pod \"fa0ca067-924f-485f-980f-b5f14d6b6b2b\" (UID: \"fa0ca067-924f-485f-980f-b5f14d6b6b2b\") "
Sep 18 13:52:14.637010 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.636991    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01669a56-1af6-4d6d-8bca-e7179f5a84cc-bundle\") pod \"01669a56-1af6-4d6d-8bca-e7179f5a84cc\" (UID: \"01669a56-1af6-4d6d-8bca-e7179f5a84cc\") "
Sep 18 13:52:14.637174 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.637020    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa0ca067-924f-485f-980f-b5f14d6b6b2b-util\") pod \"fa0ca067-924f-485f-980f-b5f14d6b6b2b\" (UID: \"fa0ca067-924f-485f-980f-b5f14d6b6b2b\") "
Sep 18 13:52:14.637174 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.637038    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01669a56-1af6-4d6d-8bca-e7179f5a84cc-util\") pod \"01669a56-1af6-4d6d-8bca-e7179f5a84cc\" (UID: \"01669a56-1af6-4d6d-8bca-e7179f5a84cc\") "
Sep 18 13:52:14.637285 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.637275    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpbrn\" (UniqueName: \"kubernetes.io/projected/1c326102-e2e1-42de-8d41-328756f86d48-kube-api-access-gpbrn\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:52:14.637339 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.637293    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-bundle\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:52:14.637339 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.637306    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c326102-e2e1-42de-8d41-328756f86d48-bundle\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:52:14.637339 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.637318    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbbl2\" (UniqueName: \"kubernetes.io/projected/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-kube-api-access-gbbl2\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:52:14.637339 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.637330    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c326102-e2e1-42de-8d41-328756f86d48-util\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:52:14.637643 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.637343    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de32c7c1-ce5b-420e-a5bc-20d4d67f01ba-util\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:52:14.637643 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.637517    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0ca067-924f-485f-980f-b5f14d6b6b2b-bundle" (OuterVolumeSpecName: "bundle") pod "fa0ca067-924f-485f-980f-b5f14d6b6b2b" (UID: "fa0ca067-924f-485f-980f-b5f14d6b6b2b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:14.637826 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.637800    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01669a56-1af6-4d6d-8bca-e7179f5a84cc-bundle" (OuterVolumeSpecName: "bundle") pod "01669a56-1af6-4d6d-8bca-e7179f5a84cc" (UID: "01669a56-1af6-4d6d-8bca-e7179f5a84cc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:14.639035 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.639008    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0ca067-924f-485f-980f-b5f14d6b6b2b-kube-api-access-6rxq7" (OuterVolumeSpecName: "kube-api-access-6rxq7") pod "fa0ca067-924f-485f-980f-b5f14d6b6b2b" (UID: "fa0ca067-924f-485f-980f-b5f14d6b6b2b"). InnerVolumeSpecName "kube-api-access-6rxq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:52:14.639381 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.639362    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01669a56-1af6-4d6d-8bca-e7179f5a84cc-kube-api-access-p9cx6" (OuterVolumeSpecName: "kube-api-access-p9cx6") pod "01669a56-1af6-4d6d-8bca-e7179f5a84cc" (UID: "01669a56-1af6-4d6d-8bca-e7179f5a84cc"). InnerVolumeSpecName "kube-api-access-p9cx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:52:14.643231 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.643201    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01669a56-1af6-4d6d-8bca-e7179f5a84cc-util" (OuterVolumeSpecName: "util") pod "01669a56-1af6-4d6d-8bca-e7179f5a84cc" (UID: "01669a56-1af6-4d6d-8bca-e7179f5a84cc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:14.643606 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.643586    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0ca067-924f-485f-980f-b5f14d6b6b2b-util" (OuterVolumeSpecName: "util") pod "fa0ca067-924f-485f-980f-b5f14d6b6b2b" (UID: "fa0ca067-924f-485f-980f-b5f14d6b6b2b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:14.737969 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.737940    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa0ca067-924f-485f-980f-b5f14d6b6b2b-bundle\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:52:14.737969 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.737965    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9cx6\" (UniqueName: \"kubernetes.io/projected/01669a56-1af6-4d6d-8bca-e7179f5a84cc-kube-api-access-p9cx6\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:52:14.738124 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.737976    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rxq7\" (UniqueName: \"kubernetes.io/projected/fa0ca067-924f-485f-980f-b5f14d6b6b2b-kube-api-access-6rxq7\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:52:14.738124 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.737992    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01669a56-1af6-4d6d-8bca-e7179f5a84cc-bundle\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:52:14.738124 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.738000    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa0ca067-924f-485f-980f-b5f14d6b6b2b-util\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:52:14.738124 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:14.738008    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01669a56-1af6-4d6d-8bca-e7179f5a84cc-util\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:52:15.267065 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:15.267028    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs" event={"ID":"fa0ca067-924f-485f-980f-b5f14d6b6b2b","Type":"ContainerDied","Data":"885abdd9e508d93be40cae3c02cd383af6a36a8fe9f95381dcba963be863277b"}
Sep 18 13:52:15.267241 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:15.267106    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="885abdd9e508d93be40cae3c02cd383af6a36a8fe9f95381dcba963be863277b"
Sep 18 13:52:15.267241 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:15.267044    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bmxxs"
Sep 18 13:52:15.269009 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:15.268984    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn"
Sep 18 13:52:15.269123 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:15.268986    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7wxcn" event={"ID":"01669a56-1af6-4d6d-8bca-e7179f5a84cc","Type":"ContainerDied","Data":"77f27af638d98260c5740827ba1fe46694c95d63e98caa2951b39e3ca0cbcaeb"}
Sep 18 13:52:15.269123 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:15.269100    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77f27af638d98260c5740827ba1fe46694c95d63e98caa2951b39e3ca0cbcaeb"
Sep 18 13:52:15.270901 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:15.270882    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj"
Sep 18 13:52:15.270998 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:15.270887    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lvtj" event={"ID":"de32c7c1-ce5b-420e-a5bc-20d4d67f01ba","Type":"ContainerDied","Data":"0f5de6f3255fc65764f25cb773e58e1e58bdc20ae2766444d17b0a50c919ee8f"}
Sep 18 13:52:15.271060 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:15.271044    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f5de6f3255fc65764f25cb773e58e1e58bdc20ae2766444d17b0a50c919ee8f"
Sep 18 13:52:15.273023 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:15.273001    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft" event={"ID":"1c326102-e2e1-42de-8d41-328756f86d48","Type":"ContainerDied","Data":"71e50e643e4dcc610e3f75c5054c75d1590623176989307c2f8122a1db7b904b"}
Sep 18 13:52:15.273105 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:15.273027    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q4fft"
Sep 18 13:52:15.273105 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:15.273039    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71e50e643e4dcc610e3f75c5054c75d1590623176989307c2f8122a1db7b904b"
Sep 18 13:52:25.716373 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716335    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-c5zpq"]
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716668    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0ca067-924f-485f-980f-b5f14d6b6b2b" containerName="util"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716679    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0ca067-924f-485f-980f-b5f14d6b6b2b" containerName="util"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716688    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01669a56-1af6-4d6d-8bca-e7179f5a84cc" containerName="pull"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716694    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="01669a56-1af6-4d6d-8bca-e7179f5a84cc" containerName="pull"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716701    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c326102-e2e1-42de-8d41-328756f86d48" containerName="util"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716706    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c326102-e2e1-42de-8d41-328756f86d48" containerName="util"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716712    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01669a56-1af6-4d6d-8bca-e7179f5a84cc" containerName="util"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716717    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="01669a56-1af6-4d6d-8bca-e7179f5a84cc" containerName="util"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716722    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c326102-e2e1-42de-8d41-328756f86d48" containerName="pull"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716727    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c326102-e2e1-42de-8d41-328756f86d48" containerName="pull"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716737    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0ca067-924f-485f-980f-b5f14d6b6b2b" containerName="pull"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716742    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0ca067-924f-485f-980f-b5f14d6b6b2b" containerName="pull"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716747    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01669a56-1af6-4d6d-8bca-e7179f5a84cc" containerName="extract"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716752    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="01669a56-1af6-4d6d-8bca-e7179f5a84cc" containerName="extract"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716760    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0ca067-924f-485f-980f-b5f14d6b6b2b" containerName="extract"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716766    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0ca067-924f-485f-980f-b5f14d6b6b2b" containerName="extract"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716771    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de32c7c1-ce5b-420e-a5bc-20d4d67f01ba" containerName="pull"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716775    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="de32c7c1-ce5b-420e-a5bc-20d4d67f01ba" containerName="pull"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716782    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c326102-e2e1-42de-8d41-328756f86d48" containerName="extract"
Sep 18 13:52:25.716776 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716788    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c326102-e2e1-42de-8d41-328756f86d48" containerName="extract"
Sep 18 13:52:25.717376 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716795    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de32c7c1-ce5b-420e-a5bc-20d4d67f01ba" containerName="extract"
Sep 18 13:52:25.717376 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716800    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="de32c7c1-ce5b-420e-a5bc-20d4d67f01ba" containerName="extract"
Sep 18 13:52:25.717376 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716807    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de32c7c1-ce5b-420e-a5bc-20d4d67f01ba" containerName="util"
Sep 18 13:52:25.717376 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716813    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="de32c7c1-ce5b-420e-a5bc-20d4d67f01ba" containerName="util"
Sep 18 13:52:25.717376 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716857    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="01669a56-1af6-4d6d-8bca-e7179f5a84cc" containerName="extract"
Sep 18 13:52:25.717376 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716867    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="de32c7c1-ce5b-420e-a5bc-20d4d67f01ba" containerName="extract"
Sep 18 13:52:25.717376 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716874    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c326102-e2e1-42de-8d41-328756f86d48" containerName="extract"
Sep 18 13:52:25.717376 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.716879    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0ca067-924f-485f-980f-b5f14d6b6b2b" containerName="extract"
Sep 18 13:52:25.720876 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.720859    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-c5zpq"
Sep 18 13:52:25.724517 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.724474    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 13:52:25.724517 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.724475    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-bs9g5\""
Sep 18 13:52:25.724715 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.724659    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 13:52:25.726284 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.726257    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thbhl\" (UniqueName: \"kubernetes.io/projected/2e44db4b-0b3c-46af-9d8b-2657f8abad67-kube-api-access-thbhl\") pod \"authorino-operator-76b478bd78-c5zpq\" (UID: \"2e44db4b-0b3c-46af-9d8b-2657f8abad67\") " pod="kuadrant-system/authorino-operator-76b478bd78-c5zpq"
Sep 18 13:52:25.733982 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.733962    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-c5zpq"]
Sep 18 13:52:25.827469 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.827436    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thbhl\" (UniqueName: \"kubernetes.io/projected/2e44db4b-0b3c-46af-9d8b-2657f8abad67-kube-api-access-thbhl\") pod \"authorino-operator-76b478bd78-c5zpq\" (UID: \"2e44db4b-0b3c-46af-9d8b-2657f8abad67\") " pod="kuadrant-system/authorino-operator-76b478bd78-c5zpq"
Sep 18 13:52:25.857246 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:25.857219    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thbhl\" (UniqueName: \"kubernetes.io/projected/2e44db4b-0b3c-46af-9d8b-2657f8abad67-kube-api-access-thbhl\") pod \"authorino-operator-76b478bd78-c5zpq\" (UID: \"2e44db4b-0b3c-46af-9d8b-2657f8abad67\") " pod="kuadrant-system/authorino-operator-76b478bd78-c5zpq"
Sep 18 13:52:26.030429 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:26.030335    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-c5zpq"
Sep 18 13:52:26.311290 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:26.311261    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-c5zpq"]
Sep 18 13:52:26.311593 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:52:26.311571    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e44db4b_0b3c_46af_9d8b_2657f8abad67.slice/crio-0b029024fed47fe3359af23d87e079921b8d5ced6982e7fd439dd4211a376ade WatchSource:0}: Error finding container 0b029024fed47fe3359af23d87e079921b8d5ced6982e7fd439dd4211a376ade: Status 404 returned error can't find the container with id 0b029024fed47fe3359af23d87e079921b8d5ced6982e7fd439dd4211a376ade
Sep 18 13:52:26.331410 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:26.331385    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-c5zpq" event={"ID":"2e44db4b-0b3c-46af-9d8b-2657f8abad67","Type":"ContainerStarted","Data":"0b029024fed47fe3359af23d87e079921b8d5ced6982e7fd439dd4211a376ade"}
Sep 18 13:52:29.348442 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:29.348404    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-c5zpq" event={"ID":"2e44db4b-0b3c-46af-9d8b-2657f8abad67","Type":"ContainerStarted","Data":"39611607f3993832976d72e1c17a1e323d8c317ab1ff3e6eb3b614906e066c3d"}
Sep 18 13:52:29.348816 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:29.348513    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-c5zpq"
Sep 18 13:52:29.368154 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:29.368111    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-c5zpq" podStartSLOduration=2.040929655 podStartE2EDuration="4.368098721s" totalImagesPullingTime="2.327169066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:52:29.36706961 +0000 UTC m=+485.302034756" watchObservedRunningTime="2026-09-18 13:52:29.368098721 +0000 UTC m=+485.303063867"
Sep 18 13:52:30.139130 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.139096    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-984kr"]
Sep 18 13:52:30.142485 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.142465    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-984kr"
Sep 18 13:52:30.145124 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.145105    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-gzsmh\""
Sep 18 13:52:30.145262 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.145241    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 18 13:52:30.154377 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.154352    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-984kr"]
Sep 18 13:52:30.158376 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.158355    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpxgn\" (UniqueName: \"kubernetes.io/projected/057ec4ca-bd80-4f90-a33d-6ab3dc34f6f1-kube-api-access-xpxgn\") pod \"dns-operator-controller-manager-7b84cd7c9c-984kr\" (UID: \"057ec4ca-bd80-4f90-a33d-6ab3dc34f6f1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-984kr"
Sep 18 13:52:30.259651 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.259624    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpxgn\" (UniqueName: \"kubernetes.io/projected/057ec4ca-bd80-4f90-a33d-6ab3dc34f6f1-kube-api-access-xpxgn\") pod \"dns-operator-controller-manager-7b84cd7c9c-984kr\" (UID: \"057ec4ca-bd80-4f90-a33d-6ab3dc34f6f1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-984kr"
Sep 18 13:52:30.269990 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.269962    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpxgn\" (UniqueName: \"kubernetes.io/projected/057ec4ca-bd80-4f90-a33d-6ab3dc34f6f1-kube-api-access-xpxgn\") pod \"dns-operator-controller-manager-7b84cd7c9c-984kr\" (UID: \"057ec4ca-bd80-4f90-a33d-6ab3dc34f6f1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-984kr"
Sep 18 13:52:30.452779 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.452751    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-984kr"
Sep 18 13:52:30.725954 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.725930    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-984kr"]
Sep 18 13:52:30.727635 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:52:30.727606    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057ec4ca_bd80_4f90_a33d_6ab3dc34f6f1.slice/crio-da4097df078f64f1200833f0b9790a10e1e1fc87a8d35ddeed70818942557407 WatchSource:0}: Error finding container da4097df078f64f1200833f0b9790a10e1e1fc87a8d35ddeed70818942557407: Status 404 returned error can't find the container with id da4097df078f64f1200833f0b9790a10e1e1fc87a8d35ddeed70818942557407
Sep 18 13:52:30.890594 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.890555    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24"]
Sep 18 13:52:30.895079 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.895056    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24"
Sep 18 13:52:30.897528 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.897506    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 18 13:52:30.897651 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.897522    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tqd2r\""
Sep 18 13:52:30.897744 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.897726    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 18 13:52:30.903990 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.903966    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24"]
Sep 18 13:52:30.963943 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.963904    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhbnr\" (UniqueName: \"kubernetes.io/projected/3f9ff756-b015-4591-aa32-dbcc4a797eca-kube-api-access-fhbnr\") pod \"kuadrant-console-plugin-fbdd4f445-xnd24\" (UID: \"3f9ff756-b015-4591-aa32-dbcc4a797eca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24"
Sep 18 13:52:30.963943 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.963940    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f9ff756-b015-4591-aa32-dbcc4a797eca-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xnd24\" (UID: \"3f9ff756-b015-4591-aa32-dbcc4a797eca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24"
Sep 18 13:52:30.964141 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:30.963963    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3f9ff756-b015-4591-aa32-dbcc4a797eca-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xnd24\" (UID: \"3f9ff756-b015-4591-aa32-dbcc4a797eca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24"
Sep 18 13:52:31.064553 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:31.064456    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhbnr\" (UniqueName: \"kubernetes.io/projected/3f9ff756-b015-4591-aa32-dbcc4a797eca-kube-api-access-fhbnr\") pod \"kuadrant-console-plugin-fbdd4f445-xnd24\" (UID: \"3f9ff756-b015-4591-aa32-dbcc4a797eca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24"
Sep 18 13:52:31.064553 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:31.064489    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f9ff756-b015-4591-aa32-dbcc4a797eca-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xnd24\" (UID: \"3f9ff756-b015-4591-aa32-dbcc4a797eca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24"
Sep 18 13:52:31.064553 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:31.064514    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3f9ff756-b015-4591-aa32-dbcc4a797eca-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xnd24\" (UID: \"3f9ff756-b015-4591-aa32-dbcc4a797eca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24"
Sep 18 13:52:31.064739 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:52:31.064641    2572 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 18 13:52:31.064739 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:52:31.064720    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f9ff756-b015-4591-aa32-dbcc4a797eca-plugin-serving-cert podName:3f9ff756-b015-4591-aa32-dbcc4a797eca nodeName:}" failed. No retries permitted until 2026-09-18 13:52:31.564697851 +0000 UTC m=+487.499662985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/3f9ff756-b015-4591-aa32-dbcc4a797eca-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-xnd24" (UID: "3f9ff756-b015-4591-aa32-dbcc4a797eca") : secret "plugin-serving-cert" not found
Sep 18 13:52:31.065287 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:31.065269    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3f9ff756-b015-4591-aa32-dbcc4a797eca-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xnd24\" (UID: \"3f9ff756-b015-4591-aa32-dbcc4a797eca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24"
Sep 18 13:52:31.073928 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:31.073901    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhbnr\" (UniqueName: \"kubernetes.io/projected/3f9ff756-b015-4591-aa32-dbcc4a797eca-kube-api-access-fhbnr\") pod \"kuadrant-console-plugin-fbdd4f445-xnd24\" (UID: \"3f9ff756-b015-4591-aa32-dbcc4a797eca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24"
Sep 18 13:52:31.360736 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:31.360659    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-984kr" event={"ID":"057ec4ca-bd80-4f90-a33d-6ab3dc34f6f1","Type":"ContainerStarted","Data":"da4097df078f64f1200833f0b9790a10e1e1fc87a8d35ddeed70818942557407"}
Sep 18 13:52:31.568472 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:31.568429    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f9ff756-b015-4591-aa32-dbcc4a797eca-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xnd24\" (UID: \"3f9ff756-b015-4591-aa32-dbcc4a797eca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24"
Sep 18 13:52:31.571306 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:31.571262    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f9ff756-b015-4591-aa32-dbcc4a797eca-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xnd24\" (UID: \"3f9ff756-b015-4591-aa32-dbcc4a797eca\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24"
Sep 18 13:52:31.804723 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:31.804696    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24"
Sep 18 13:52:32.367482 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:32.367453    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-984kr" event={"ID":"057ec4ca-bd80-4f90-a33d-6ab3dc34f6f1","Type":"ContainerStarted","Data":"4e4d7f90ca2cd37aa0105655eeb419fa4cac867878a95731492a7351bbf3784d"}
Sep 18 13:52:32.367676 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:32.367578    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-984kr"
Sep 18 13:52:32.379061 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:32.379041    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24"]
Sep 18 13:52:32.380690 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:52:32.380666    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f9ff756_b015_4591_aa32_dbcc4a797eca.slice/crio-b5991d558f8697ded98a886684c05fa76f3d95972580ae7dca4b6f95024a45a1 WatchSource:0}: Error finding container b5991d558f8697ded98a886684c05fa76f3d95972580ae7dca4b6f95024a45a1: Status 404 returned error can't find the container with id b5991d558f8697ded98a886684c05fa76f3d95972580ae7dca4b6f95024a45a1
Sep 18 13:52:32.384885 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:32.384836    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-984kr" podStartSLOduration=0.930022366 podStartE2EDuration="2.384817209s" totalImagesPullingTime="1.454794843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:52:32.384093539 +0000 UTC m=+488.319058685" watchObservedRunningTime="2026-09-18 13:52:32.384817209 +0000 UTC m=+488.319782355"
Sep 18 13:52:33.378110 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:33.377711    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24" event={"ID":"3f9ff756-b015-4591-aa32-dbcc4a797eca","Type":"ContainerStarted","Data":"b5991d558f8697ded98a886684c05fa76f3d95972580ae7dca4b6f95024a45a1"}
Sep 18 13:52:37.401909 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:37.401873    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24" event={"ID":"3f9ff756-b015-4591-aa32-dbcc4a797eca","Type":"ContainerStarted","Data":"0f8a3a24da8e696b5e7b7bfc638e8dabc3af5c60d1ad754161ae71e600fc70a2"}
Sep 18 13:52:37.418455 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:37.418406    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xnd24" podStartSLOduration=3.007616996 podStartE2EDuration="7.418390688s" totalImagesPullingTime="4.410773692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:52:37.417434693 +0000 UTC m=+493.352399841" watchObservedRunningTime="2026-09-18 13:52:37.418390688 +0000 UTC m=+493.353355832"
Sep 18 13:52:40.355682 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:40.355653    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-c5zpq"
Sep 18 13:52:42.025450 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:42.025414    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"]
Sep 18 13:52:42.029036 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:42.029014    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"
Sep 18 13:52:42.032220 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:42.032202    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r7g84\""
Sep 18 13:52:42.053277 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:42.053251    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"]
Sep 18 13:52:42.060898 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:42.060872    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2jmb\" (UniqueName: \"kubernetes.io/projected/7b8d50ee-46b2-4558-8de2-63491740f19e-kube-api-access-c2jmb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n9pw5\" (UID: \"7b8d50ee-46b2-4558-8de2-63491740f19e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"
Sep 18 13:52:42.061034 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:42.060930    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b8d50ee-46b2-4558-8de2-63491740f19e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n9pw5\" (UID: \"7b8d50ee-46b2-4558-8de2-63491740f19e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"
Sep 18 13:52:42.161872 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:42.161839    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2jmb\" (UniqueName: \"kubernetes.io/projected/7b8d50ee-46b2-4558-8de2-63491740f19e-kube-api-access-c2jmb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n9pw5\" (UID: \"7b8d50ee-46b2-4558-8de2-63491740f19e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"
Sep 18 13:52:42.162009 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:42.161908    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b8d50ee-46b2-4558-8de2-63491740f19e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n9pw5\" (UID: \"7b8d50ee-46b2-4558-8de2-63491740f19e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"
Sep 18 13:52:42.162300 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:42.162280    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b8d50ee-46b2-4558-8de2-63491740f19e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n9pw5\" (UID: \"7b8d50ee-46b2-4558-8de2-63491740f19e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"
Sep 18 13:52:42.173391 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:42.173367    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2jmb\" (UniqueName: \"kubernetes.io/projected/7b8d50ee-46b2-4558-8de2-63491740f19e-kube-api-access-c2jmb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-n9pw5\" (UID: \"7b8d50ee-46b2-4558-8de2-63491740f19e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"
Sep 18 13:52:42.339556 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:42.339471    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"
Sep 18 13:52:42.617498 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:42.617463    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"]
Sep 18 13:52:42.619226 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:52:42.619199    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8d50ee_46b2_4558_8de2_63491740f19e.slice/crio-801f7356e575813153b84026aba887527639d81c659c45fff77ce336f9f729c6 WatchSource:0}: Error finding container 801f7356e575813153b84026aba887527639d81c659c45fff77ce336f9f729c6: Status 404 returned error can't find the container with id 801f7356e575813153b84026aba887527639d81c659c45fff77ce336f9f729c6
Sep 18 13:52:43.380250 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:43.380219    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-984kr"
Sep 18 13:52:43.434648 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:43.434612    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5" event={"ID":"7b8d50ee-46b2-4558-8de2-63491740f19e","Type":"ContainerStarted","Data":"801f7356e575813153b84026aba887527639d81c659c45fff77ce336f9f729c6"}
Sep 18 13:52:46.454957 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:46.454918    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5" event={"ID":"7b8d50ee-46b2-4558-8de2-63491740f19e","Type":"ContainerStarted","Data":"63f31dca88b59dca2bcb24520693f339649b58cf0f3e7ededdaf27b6a262992c"}
Sep 18 13:52:46.455329 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:46.454988    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"
Sep 18 13:52:46.488212 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:46.488164    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5" podStartSLOduration=0.816154743 podStartE2EDuration="4.488149909s" totalImagesPullingTime="3.671995166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:52:46.485671885 +0000 UTC m=+502.420637030" watchObservedRunningTime="2026-09-18 13:52:46.488149909 +0000 UTC m=+502.423115054"
Sep 18 13:52:57.462357 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:52:57.462327    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"
Sep 18 13:53:20.279513 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:20.279442    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-w8lhc"]
Sep 18 13:53:20.283263 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:20.283241    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-w8lhc"
Sep 18 13:53:20.285268 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:20.285252    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-87ggk\""
Sep 18 13:53:20.289285 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:20.289261    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-w8lhc"]
Sep 18 13:53:20.377420 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:20.377392    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swswg\" (UniqueName: \"kubernetes.io/projected/940f1929-f76c-41e4-b354-94e17e2a780c-kube-api-access-swswg\") pod \"authorino-59c974bd48-w8lhc\" (UID: \"940f1929-f76c-41e4-b354-94e17e2a780c\") " pod="kuadrant-system/authorino-59c974bd48-w8lhc"
Sep 18 13:53:20.478289 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:20.478255    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swswg\" (UniqueName: \"kubernetes.io/projected/940f1929-f76c-41e4-b354-94e17e2a780c-kube-api-access-swswg\") pod \"authorino-59c974bd48-w8lhc\" (UID: \"940f1929-f76c-41e4-b354-94e17e2a780c\") " pod="kuadrant-system/authorino-59c974bd48-w8lhc"
Sep 18 13:53:20.487842 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:20.487811    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swswg\" (UniqueName: \"kubernetes.io/projected/940f1929-f76c-41e4-b354-94e17e2a780c-kube-api-access-swswg\") pod \"authorino-59c974bd48-w8lhc\" (UID: \"940f1929-f76c-41e4-b354-94e17e2a780c\") " pod="kuadrant-system/authorino-59c974bd48-w8lhc"
Sep 18 13:53:20.592829 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:20.592748    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-w8lhc"
Sep 18 13:53:20.875341 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:20.875314    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-w8lhc"]
Sep 18 13:53:20.877426 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:53:20.877399    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod940f1929_f76c_41e4_b354_94e17e2a780c.slice/crio-67f8b50a1910e3434a089b3f87091251001b7de37267b80c5d3dc11da468286e WatchSource:0}: Error finding container 67f8b50a1910e3434a089b3f87091251001b7de37267b80c5d3dc11da468286e: Status 404 returned error can't find the container with id 67f8b50a1910e3434a089b3f87091251001b7de37267b80c5d3dc11da468286e
Sep 18 13:53:21.634605 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:21.634566    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-w8lhc" event={"ID":"940f1929-f76c-41e4-b354-94e17e2a780c","Type":"ContainerStarted","Data":"67f8b50a1910e3434a089b3f87091251001b7de37267b80c5d3dc11da468286e"}
Sep 18 13:53:22.641112 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:22.641011    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-w8lhc" event={"ID":"940f1929-f76c-41e4-b354-94e17e2a780c","Type":"ContainerStarted","Data":"0a409adeafd49bfae8b16d1f9a745e5b19a53b0d2d0aaa152ea00af91c405acc"}
Sep 18 13:53:22.657718 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:22.657670    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-w8lhc" podStartSLOduration=1.267565284 podStartE2EDuration="2.657654975s" totalImagesPullingTime="1.390089691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:53:22.655486766 +0000 UTC m=+538.590451913" watchObservedRunningTime="2026-09-18 13:53:22.657654975 +0000 UTC m=+538.592620119"
Sep 18 13:53:49.351036 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:49.351004    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-w8lhc"]
Sep 18 13:53:49.351513 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:49.351181    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-w8lhc" podUID="940f1929-f76c-41e4-b354-94e17e2a780c" containerName="authorino" containerID="cri-o://0a409adeafd49bfae8b16d1f9a745e5b19a53b0d2d0aaa152ea00af91c405acc" gracePeriod=30
Sep 18 13:53:49.584686 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:49.584664    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-w8lhc"
Sep 18 13:53:49.707251 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:49.707223    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swswg\" (UniqueName: \"kubernetes.io/projected/940f1929-f76c-41e4-b354-94e17e2a780c-kube-api-access-swswg\") pod \"940f1929-f76c-41e4-b354-94e17e2a780c\" (UID: \"940f1929-f76c-41e4-b354-94e17e2a780c\") "
Sep 18 13:53:49.709342 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:49.709321    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940f1929-f76c-41e4-b354-94e17e2a780c-kube-api-access-swswg" (OuterVolumeSpecName: "kube-api-access-swswg") pod "940f1929-f76c-41e4-b354-94e17e2a780c" (UID: "940f1929-f76c-41e4-b354-94e17e2a780c"). InnerVolumeSpecName "kube-api-access-swswg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:53:49.785542 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:49.785502    2572 generic.go:356] "Generic (PLEG): container finished" podID="940f1929-f76c-41e4-b354-94e17e2a780c" containerID="0a409adeafd49bfae8b16d1f9a745e5b19a53b0d2d0aaa152ea00af91c405acc" exitCode=0
Sep 18 13:53:49.785652 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:49.785563    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-w8lhc"
Sep 18 13:53:49.785652 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:49.785587    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-w8lhc" event={"ID":"940f1929-f76c-41e4-b354-94e17e2a780c","Type":"ContainerDied","Data":"0a409adeafd49bfae8b16d1f9a745e5b19a53b0d2d0aaa152ea00af91c405acc"}
Sep 18 13:53:49.785652 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:49.785629    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-w8lhc" event={"ID":"940f1929-f76c-41e4-b354-94e17e2a780c","Type":"ContainerDied","Data":"67f8b50a1910e3434a089b3f87091251001b7de37267b80c5d3dc11da468286e"}
Sep 18 13:53:49.785652 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:49.785646    2572 scope.go:117] "RemoveContainer" containerID="0a409adeafd49bfae8b16d1f9a745e5b19a53b0d2d0aaa152ea00af91c405acc"
Sep 18 13:53:49.794449 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:49.794425    2572 scope.go:117] "RemoveContainer" containerID="0a409adeafd49bfae8b16d1f9a745e5b19a53b0d2d0aaa152ea00af91c405acc"
Sep 18 13:53:49.794890 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:53:49.794866    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a409adeafd49bfae8b16d1f9a745e5b19a53b0d2d0aaa152ea00af91c405acc\": container with ID starting with 0a409adeafd49bfae8b16d1f9a745e5b19a53b0d2d0aaa152ea00af91c405acc not found: ID does not exist" containerID="0a409adeafd49bfae8b16d1f9a745e5b19a53b0d2d0aaa152ea00af91c405acc"
Sep 18 13:53:49.794983 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:49.794896    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a409adeafd49bfae8b16d1f9a745e5b19a53b0d2d0aaa152ea00af91c405acc"} err="failed to get container status \"0a409adeafd49bfae8b16d1f9a745e5b19a53b0d2d0aaa152ea00af91c405acc\": rpc error: code = NotFound desc = could not find container \"0a409adeafd49bfae8b16d1f9a745e5b19a53b0d2d0aaa152ea00af91c405acc\": container with ID starting with 0a409adeafd49bfae8b16d1f9a745e5b19a53b0d2d0aaa152ea00af91c405acc not found: ID does not exist"
Sep 18 13:53:49.808399 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:49.808379    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swswg\" (UniqueName: \"kubernetes.io/projected/940f1929-f76c-41e4-b354-94e17e2a780c-kube-api-access-swswg\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:53:49.814158 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:49.814132    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-w8lhc"]
Sep 18 13:53:49.827400 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:49.827379    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-w8lhc"]
Sep 18 13:53:50.196798 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:50.196768    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="940f1929-f76c-41e4-b354-94e17e2a780c" path="/var/lib/kubelet/pods/940f1929-f76c-41e4-b354-94e17e2a780c/volumes"
Sep 18 13:53:54.250796 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.250759    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58db965b5-wscbg"]
Sep 18 13:53:54.251283 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.251229    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="940f1929-f76c-41e4-b354-94e17e2a780c" containerName="authorino"
Sep 18 13:53:54.251283 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.251248    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="940f1929-f76c-41e4-b354-94e17e2a780c" containerName="authorino"
Sep 18 13:53:54.251398 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.251337    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="940f1929-f76c-41e4-b354-94e17e2a780c" containerName="authorino"
Sep 18 13:53:54.257669 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.257643    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58db965b5-wscbg"
Sep 18 13:53:54.260295 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.260270    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 13:53:54.260400 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.260348    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 13:53:54.260986 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.260968    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 13:53:54.261040 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.260998    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nhzv4\""
Sep 18 13:53:54.268231 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.268202    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58db965b5-wscbg"]
Sep 18 13:53:54.346393 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.346366    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvn52\" (UniqueName: \"kubernetes.io/projected/6fe90d56-762e-42f2-be8e-eaa5444bed8d-kube-api-access-cvn52\") pod \"maas-controller-58db965b5-wscbg\" (UID: \"6fe90d56-762e-42f2-be8e-eaa5444bed8d\") " pod="opendatahub/maas-controller-58db965b5-wscbg"
Sep 18 13:53:54.346518 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.346429    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe90d56-762e-42f2-be8e-eaa5444bed8d-metrics-certs\") pod \"maas-controller-58db965b5-wscbg\" (UID: \"6fe90d56-762e-42f2-be8e-eaa5444bed8d\") " pod="opendatahub/maas-controller-58db965b5-wscbg"
Sep 18 13:53:54.346599 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.346515    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fe90d56-762e-42f2-be8e-eaa5444bed8d-webhook-cert\") pod \"maas-controller-58db965b5-wscbg\" (UID: \"6fe90d56-762e-42f2-be8e-eaa5444bed8d\") " pod="opendatahub/maas-controller-58db965b5-wscbg"
Sep 18 13:53:54.447796 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.447770    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvn52\" (UniqueName: \"kubernetes.io/projected/6fe90d56-762e-42f2-be8e-eaa5444bed8d-kube-api-access-cvn52\") pod \"maas-controller-58db965b5-wscbg\" (UID: \"6fe90d56-762e-42f2-be8e-eaa5444bed8d\") " pod="opendatahub/maas-controller-58db965b5-wscbg"
Sep 18 13:53:54.447899 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.447820    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe90d56-762e-42f2-be8e-eaa5444bed8d-metrics-certs\") pod \"maas-controller-58db965b5-wscbg\" (UID: \"6fe90d56-762e-42f2-be8e-eaa5444bed8d\") " pod="opendatahub/maas-controller-58db965b5-wscbg"
Sep 18 13:53:54.447899 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.447858    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fe90d56-762e-42f2-be8e-eaa5444bed8d-webhook-cert\") pod \"maas-controller-58db965b5-wscbg\" (UID: \"6fe90d56-762e-42f2-be8e-eaa5444bed8d\") " pod="opendatahub/maas-controller-58db965b5-wscbg"
Sep 18 13:53:54.447984 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:53:54.447969    2572 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 18 13:53:54.448049 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:53:54.448039    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe90d56-762e-42f2-be8e-eaa5444bed8d-webhook-cert podName:6fe90d56-762e-42f2-be8e-eaa5444bed8d nodeName:}" failed. No retries permitted until 2026-09-18 13:53:54.9480189 +0000 UTC m=+570.882984037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/6fe90d56-762e-42f2-be8e-eaa5444bed8d-webhook-cert") pod "maas-controller-58db965b5-wscbg" (UID: "6fe90d56-762e-42f2-be8e-eaa5444bed8d") : secret "maas-controller-webhook-cert" not found
Sep 18 13:53:54.450201 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.450180    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe90d56-762e-42f2-be8e-eaa5444bed8d-metrics-certs\") pod \"maas-controller-58db965b5-wscbg\" (UID: \"6fe90d56-762e-42f2-be8e-eaa5444bed8d\") " pod="opendatahub/maas-controller-58db965b5-wscbg"
Sep 18 13:53:54.460249 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.460220    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvn52\" (UniqueName: \"kubernetes.io/projected/6fe90d56-762e-42f2-be8e-eaa5444bed8d-kube-api-access-cvn52\") pod \"maas-controller-58db965b5-wscbg\" (UID: \"6fe90d56-762e-42f2-be8e-eaa5444bed8d\") " pod="opendatahub/maas-controller-58db965b5-wscbg"
Sep 18 13:53:54.953402 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.953370    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fe90d56-762e-42f2-be8e-eaa5444bed8d-webhook-cert\") pod \"maas-controller-58db965b5-wscbg\" (UID: \"6fe90d56-762e-42f2-be8e-eaa5444bed8d\") " pod="opendatahub/maas-controller-58db965b5-wscbg"
Sep 18 13:53:54.955814 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:54.955782    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fe90d56-762e-42f2-be8e-eaa5444bed8d-webhook-cert\") pod \"maas-controller-58db965b5-wscbg\" (UID: \"6fe90d56-762e-42f2-be8e-eaa5444bed8d\") " pod="opendatahub/maas-controller-58db965b5-wscbg"
Sep 18 13:53:55.167801 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:55.167762    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58db965b5-wscbg"
Sep 18 13:53:55.439765 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:55.439741    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58db965b5-wscbg"]
Sep 18 13:53:55.441603 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:53:55.441579    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe90d56_762e_42f2_be8e_eaa5444bed8d.slice/crio-c6424f30e8fb95e5ea89f0f45f890c94c23f5a6e7615b140f266e8328791643f WatchSource:0}: Error finding container c6424f30e8fb95e5ea89f0f45f890c94c23f5a6e7615b140f266e8328791643f: Status 404 returned error can't find the container with id c6424f30e8fb95e5ea89f0f45f890c94c23f5a6e7615b140f266e8328791643f
Sep 18 13:53:55.819459 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:55.819416    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58db965b5-wscbg" event={"ID":"6fe90d56-762e-42f2-be8e-eaa5444bed8d","Type":"ContainerStarted","Data":"c6424f30e8fb95e5ea89f0f45f890c94c23f5a6e7615b140f266e8328791643f"}
Sep 18 13:53:57.832305 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:57.832267    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58db965b5-wscbg" event={"ID":"6fe90d56-762e-42f2-be8e-eaa5444bed8d","Type":"ContainerStarted","Data":"ce106636041b21ca5c70615c2ea886ae609461a7df1005d102f39e2a6427ece0"}
Sep 18 13:53:57.832705 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:57.832429    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58db965b5-wscbg"
Sep 18 13:53:57.851115 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:53:57.851069    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58db965b5-wscbg" podStartSLOduration=1.535745735 podStartE2EDuration="3.851055936s" totalImagesPullingTime="2.315310201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:53:57.850302081 +0000 UTC m=+573.785267226" watchObservedRunningTime="2026-09-18 13:53:57.851055936 +0000 UTC m=+573.786021097"
Sep 18 13:54:08.839862 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:08.839830    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58db965b5-wscbg"
Sep 18 13:54:26.162761 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:26.162734    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 13:54:26.163458 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:26.163438    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 13:54:30.035331 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.035293    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"]
Sep 18 13:54:30.041807 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.041785    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 13:54:30.044101 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.044080    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 13:54:30.044216 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.044119    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 13:54:30.044216 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.044197    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 13:54:30.044315 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.044257    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5jwtb\""
Sep 18 13:54:30.044581 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.044563    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:54:30.050654 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.050636    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"]
Sep 18 13:54:30.150843 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.150813    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t9dk\" (UniqueName: \"kubernetes.io/projected/9d506203-21f4-4793-80dd-14e00744995a-kube-api-access-2t9dk\") pod \"maas-api-5fc7cc57dd-7pfjw\" (UID: \"9d506203-21f4-4793-80dd-14e00744995a\") " pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 13:54:30.150989 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.150885    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d506203-21f4-4793-80dd-14e00744995a-openshift-service-ca\") pod \"maas-api-5fc7cc57dd-7pfjw\" (UID: \"9d506203-21f4-4793-80dd-14e00744995a\") " pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 13:54:30.150989 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.150930    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d506203-21f4-4793-80dd-14e00744995a-maas-api-tls\") pod \"maas-api-5fc7cc57dd-7pfjw\" (UID: \"9d506203-21f4-4793-80dd-14e00744995a\") " pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 13:54:30.150989 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.150946    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d506203-21f4-4793-80dd-14e00744995a-metrics-certs\") pod \"maas-api-5fc7cc57dd-7pfjw\" (UID: \"9d506203-21f4-4793-80dd-14e00744995a\") " pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 13:54:30.251665 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.251617    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d506203-21f4-4793-80dd-14e00744995a-openshift-service-ca\") pod \"maas-api-5fc7cc57dd-7pfjw\" (UID: \"9d506203-21f4-4793-80dd-14e00744995a\") " pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 13:54:30.251834 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.251667    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d506203-21f4-4793-80dd-14e00744995a-maas-api-tls\") pod \"maas-api-5fc7cc57dd-7pfjw\" (UID: \"9d506203-21f4-4793-80dd-14e00744995a\") " pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 13:54:30.251834 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.251692    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d506203-21f4-4793-80dd-14e00744995a-metrics-certs\") pod \"maas-api-5fc7cc57dd-7pfjw\" (UID: \"9d506203-21f4-4793-80dd-14e00744995a\") " pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 13:54:30.251834 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.251731    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t9dk\" (UniqueName: \"kubernetes.io/projected/9d506203-21f4-4793-80dd-14e00744995a-kube-api-access-2t9dk\") pod \"maas-api-5fc7cc57dd-7pfjw\" (UID: \"9d506203-21f4-4793-80dd-14e00744995a\") " pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 13:54:30.252719 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.252691    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d506203-21f4-4793-80dd-14e00744995a-openshift-service-ca\") pod \"maas-api-5fc7cc57dd-7pfjw\" (UID: \"9d506203-21f4-4793-80dd-14e00744995a\") " pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 13:54:30.255492 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.255468    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d506203-21f4-4793-80dd-14e00744995a-metrics-certs\") pod \"maas-api-5fc7cc57dd-7pfjw\" (UID: \"9d506203-21f4-4793-80dd-14e00744995a\") " pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 13:54:30.255656 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.255626    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d506203-21f4-4793-80dd-14e00744995a-maas-api-tls\") pod \"maas-api-5fc7cc57dd-7pfjw\" (UID: \"9d506203-21f4-4793-80dd-14e00744995a\") " pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 13:54:30.259785 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.259762    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t9dk\" (UniqueName: \"kubernetes.io/projected/9d506203-21f4-4793-80dd-14e00744995a-kube-api-access-2t9dk\") pod \"maas-api-5fc7cc57dd-7pfjw\" (UID: \"9d506203-21f4-4793-80dd-14e00744995a\") " pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 13:54:30.353374 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.353306    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 13:54:30.646675 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:54:30.646634    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d506203_21f4_4793_80dd_14e00744995a.slice/crio-352356ab197d979e985d76617a3fb0d03b0889f28d94669c7d411fc87c60eb3c WatchSource:0}: Error finding container 352356ab197d979e985d76617a3fb0d03b0889f28d94669c7d411fc87c60eb3c: Status 404 returned error can't find the container with id 352356ab197d979e985d76617a3fb0d03b0889f28d94669c7d411fc87c60eb3c
Sep 18 13:54:30.647372 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.647350    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"]
Sep 18 13:54:30.648071 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:30.648052    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:54:31.012086 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:31.012053    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw" event={"ID":"9d506203-21f4-4793-80dd-14e00744995a","Type":"ContainerStarted","Data":"352356ab197d979e985d76617a3fb0d03b0889f28d94669c7d411fc87c60eb3c"}
Sep 18 13:54:34.034815 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:34.034770    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw" event={"ID":"9d506203-21f4-4793-80dd-14e00744995a","Type":"ContainerStarted","Data":"c7d3d8e86d6b74e538b232489a26cfbe2ab7ddf9e33a4d64f8783978b41caee6"}
Sep 18 13:54:34.035211 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:34.034840    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 13:54:34.055873 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:34.055819    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw" podStartSLOduration=1.5958834450000001 podStartE2EDuration="4.05580414s" totalImagesPullingTime="2.459920695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:54:34.053644753 +0000 UTC m=+609.988609898" watchObservedRunningTime="2026-09-18 13:54:34.05580414 +0000 UTC m=+609.990769286"
Sep 18 13:54:40.046308 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:40.046236    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 13:54:43.778036 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.778000    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"]
Sep 18 13:54:43.781782 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.781757    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.784064 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.784037    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 13:54:43.784182 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.784131    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t99ts\""
Sep 18 13:54:43.784785 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.784751    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 18 13:54:43.784902 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.784800    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 13:54:43.791098 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.791076    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"]
Sep 18 13:54:43.864069 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.864037    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9751eb95-f681-4e01-829f-462338c0a314-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.864202 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.864086    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9751eb95-f681-4e01-829f-462338c0a314-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.864202 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.864110    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9751eb95-f681-4e01-829f-462338c0a314-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.864309 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.864201    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9751eb95-f681-4e01-829f-462338c0a314-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.864309 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.864227    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9751eb95-f681-4e01-829f-462338c0a314-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.864309 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.864254    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76kng\" (UniqueName: \"kubernetes.io/projected/9751eb95-f681-4e01-829f-462338c0a314-kube-api-access-76kng\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.965315 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.965282    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76kng\" (UniqueName: \"kubernetes.io/projected/9751eb95-f681-4e01-829f-462338c0a314-kube-api-access-76kng\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.965483 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.965374    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9751eb95-f681-4e01-829f-462338c0a314-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.965483 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.965425    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9751eb95-f681-4e01-829f-462338c0a314-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.965483 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.965442    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9751eb95-f681-4e01-829f-462338c0a314-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.965705 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.965512    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9751eb95-f681-4e01-829f-462338c0a314-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.965705 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.965562    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9751eb95-f681-4e01-829f-462338c0a314-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.965933 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.965908    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9751eb95-f681-4e01-829f-462338c0a314-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.966008 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.965937    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9751eb95-f681-4e01-829f-462338c0a314-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.966008 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.965970    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9751eb95-f681-4e01-829f-462338c0a314-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.967775 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.967759    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9751eb95-f681-4e01-829f-462338c0a314-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.967958 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.967940    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9751eb95-f681-4e01-829f-462338c0a314-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:43.972970 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:43.972949    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76kng\" (UniqueName: \"kubernetes.io/projected/9751eb95-f681-4e01-829f-462338c0a314-kube-api-access-76kng\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x\" (UID: \"9751eb95-f681-4e01-829f-462338c0a314\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:44.094394 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:44.094311    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:44.367886 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:44.367857    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"]
Sep 18 13:54:44.369677 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:54:44.369638    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9751eb95_f681_4e01_829f_462338c0a314.slice/crio-707e6da9039fa1d5920af56ecc74a8346d0f138f3539e1b29ed4dbd7f98d4743 WatchSource:0}: Error finding container 707e6da9039fa1d5920af56ecc74a8346d0f138f3539e1b29ed4dbd7f98d4743: Status 404 returned error can't find the container with id 707e6da9039fa1d5920af56ecc74a8346d0f138f3539e1b29ed4dbd7f98d4743
Sep 18 13:54:45.093345 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:45.093308    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x" event={"ID":"9751eb95-f681-4e01-829f-462338c0a314","Type":"ContainerStarted","Data":"707e6da9039fa1d5920af56ecc74a8346d0f138f3539e1b29ed4dbd7f98d4743"}
Sep 18 13:54:46.100657 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:46.100613    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x" event={"ID":"9751eb95-f681-4e01-829f-462338c0a314","Type":"ContainerStarted","Data":"4a5e33992872d3da201878a58f8c9dbf309dc25353618e7da94bbc47817ec001"}
Sep 18 13:54:46.121475 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:46.121421    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x" podStartSLOduration=2.156995435 podStartE2EDuration="3.121401805s" totalImagesPullingTime="964.40637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:54:46.117930276 +0000 UTC m=+622.052895422" watchObservedRunningTime="2026-09-18 13:54:46.121401805 +0000 UTC m=+622.056366951"
Sep 18 13:54:54.094765 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:54.094730    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:54.095203 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:54.094861    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:54.107653 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:54.107626    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:54:54.155883 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:54:54.155856    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x"
Sep 18 13:55:03.173127 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.173094    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"]
Sep 18 13:55:03.178361 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.178339    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.180717 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.180690    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 18 13:55:03.185527 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.185503    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"]
Sep 18 13:55:03.334168 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.334132    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.334323 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.334176    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z5lx\" (UniqueName: \"kubernetes.io/projected/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-kube-api-access-8z5lx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.334323 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.334203    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.334323 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.334271    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.334323 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.334314    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.334463 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.334366    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.435429 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.435357    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.435429 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.435417    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.435695 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.435441    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.435695 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.435466    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z5lx\" (UniqueName: \"kubernetes.io/projected/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-kube-api-access-8z5lx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.435695 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.435492    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.435695 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.435509    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.435882 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.435817    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.435882 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.435856    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.435991 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.435886    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.437749 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.437715    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.437884 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.437828    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.443826 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.443808    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z5lx\" (UniqueName: \"kubernetes.io/projected/8dd86f94-9b6d-4606-845b-4a6ffdee9e48-kube-api-access-8z5lx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6\" (UID: \"8dd86f94-9b6d-4606-845b-4a6ffdee9e48\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.492402 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.492376    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:03.765766 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:03.765734    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"]
Sep 18 13:55:03.767182 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:55:03.767158    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd86f94_9b6d_4606_845b_4a6ffdee9e48.slice/crio-ab4e520675a6acdf8ce582de82811d04866bb370380c71dc9b4610ece24fa585 WatchSource:0}: Error finding container ab4e520675a6acdf8ce582de82811d04866bb370380c71dc9b4610ece24fa585: Status 404 returned error can't find the container with id ab4e520675a6acdf8ce582de82811d04866bb370380c71dc9b4610ece24fa585
Sep 18 13:55:04.212444 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:04.212406    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6" event={"ID":"8dd86f94-9b6d-4606-845b-4a6ffdee9e48","Type":"ContainerStarted","Data":"ab4e520675a6acdf8ce582de82811d04866bb370380c71dc9b4610ece24fa585"}
Sep 18 13:55:05.220244 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:05.220201    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6" event={"ID":"8dd86f94-9b6d-4606-845b-4a6ffdee9e48","Type":"ContainerStarted","Data":"5dafba5ff9a40182f2d9f7cc2068695a86a796a657df27f02591a9213de7bcc5"}
Sep 18 13:55:05.241263 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:05.241212    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6" podStartSLOduration=1.747115916 podStartE2EDuration="2.241193858s" totalImagesPullingTime="494.077942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:55:05.24065173 +0000 UTC m=+641.175616877" watchObservedRunningTime="2026-09-18 13:55:05.241193858 +0000 UTC m=+641.176159002"
Sep 18 13:55:13.493513 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:13.493471    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:13.493989 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:13.493648    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:13.505948 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:13.505925    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:55:14.276669 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:55:14.276640    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6"
Sep 18 13:58:25.384449 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:25.384418    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"]
Sep 18 13:58:25.388127 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:25.388106    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:25.391546 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:25.391506    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-a02b2c-serving-cert\""
Sep 18 13:58:25.404601 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:25.404575    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"]
Sep 18 13:58:25.459998 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:25.459972    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:25.460132 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:25.460015    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drrkh\" (UniqueName: \"kubernetes.io/projected/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-kube-api-access-drrkh\") pod \"maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:25.460132 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:25.460040    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-metrics-certs\") pod \"maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:25.460212 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:25.460137    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-maas-api-tls\") pod \"maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:25.561108 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:25.561071    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrkh\" (UniqueName: \"kubernetes.io/projected/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-kube-api-access-drrkh\") pod \"maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:25.561282 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:25.561117    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-metrics-certs\") pod \"maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:25.561332 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:25.561299    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-maas-api-tls\") pod \"maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:25.561390 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:25.561375    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:25.561454 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:58:25.561432    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-serving-cert: secret "maas-api-e2e-worker-w4-a02b2c-serving-cert" not found
Sep 18 13:58:25.561526 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:58:25.561514    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-maas-api-tls podName:9ce59ddc-1186-404f-9e2e-d4d3ce615a83 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:26.061491175 +0000 UTC m=+841.996456299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-maas-api-tls") pod "maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln" (UID: "9ce59ddc-1186-404f-9e2e-d4d3ce615a83") : secret "maas-api-e2e-worker-w4-a02b2c-serving-cert" not found
Sep 18 13:58:25.562075 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:25.562057    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:25.563546 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:25.563498    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-metrics-certs\") pod \"maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:25.573595 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:25.573573    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drrkh\" (UniqueName: \"kubernetes.io/projected/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-kube-api-access-drrkh\") pod \"maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:26.065600 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.065567    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-maas-api-tls\") pod \"maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:26.068381 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.068351    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-maas-api-tls\") pod \"maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:26.179833 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.179799    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"]
Sep 18 13:58:26.183659 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.183634    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"
Sep 18 13:58:26.186067 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.186045    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-a02b2c\""
Sep 18 13:58:26.186545 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.186515    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-a02b2c-dockercfg-fgdc9\""
Sep 18 13:58:26.199217 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.199196    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"]
Sep 18 13:58:26.267237 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.267208    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0135fa54-68df-4431-abfe-74266565f05e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2\" (UID: \"0135fa54-68df-4431-abfe-74266565f05e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"
Sep 18 13:58:26.267393 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.267274    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drbjl\" (UniqueName: \"kubernetes.io/projected/0135fa54-68df-4431-abfe-74266565f05e-kube-api-access-drbjl\") pod \"payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2\" (UID: \"0135fa54-68df-4431-abfe-74266565f05e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"
Sep 18 13:58:26.298897 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.298869    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:26.368956 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.368166    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drbjl\" (UniqueName: \"kubernetes.io/projected/0135fa54-68df-4431-abfe-74266565f05e-kube-api-access-drbjl\") pod \"payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2\" (UID: \"0135fa54-68df-4431-abfe-74266565f05e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"
Sep 18 13:58:26.368956 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.368292    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0135fa54-68df-4431-abfe-74266565f05e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2\" (UID: \"0135fa54-68df-4431-abfe-74266565f05e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"
Sep 18 13:58:26.369347 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.369317    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0135fa54-68df-4431-abfe-74266565f05e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2\" (UID: \"0135fa54-68df-4431-abfe-74266565f05e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"
Sep 18 13:58:26.378100 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.378044    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drbjl\" (UniqueName: \"kubernetes.io/projected/0135fa54-68df-4431-abfe-74266565f05e-kube-api-access-drbjl\") pod \"payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2\" (UID: \"0135fa54-68df-4431-abfe-74266565f05e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"
Sep 18 13:58:26.496722 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.496687    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"
Sep 18 13:58:26.579743 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.579721    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"]
Sep 18 13:58:26.583598 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:58:26.583569    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce59ddc_1186_404f_9e2e_d4d3ce615a83.slice/crio-84276a8a967f8009f4969e8d6ed72111f8becbdd426cf4bd077d9a3792520615 WatchSource:0}: Error finding container 84276a8a967f8009f4969e8d6ed72111f8becbdd426cf4bd077d9a3792520615: Status 404 returned error can't find the container with id 84276a8a967f8009f4969e8d6ed72111f8becbdd426cf4bd077d9a3792520615
Sep 18 13:58:26.692058 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.692033    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"]
Sep 18 13:58:26.696723 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.696705    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 13:58:26.698850 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.698824    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-40970f-serving-cert\""
Sep 18 13:58:26.707489 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.707463    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"]
Sep 18 13:58:26.771447 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.771421    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c769edea-c5d3-4901-908c-c2c9faf72504-maas-api-tls\") pod \"maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9\" (UID: \"c769edea-c5d3-4901-908c-c2c9faf72504\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 13:58:26.771624 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.771455    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8zf5\" (UniqueName: \"kubernetes.io/projected/c769edea-c5d3-4901-908c-c2c9faf72504-kube-api-access-g8zf5\") pod \"maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9\" (UID: \"c769edea-c5d3-4901-908c-c2c9faf72504\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 13:58:26.771624 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.771484    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c769edea-c5d3-4901-908c-c2c9faf72504-metrics-certs\") pod \"maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9\" (UID: \"c769edea-c5d3-4901-908c-c2c9faf72504\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 13:58:26.771624 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.771592    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c769edea-c5d3-4901-908c-c2c9faf72504-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9\" (UID: \"c769edea-c5d3-4901-908c-c2c9faf72504\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 13:58:26.776094 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.776070    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"]
Sep 18 13:58:26.777172 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:58:26.777142    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0135fa54_68df_4431_abfe_74266565f05e.slice/crio-b36551d9f08ebb58c03d51ecc9606a8e05cca5ad36b41d18a05f4efa22c7ae24 WatchSource:0}: Error finding container b36551d9f08ebb58c03d51ecc9606a8e05cca5ad36b41d18a05f4efa22c7ae24: Status 404 returned error can't find the container with id b36551d9f08ebb58c03d51ecc9606a8e05cca5ad36b41d18a05f4efa22c7ae24
Sep 18 13:58:26.872296 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.872270    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c769edea-c5d3-4901-908c-c2c9faf72504-maas-api-tls\") pod \"maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9\" (UID: \"c769edea-c5d3-4901-908c-c2c9faf72504\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 13:58:26.872430 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.872305    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8zf5\" (UniqueName: \"kubernetes.io/projected/c769edea-c5d3-4901-908c-c2c9faf72504-kube-api-access-g8zf5\") pod \"maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9\" (UID: \"c769edea-c5d3-4901-908c-c2c9faf72504\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 13:58:26.872430 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.872326    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c769edea-c5d3-4901-908c-c2c9faf72504-metrics-certs\") pod \"maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9\" (UID: \"c769edea-c5d3-4901-908c-c2c9faf72504\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 13:58:26.872575 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.872443    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c769edea-c5d3-4901-908c-c2c9faf72504-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9\" (UID: \"c769edea-c5d3-4901-908c-c2c9faf72504\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 13:58:26.873073 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.873053    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c769edea-c5d3-4901-908c-c2c9faf72504-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9\" (UID: \"c769edea-c5d3-4901-908c-c2c9faf72504\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 13:58:26.874794 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.874768    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c769edea-c5d3-4901-908c-c2c9faf72504-metrics-certs\") pod \"maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9\" (UID: \"c769edea-c5d3-4901-908c-c2c9faf72504\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 13:58:26.874876 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.874774    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c769edea-c5d3-4901-908c-c2c9faf72504-maas-api-tls\") pod \"maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9\" (UID: \"c769edea-c5d3-4901-908c-c2c9faf72504\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 13:58:26.883138 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:26.883120    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8zf5\" (UniqueName: \"kubernetes.io/projected/c769edea-c5d3-4901-908c-c2c9faf72504-kube-api-access-g8zf5\") pod \"maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9\" (UID: \"c769edea-c5d3-4901-908c-c2c9faf72504\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 13:58:27.008412 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:27.008389    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 13:58:27.295712 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:27.295685    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"]
Sep 18 13:58:27.298518 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:58:27.298485    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc769edea_c5d3_4901_908c_c2c9faf72504.slice/crio-c2817dcf5f1f28424f5f0fec197cff3cede96597fc193832746d6c81b5892501 WatchSource:0}: Error finding container c2817dcf5f1f28424f5f0fec197cff3cede96597fc193832746d6c81b5892501: Status 404 returned error can't find the container with id c2817dcf5f1f28424f5f0fec197cff3cede96597fc193832746d6c81b5892501
Sep 18 13:58:27.299482 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:27.299448    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2" event={"ID":"0135fa54-68df-4431-abfe-74266565f05e","Type":"ContainerStarted","Data":"b36551d9f08ebb58c03d51ecc9606a8e05cca5ad36b41d18a05f4efa22c7ae24"}
Sep 18 13:58:27.301427 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:27.301369    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln" event={"ID":"9ce59ddc-1186-404f-9e2e-d4d3ce615a83","Type":"ContainerStarted","Data":"8c5b156d635d8595ea31c664ac3628abb15e09e54fb8f52ee34f280fb6b46469"}
Sep 18 13:58:27.301604 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:27.301454    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln" event={"ID":"9ce59ddc-1186-404f-9e2e-d4d3ce615a83","Type":"ContainerStarted","Data":"84276a8a967f8009f4969e8d6ed72111f8becbdd426cf4bd077d9a3792520615"}
Sep 18 13:58:27.301604 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:27.301595    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:27.323304 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:27.323253    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln" podStartSLOduration=1.912504589 podStartE2EDuration="2.323233556s" totalImagesPullingTime="410.728967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:27.322830911 +0000 UTC m=+843.257796049" watchObservedRunningTime="2026-09-18 13:58:27.323233556 +0000 UTC m=+843.258198702"
Sep 18 13:58:27.878472 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:27.878405    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"]
Sep 18 13:58:27.884164 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:27.884114    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:27.886688 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:27.886666    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-036fc2-serving-cert\""
Sep 18 13:58:27.897711 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:27.897687    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"]
Sep 18 13:58:27.985952 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:27.985903    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rzp4\" (UniqueName: \"kubernetes.io/projected/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-kube-api-access-2rzp4\") pod \"maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:27.986132 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:27.985975    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-maas-api-tls\") pod \"maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:27.986132 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:27.986001    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-metrics-certs\") pod \"maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:27.986132 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:27.986050    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:28.087149 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.087112    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rzp4\" (UniqueName: \"kubernetes.io/projected/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-kube-api-access-2rzp4\") pod \"maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:28.087335 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.087175    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-maas-api-tls\") pod \"maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:28.087335 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.087202    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-metrics-certs\") pod \"maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:28.087335 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.087252    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:28.087525 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:58:28.087352    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-serving-cert: secret "maas-api-e2e-inf-a-036fc2-serving-cert" not found
Sep 18 13:58:28.087525 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:58:28.087427    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-maas-api-tls podName:a88ecf2b-8437-4a65-9eb1-1c80054a9ae4 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:28.587406711 +0000 UTC m=+844.522371850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-maas-api-tls") pod "maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh" (UID: "a88ecf2b-8437-4a65-9eb1-1c80054a9ae4") : secret "maas-api-e2e-inf-a-036fc2-serving-cert" not found
Sep 18 13:58:28.088084 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.088060    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:28.089917 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.089894    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-metrics-certs\") pod \"maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:28.098045 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.097999    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rzp4\" (UniqueName: \"kubernetes.io/projected/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-kube-api-access-2rzp4\") pod \"maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:28.309404 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.309370    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9" event={"ID":"c769edea-c5d3-4901-908c-c2c9faf72504","Type":"ContainerStarted","Data":"31f1d58b05f606dd4bcc47db22fc67f96e893da5de4d467916581aaa0deb0eda"}
Sep 18 13:58:28.309573 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.309438    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 13:58:28.309573 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.309467    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9" event={"ID":"c769edea-c5d3-4901-908c-c2c9faf72504","Type":"ContainerStarted","Data":"c2817dcf5f1f28424f5f0fec197cff3cede96597fc193832746d6c81b5892501"}
Sep 18 13:58:28.311183 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.311154    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2" event={"ID":"0135fa54-68df-4431-abfe-74266565f05e","Type":"ContainerStarted","Data":"bfae8d760d981b5f20276917dd029f998f10780c96032bbb74230e3b48352888"}
Sep 18 13:58:28.311296 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.311210    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"
Sep 18 13:58:28.328999 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.328954    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9" podStartSLOduration=1.9650417249999999 podStartE2EDuration="2.328940011s" totalImagesPullingTime="363.898286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:28.326765862 +0000 UTC m=+844.261731019" watchObservedRunningTime="2026-09-18 13:58:28.328940011 +0000 UTC m=+844.263905156"
Sep 18 13:58:28.345679 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.345635    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2" podStartSLOduration=0.927149079 podStartE2EDuration="2.34562255s" totalImagesPullingTime="1.418473471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:28.34461539 +0000 UTC m=+844.279580536" watchObservedRunningTime="2026-09-18 13:58:28.34562255 +0000 UTC m=+844.280587695"
Sep 18 13:58:28.591523 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.591432    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-maas-api-tls\") pod \"maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:28.593982 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.593958    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-maas-api-tls\") pod \"maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:28.801668 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:28.801632    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:29.084331 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:58:29.084290    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88ecf2b_8437_4a65_9eb1_1c80054a9ae4.slice/crio-165ab52cccc4d09363a8ec94a8e231197c5ba7c7499d7b8de6f31bbf533371db WatchSource:0}: Error finding container 165ab52cccc4d09363a8ec94a8e231197c5ba7c7499d7b8de6f31bbf533371db: Status 404 returned error can't find the container with id 165ab52cccc4d09363a8ec94a8e231197c5ba7c7499d7b8de6f31bbf533371db
Sep 18 13:58:29.084697 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:29.084488    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"]
Sep 18 13:58:29.318899 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:29.318854    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh" event={"ID":"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4","Type":"ContainerStarted","Data":"165ab52cccc4d09363a8ec94a8e231197c5ba7c7499d7b8de6f31bbf533371db"}
Sep 18 13:58:30.326382 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:30.326338    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh" event={"ID":"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4","Type":"ContainerStarted","Data":"84ab4fe908e5fb22d19ec1e1ed30bcf222ea46b9009eeab4160417016e7b1aac"}
Sep 18 13:58:30.326779 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:30.326411    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:30.356994 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:30.356943    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh" podStartSLOduration=3.030170305 podStartE2EDuration="3.35692474s" totalImagesPullingTime="326.754435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:30.352990876 +0000 UTC m=+846.287956021" watchObservedRunningTime="2026-09-18 13:58:30.35692474 +0000 UTC m=+846.291889889"
Sep 18 13:58:33.318500 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:33.318470    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 13:58:34.325648 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:34.325619    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 13:58:36.190546 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.190493    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"]
Sep 18 13:58:36.194422 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.194396    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 13:58:36.196903 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.196883    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-13658e-serving-cert\""
Sep 18 13:58:36.205579 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.205556    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"]
Sep 18 13:58:36.338813 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.338786    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 13:58:36.368488 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.368458    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjrdh\" (UniqueName: \"kubernetes.io/projected/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-kube-api-access-cjrdh\") pod \"maas-api-e2e-autores-13658e-dd7d878fd-hr2xv\" (UID: \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 13:58:36.368661 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.368518    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-metrics-certs\") pod \"maas-api-e2e-autores-13658e-dd7d878fd-hr2xv\" (UID: \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 13:58:36.368753 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.368733    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-maas-api-tls\") pod \"maas-api-e2e-autores-13658e-dd7d878fd-hr2xv\" (UID: \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 13:58:36.368806 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.368794    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-openshift-service-ca\") pod \"maas-api-e2e-autores-13658e-dd7d878fd-hr2xv\" (UID: \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 13:58:36.470142 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.470044    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjrdh\" (UniqueName: \"kubernetes.io/projected/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-kube-api-access-cjrdh\") pod \"maas-api-e2e-autores-13658e-dd7d878fd-hr2xv\" (UID: \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 13:58:36.470320 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.470163    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-metrics-certs\") pod \"maas-api-e2e-autores-13658e-dd7d878fd-hr2xv\" (UID: \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 13:58:36.470403 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.470342    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-maas-api-tls\") pod \"maas-api-e2e-autores-13658e-dd7d878fd-hr2xv\" (UID: \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 13:58:36.470462 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.470409    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-openshift-service-ca\") pod \"maas-api-e2e-autores-13658e-dd7d878fd-hr2xv\" (UID: \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 13:58:36.470992 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.470969    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-openshift-service-ca\") pod \"maas-api-e2e-autores-13658e-dd7d878fd-hr2xv\" (UID: \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 13:58:36.472495 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.472475    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-metrics-certs\") pod \"maas-api-e2e-autores-13658e-dd7d878fd-hr2xv\" (UID: \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 13:58:36.472611 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.472520    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-maas-api-tls\") pod \"maas-api-e2e-autores-13658e-dd7d878fd-hr2xv\" (UID: \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 13:58:36.478520 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.478500    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjrdh\" (UniqueName: \"kubernetes.io/projected/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-kube-api-access-cjrdh\") pod \"maas-api-e2e-autores-13658e-dd7d878fd-hr2xv\" (UID: \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 13:58:36.507340 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.507316    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 13:58:36.780719 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.780689    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"]
Sep 18 13:58:36.781203 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:58:36.781181    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b790eec_5aea_4bff_b4ef_c5fa802e29cd.slice/crio-9a873ca39055f489963dc539657762eaeb023dff5156d886a1c97acfd50004b7 WatchSource:0}: Error finding container 9a873ca39055f489963dc539657762eaeb023dff5156d886a1c97acfd50004b7: Status 404 returned error can't find the container with id 9a873ca39055f489963dc539657762eaeb023dff5156d886a1c97acfd50004b7
Sep 18 13:58:36.993403 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.993368    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"]
Sep 18 13:58:36.998488 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:36.998471    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"
Sep 18 13:58:37.001013 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.000991    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-13658e-dockercfg-97gr2\""
Sep 18 13:58:37.001127 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.001013    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-13658e\""
Sep 18 13:58:37.009278 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.009249    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"]
Sep 18 13:58:37.029036 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.028971    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"]
Sep 18 13:58:37.032386 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.032369    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"
Sep 18 13:58:37.052583 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.052552    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"]
Sep 18 13:58:37.075836 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.075803    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddtmw\" (UniqueName: \"kubernetes.io/projected/a06c46fb-97e2-4ffd-8d04-ef2252a23030-kube-api-access-ddtmw\") pod \"payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc\" (UID: \"a06c46fb-97e2-4ffd-8d04-ef2252a23030\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"
Sep 18 13:58:37.075986 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.075949    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06c46fb-97e2-4ffd-8d04-ef2252a23030-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc\" (UID: \"a06c46fb-97e2-4ffd-8d04-ef2252a23030\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"
Sep 18 13:58:37.176383 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.176351    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddtmw\" (UniqueName: \"kubernetes.io/projected/a06c46fb-97e2-4ffd-8d04-ef2252a23030-kube-api-access-ddtmw\") pod \"payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc\" (UID: \"a06c46fb-97e2-4ffd-8d04-ef2252a23030\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"
Sep 18 13:58:37.176528 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.176470    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06c46fb-97e2-4ffd-8d04-ef2252a23030-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc\" (UID: \"a06c46fb-97e2-4ffd-8d04-ef2252a23030\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"
Sep 18 13:58:37.176528 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.176523    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3ef3b5e-2d17-41cf-8b44-1892903435e2-plugins-config-volume\") pod \"payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht\" (UID: \"e3ef3b5e-2d17-41cf-8b44-1892903435e2\") " pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"
Sep 18 13:58:37.176694 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.176600    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvsh5\" (UniqueName: \"kubernetes.io/projected/e3ef3b5e-2d17-41cf-8b44-1892903435e2-kube-api-access-mvsh5\") pod \"payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht\" (UID: \"e3ef3b5e-2d17-41cf-8b44-1892903435e2\") " pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"
Sep 18 13:58:37.177286 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.177260    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06c46fb-97e2-4ffd-8d04-ef2252a23030-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc\" (UID: \"a06c46fb-97e2-4ffd-8d04-ef2252a23030\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"
Sep 18 13:58:37.186390 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.186369    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddtmw\" (UniqueName: \"kubernetes.io/projected/a06c46fb-97e2-4ffd-8d04-ef2252a23030-kube-api-access-ddtmw\") pod \"payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc\" (UID: \"a06c46fb-97e2-4ffd-8d04-ef2252a23030\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"
Sep 18 13:58:37.276976 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.276941    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3ef3b5e-2d17-41cf-8b44-1892903435e2-plugins-config-volume\") pod \"payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht\" (UID: \"e3ef3b5e-2d17-41cf-8b44-1892903435e2\") " pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"
Sep 18 13:58:37.277337 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.276989    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvsh5\" (UniqueName: \"kubernetes.io/projected/e3ef3b5e-2d17-41cf-8b44-1892903435e2-kube-api-access-mvsh5\") pod \"payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht\" (UID: \"e3ef3b5e-2d17-41cf-8b44-1892903435e2\") " pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"
Sep 18 13:58:37.277732 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.277712    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3ef3b5e-2d17-41cf-8b44-1892903435e2-plugins-config-volume\") pod \"payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht\" (UID: \"e3ef3b5e-2d17-41cf-8b44-1892903435e2\") " pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"
Sep 18 13:58:37.291308 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.291243    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvsh5\" (UniqueName: \"kubernetes.io/projected/e3ef3b5e-2d17-41cf-8b44-1892903435e2-kube-api-access-mvsh5\") pod \"payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht\" (UID: \"e3ef3b5e-2d17-41cf-8b44-1892903435e2\") " pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"
Sep 18 13:58:37.305730 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.305705    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"]
Sep 18 13:58:37.309404 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.309387    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:37.310398 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.310380    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"
Sep 18 13:58:37.314792 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.314767    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-57fc5a8f-serving-cert\""
Sep 18 13:58:37.329200 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.329027    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"]
Sep 18 13:58:37.342180 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.342155    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"
Sep 18 13:58:37.378173 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.378109    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf11e73-504b-4815-a844-60e7afc4551d-metrics-certs\") pod \"maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:37.378392 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.378339    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf11e73-504b-4815-a844-60e7afc4551d-maas-api-tls\") pod \"maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:37.378634 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.378580    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8659\" (UniqueName: \"kubernetes.io/projected/3bf11e73-504b-4815-a844-60e7afc4551d-kube-api-access-l8659\") pod \"maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:37.378737 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.378682    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv" event={"ID":"2b790eec-5aea-4bff-b4ef-c5fa802e29cd","Type":"ContainerStarted","Data":"b438cbff94166461630d659b05fe5cff69d8fb70b8e25b90f839224b29aa2c36"}
Sep 18 13:58:37.378804 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.378738    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv" event={"ID":"2b790eec-5aea-4bff-b4ef-c5fa802e29cd","Type":"ContainerStarted","Data":"9a873ca39055f489963dc539657762eaeb023dff5156d886a1c97acfd50004b7"}
Sep 18 13:58:37.379121 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.378938    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf11e73-504b-4815-a844-60e7afc4551d-openshift-service-ca\") pod \"maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:37.379255 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.379135    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 13:58:37.422611 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.422338    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv" podStartSLOduration=1.028607574 podStartE2EDuration="1.422317358s" totalImagesPullingTime="393.709784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:37.42198218 +0000 UTC m=+853.356947336" watchObservedRunningTime="2026-09-18 13:58:37.422317358 +0000 UTC m=+853.357282504"
Sep 18 13:58:37.480641 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.480606    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf11e73-504b-4815-a844-60e7afc4551d-maas-api-tls\") pod \"maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:37.480807 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.480685    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8659\" (UniqueName: \"kubernetes.io/projected/3bf11e73-504b-4815-a844-60e7afc4551d-kube-api-access-l8659\") pod \"maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:37.480807 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:58:37.480735    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-serving-cert: secret "maas-api-e2e-ait-57fc5a8f-serving-cert" not found
Sep 18 13:58:37.480807 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.480769    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf11e73-504b-4815-a844-60e7afc4551d-openshift-service-ca\") pod \"maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:37.480807 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:58:37.480793    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf11e73-504b-4815-a844-60e7afc4551d-maas-api-tls podName:3bf11e73-504b-4815-a844-60e7afc4551d nodeName:}" failed. No retries permitted until 2026-09-18 13:58:37.980777116 +0000 UTC m=+853.915742240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3bf11e73-504b-4815-a844-60e7afc4551d-maas-api-tls") pod "maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf" (UID: "3bf11e73-504b-4815-a844-60e7afc4551d") : secret "maas-api-e2e-ait-57fc5a8f-serving-cert" not found
Sep 18 13:58:37.481038 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.480894    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf11e73-504b-4815-a844-60e7afc4551d-metrics-certs\") pod \"maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:37.481473 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.481447    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf11e73-504b-4815-a844-60e7afc4551d-openshift-service-ca\") pod \"maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:37.483183 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.483162    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf11e73-504b-4815-a844-60e7afc4551d-metrics-certs\") pod \"maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:37.491261 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.491241    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8659\" (UniqueName: \"kubernetes.io/projected/3bf11e73-504b-4815-a844-60e7afc4551d-kube-api-access-l8659\") pod \"maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:37.592971 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.592939    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"]
Sep 18 13:58:37.594621 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:58:37.594596    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06c46fb_97e2_4ffd_8d04_ef2252a23030.slice/crio-998fd31fd05eb356cb5312ad1289baf6fd898f581efd32d468026173050404c9 WatchSource:0}: Error finding container 998fd31fd05eb356cb5312ad1289baf6fd898f581efd32d468026173050404c9: Status 404 returned error can't find the container with id 998fd31fd05eb356cb5312ad1289baf6fd898f581efd32d468026173050404c9
Sep 18 13:58:37.627922 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.627894    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"]
Sep 18 13:58:37.633713 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:58:37.633686    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ef3b5e_2d17_41cf_8b44_1892903435e2.slice/crio-7f3ed4860dc2a20f3b7ef03c4d565eb8a793e0b0ceb17fd1d9acb9d576dfa1e0 WatchSource:0}: Error finding container 7f3ed4860dc2a20f3b7ef03c4d565eb8a793e0b0ceb17fd1d9acb9d576dfa1e0: Status 404 returned error can't find the container with id 7f3ed4860dc2a20f3b7ef03c4d565eb8a793e0b0ceb17fd1d9acb9d576dfa1e0
Sep 18 13:58:37.985921 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.985887    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf11e73-504b-4815-a844-60e7afc4551d-maas-api-tls\") pod \"maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:37.992446 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:37.992415    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf11e73-504b-4815-a844-60e7afc4551d-maas-api-tls\") pod \"maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:38.159527 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.159491    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"]
Sep 18 13:58:38.163172 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.163155    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"
Sep 18 13:58:38.165681 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.165650    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-57fc5a8f\""
Sep 18 13:58:38.165789 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.165703    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-57fc5a8f-dockercfg-ccfwb\""
Sep 18 13:58:38.175633 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.175612    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"]
Sep 18 13:58:38.188225 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.188203    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f89f2d-89db-42ca-8153-229a22508c4d-plugins-config-volume\") pod \"payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck\" (UID: \"c3f89f2d-89db-42ca-8153-229a22508c4d\") " pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"
Sep 18 13:58:38.188318 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.188262    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldllx\" (UniqueName: \"kubernetes.io/projected/c3f89f2d-89db-42ca-8153-229a22508c4d-kube-api-access-ldllx\") pod \"payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck\" (UID: \"c3f89f2d-89db-42ca-8153-229a22508c4d\") " pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"
Sep 18 13:58:38.223724 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.223693    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:38.231017 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.230993    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"]
Sep 18 13:58:38.234612 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.234593    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"
Sep 18 13:58:38.245246 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.245165    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"]
Sep 18 13:58:38.289520 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.289025    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f75de0ea-5b81-4f12-a58c-38e752eaa14c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b\" (UID: \"f75de0ea-5b81-4f12-a58c-38e752eaa14c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"
Sep 18 13:58:38.289520 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.289140    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldllx\" (UniqueName: \"kubernetes.io/projected/c3f89f2d-89db-42ca-8153-229a22508c4d-kube-api-access-ldllx\") pod \"payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck\" (UID: \"c3f89f2d-89db-42ca-8153-229a22508c4d\") " pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"
Sep 18 13:58:38.289520 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.289194    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ccff\" (UniqueName: \"kubernetes.io/projected/f75de0ea-5b81-4f12-a58c-38e752eaa14c-kube-api-access-6ccff\") pod \"payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b\" (UID: \"f75de0ea-5b81-4f12-a58c-38e752eaa14c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"
Sep 18 13:58:38.289520 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.289336    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f89f2d-89db-42ca-8153-229a22508c4d-plugins-config-volume\") pod \"payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck\" (UID: \"c3f89f2d-89db-42ca-8153-229a22508c4d\") " pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"
Sep 18 13:58:38.291596 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.291524    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f89f2d-89db-42ca-8153-229a22508c4d-plugins-config-volume\") pod \"payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck\" (UID: \"c3f89f2d-89db-42ca-8153-229a22508c4d\") " pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"
Sep 18 13:58:38.300622 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.300594    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldllx\" (UniqueName: \"kubernetes.io/projected/c3f89f2d-89db-42ca-8153-229a22508c4d-kube-api-access-ldllx\") pod \"payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck\" (UID: \"c3f89f2d-89db-42ca-8153-229a22508c4d\") " pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"
Sep 18 13:58:38.387479 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.387444    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht" event={"ID":"e3ef3b5e-2d17-41cf-8b44-1892903435e2","Type":"ContainerStarted","Data":"188bebe98bbd22c9988e635567c8739c152fd1fdc4a9574df5f8dea5d584f278"}
Sep 18 13:58:38.387671 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.387499    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht" event={"ID":"e3ef3b5e-2d17-41cf-8b44-1892903435e2","Type":"ContainerStarted","Data":"7f3ed4860dc2a20f3b7ef03c4d565eb8a793e0b0ceb17fd1d9acb9d576dfa1e0"}
Sep 18 13:58:38.387671 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.387571    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"
Sep 18 13:58:38.389350 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.389323    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc" event={"ID":"a06c46fb-97e2-4ffd-8d04-ef2252a23030","Type":"ContainerStarted","Data":"a1dd5e15485f29e186e8b7ff5beb929f7336e746af7755a021bfcd5cdfa33cba"}
Sep 18 13:58:38.389674 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.389368    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc" event={"ID":"a06c46fb-97e2-4ffd-8d04-ef2252a23030","Type":"ContainerStarted","Data":"998fd31fd05eb356cb5312ad1289baf6fd898f581efd32d468026173050404c9"}
Sep 18 13:58:38.389674 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.389413    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"
Sep 18 13:58:38.390793 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.390288    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ccff\" (UniqueName: \"kubernetes.io/projected/f75de0ea-5b81-4f12-a58c-38e752eaa14c-kube-api-access-6ccff\") pod \"payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b\" (UID: \"f75de0ea-5b81-4f12-a58c-38e752eaa14c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"
Sep 18 13:58:38.390793 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.390412    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f75de0ea-5b81-4f12-a58c-38e752eaa14c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b\" (UID: \"f75de0ea-5b81-4f12-a58c-38e752eaa14c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"
Sep 18 13:58:38.391360 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.391334    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f75de0ea-5b81-4f12-a58c-38e752eaa14c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b\" (UID: \"f75de0ea-5b81-4f12-a58c-38e752eaa14c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"
Sep 18 13:58:38.400865 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.400842    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ccff\" (UniqueName: \"kubernetes.io/projected/f75de0ea-5b81-4f12-a58c-38e752eaa14c-kube-api-access-6ccff\") pod \"payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b\" (UID: \"f75de0ea-5b81-4f12-a58c-38e752eaa14c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"
Sep 18 13:58:38.407176 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.407133    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht" podStartSLOduration=1.4071203429999999 podStartE2EDuration="1.407120343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:38.404596002 +0000 UTC m=+854.339561146" watchObservedRunningTime="2026-09-18 13:58:38.407120343 +0000 UTC m=+854.342085487"
Sep 18 13:58:38.424379 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.424330    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc" podStartSLOduration=2.424316355 podStartE2EDuration="2.424316355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:38.422001948 +0000 UTC m=+854.356967097" watchObservedRunningTime="2026-09-18 13:58:38.424316355 +0000 UTC m=+854.359281499"
Sep 18 13:58:38.474624 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.474589    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"
Sep 18 13:58:38.510926 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.510900    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"]
Sep 18 13:58:38.511945 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:58:38.511911    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf11e73_504b_4815_a844_60e7afc4551d.slice/crio-862a5c34e18edb3764db8131073b1ad1035c6662970c21f8310121d42da1a149 WatchSource:0}: Error finding container 862a5c34e18edb3764db8131073b1ad1035c6662970c21f8310121d42da1a149: Status 404 returned error can't find the container with id 862a5c34e18edb3764db8131073b1ad1035c6662970c21f8310121d42da1a149
Sep 18 13:58:38.549699 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.549672    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"
Sep 18 13:58:38.771191 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.771164    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"]
Sep 18 13:58:38.772638 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:58:38.772610    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f89f2d_89db_42ca_8153_229a22508c4d.slice/crio-00d473195d727d237f532a5a2a88ce505d2aafa210f303c4cd973842963d110d WatchSource:0}: Error finding container 00d473195d727d237f532a5a2a88ce505d2aafa210f303c4cd973842963d110d: Status 404 returned error can't find the container with id 00d473195d727d237f532a5a2a88ce505d2aafa210f303c4cd973842963d110d
Sep 18 13:58:38.842844 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:38.842816    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"]
Sep 18 13:58:38.843838 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:58:38.843809    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf75de0ea_5b81_4f12_a58c_38e752eaa14c.slice/crio-db6dd71967eb9223406f048da6e7165acc85780bea23ba6228deee6a93e08133 WatchSource:0}: Error finding container db6dd71967eb9223406f048da6e7165acc85780bea23ba6228deee6a93e08133: Status 404 returned error can't find the container with id db6dd71967eb9223406f048da6e7165acc85780bea23ba6228deee6a93e08133
Sep 18 13:58:39.321724 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:39.321606    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"
Sep 18 13:58:39.398181 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:39.398109    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf" event={"ID":"3bf11e73-504b-4815-a844-60e7afc4551d","Type":"ContainerStarted","Data":"ab49f84bd324a104702cf6e5d662600281c23713419b56592f377a2ad020680d"}
Sep 18 13:58:39.398359 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:39.398307    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:39.398359 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:39.398352    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf" event={"ID":"3bf11e73-504b-4815-a844-60e7afc4551d","Type":"ContainerStarted","Data":"862a5c34e18edb3764db8131073b1ad1035c6662970c21f8310121d42da1a149"}
Sep 18 13:58:39.400027 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:39.400002    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b" event={"ID":"f75de0ea-5b81-4f12-a58c-38e752eaa14c","Type":"ContainerStarted","Data":"a4c22f3e4c272c2a8c78f7af9400cb9eb4c5463b2d8dd4365b78c2dbe8e258d9"}
Sep 18 13:58:39.400189 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:39.400041    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b" event={"ID":"f75de0ea-5b81-4f12-a58c-38e752eaa14c","Type":"ContainerStarted","Data":"db6dd71967eb9223406f048da6e7165acc85780bea23ba6228deee6a93e08133"}
Sep 18 13:58:39.400189 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:39.400077    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"
Sep 18 13:58:39.401788 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:39.401761    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck" event={"ID":"c3f89f2d-89db-42ca-8153-229a22508c4d","Type":"ContainerStarted","Data":"756473ab971d0b32dd9cbf875e7bc4296fc5747de84966c87962febf2b1c947e"}
Sep 18 13:58:39.401936 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:39.401804    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck" event={"ID":"c3f89f2d-89db-42ca-8153-229a22508c4d","Type":"ContainerStarted","Data":"00d473195d727d237f532a5a2a88ce505d2aafa210f303c4cd973842963d110d"}
Sep 18 13:58:39.430911 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:39.430853    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf" podStartSLOduration=2.092360094 podStartE2EDuration="2.430832722s" totalImagesPullingTime="338.472628ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:39.427874594 +0000 UTC m=+855.362839739" watchObservedRunningTime="2026-09-18 13:58:39.430832722 +0000 UTC m=+855.365797867"
Sep 18 13:58:39.452374 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:39.452327    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck" podStartSLOduration=1.452312011 podStartE2EDuration="1.452312011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:39.44970281 +0000 UTC m=+855.384667961" watchObservedRunningTime="2026-09-18 13:58:39.452312011 +0000 UTC m=+855.387277156"
Sep 18 13:58:39.477901 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:39.477847    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b" podStartSLOduration=1.477830123 podStartE2EDuration="1.477830123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:39.477370766 +0000 UTC m=+855.412335913" watchObservedRunningTime="2026-09-18 13:58:39.477830123 +0000 UTC m=+855.412795272"
Sep 18 13:58:40.408146 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:40.408113    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"
Sep 18 13:58:43.394871 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:43.394841    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 13:58:45.414743 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:45.414708    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:58:49.404055 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:49.404022    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"
Sep 18 13:58:49.404430 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:49.404333    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"
Sep 18 13:58:50.410986 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:50.410956    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"
Sep 18 13:58:51.416629 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:58:51.416599    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"
Sep 18 13:59:04.924593 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:04.924557    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"]
Sep 18 13:59:04.925101 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:04.924832    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf" podUID="3bf11e73-504b-4815-a844-60e7afc4551d" containerName="maas-api" containerID="cri-o://ab49f84bd324a104702cf6e5d662600281c23713419b56592f377a2ad020680d" gracePeriod=30
Sep 18 13:59:04.993069 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:04.993035    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"]
Sep 18 13:59:04.993328 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:04.993294    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck" podUID="c3f89f2d-89db-42ca-8153-229a22508c4d" containerName="payload-processing" containerID="cri-o://756473ab971d0b32dd9cbf875e7bc4296fc5747de84966c87962febf2b1c947e" gracePeriod=30
Sep 18 13:59:05.001050 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.001023    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"]
Sep 18 13:59:05.001280 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.001240    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b" podUID="f75de0ea-5b81-4f12-a58c-38e752eaa14c" containerName="payload-pre-processing" containerID="cri-o://a4c22f3e4c272c2a8c78f7af9400cb9eb4c5463b2d8dd4365b78c2dbe8e258d9" gracePeriod=30
Sep 18 13:59:05.216225 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.216187    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:59:05.309547 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.309509    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"
Sep 18 13:59:05.318701 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.318675    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"
Sep 18 13:59:05.341952 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.341735    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8659\" (UniqueName: \"kubernetes.io/projected/3bf11e73-504b-4815-a844-60e7afc4551d-kube-api-access-l8659\") pod \"3bf11e73-504b-4815-a844-60e7afc4551d\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") "
Sep 18 13:59:05.341952 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.341793    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf11e73-504b-4815-a844-60e7afc4551d-maas-api-tls\") pod \"3bf11e73-504b-4815-a844-60e7afc4551d\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") "
Sep 18 13:59:05.341952 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.341823    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf11e73-504b-4815-a844-60e7afc4551d-openshift-service-ca\") pod \"3bf11e73-504b-4815-a844-60e7afc4551d\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") "
Sep 18 13:59:05.341952 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.341883    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf11e73-504b-4815-a844-60e7afc4551d-metrics-certs\") pod \"3bf11e73-504b-4815-a844-60e7afc4551d\" (UID: \"3bf11e73-504b-4815-a844-60e7afc4551d\") "
Sep 18 13:59:05.343194 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.342764    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf11e73-504b-4815-a844-60e7afc4551d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3bf11e73-504b-4815-a844-60e7afc4551d" (UID: "3bf11e73-504b-4815-a844-60e7afc4551d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:59:05.346103 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.346057    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf11e73-504b-4815-a844-60e7afc4551d-kube-api-access-l8659" (OuterVolumeSpecName: "kube-api-access-l8659") pod "3bf11e73-504b-4815-a844-60e7afc4551d" (UID: "3bf11e73-504b-4815-a844-60e7afc4551d"). InnerVolumeSpecName "kube-api-access-l8659". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:59:05.346224 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.346152    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf11e73-504b-4815-a844-60e7afc4551d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3bf11e73-504b-4815-a844-60e7afc4551d" (UID: "3bf11e73-504b-4815-a844-60e7afc4551d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:59:05.346428 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.346405    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf11e73-504b-4815-a844-60e7afc4551d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3bf11e73-504b-4815-a844-60e7afc4551d" (UID: "3bf11e73-504b-4815-a844-60e7afc4551d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:59:05.442635 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.442602    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f75de0ea-5b81-4f12-a58c-38e752eaa14c-plugins-config-volume\") pod \"f75de0ea-5b81-4f12-a58c-38e752eaa14c\" (UID: \"f75de0ea-5b81-4f12-a58c-38e752eaa14c\") "
Sep 18 13:59:05.442788 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.442660    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ccff\" (UniqueName: \"kubernetes.io/projected/f75de0ea-5b81-4f12-a58c-38e752eaa14c-kube-api-access-6ccff\") pod \"f75de0ea-5b81-4f12-a58c-38e752eaa14c\" (UID: \"f75de0ea-5b81-4f12-a58c-38e752eaa14c\") "
Sep 18 13:59:05.442788 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.442772    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f89f2d-89db-42ca-8153-229a22508c4d-plugins-config-volume\") pod \"c3f89f2d-89db-42ca-8153-229a22508c4d\" (UID: \"c3f89f2d-89db-42ca-8153-229a22508c4d\") "
Sep 18 13:59:05.442896 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.442808    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldllx\" (UniqueName: \"kubernetes.io/projected/c3f89f2d-89db-42ca-8153-229a22508c4d-kube-api-access-ldllx\") pod \"c3f89f2d-89db-42ca-8153-229a22508c4d\" (UID: \"c3f89f2d-89db-42ca-8153-229a22508c4d\") "
Sep 18 13:59:05.443056 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.443017    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f75de0ea-5b81-4f12-a58c-38e752eaa14c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f75de0ea-5b81-4f12-a58c-38e752eaa14c" (UID: "f75de0ea-5b81-4f12-a58c-38e752eaa14c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:59:05.443147 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.443057    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8659\" (UniqueName: \"kubernetes.io/projected/3bf11e73-504b-4815-a844-60e7afc4551d-kube-api-access-l8659\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:59:05.443147 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.443075    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf11e73-504b-4815-a844-60e7afc4551d-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:59:05.443147 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.443091    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf11e73-504b-4815-a844-60e7afc4551d-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:59:05.443147 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.443105    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf11e73-504b-4815-a844-60e7afc4551d-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:59:05.443360 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.443322    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f89f2d-89db-42ca-8153-229a22508c4d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3f89f2d-89db-42ca-8153-229a22508c4d" (UID: "c3f89f2d-89db-42ca-8153-229a22508c4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:59:05.445235 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.445200    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f89f2d-89db-42ca-8153-229a22508c4d-kube-api-access-ldllx" (OuterVolumeSpecName: "kube-api-access-ldllx") pod "c3f89f2d-89db-42ca-8153-229a22508c4d" (UID: "c3f89f2d-89db-42ca-8153-229a22508c4d"). InnerVolumeSpecName "kube-api-access-ldllx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:59:05.445339 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.445312    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f75de0ea-5b81-4f12-a58c-38e752eaa14c-kube-api-access-6ccff" (OuterVolumeSpecName: "kube-api-access-6ccff") pod "f75de0ea-5b81-4f12-a58c-38e752eaa14c" (UID: "f75de0ea-5b81-4f12-a58c-38e752eaa14c"). InnerVolumeSpecName "kube-api-access-6ccff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:59:05.544648 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.544609    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f75de0ea-5b81-4f12-a58c-38e752eaa14c-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:59:05.544648 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.544640    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ccff\" (UniqueName: \"kubernetes.io/projected/f75de0ea-5b81-4f12-a58c-38e752eaa14c-kube-api-access-6ccff\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:59:05.544648 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.544650    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f89f2d-89db-42ca-8153-229a22508c4d-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:59:05.544874 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.544659    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldllx\" (UniqueName: \"kubernetes.io/projected/c3f89f2d-89db-42ca-8153-229a22508c4d-kube-api-access-ldllx\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:59:05.562797 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.562768    2572 generic.go:356] "Generic (PLEG): container finished" podID="f75de0ea-5b81-4f12-a58c-38e752eaa14c" containerID="a4c22f3e4c272c2a8c78f7af9400cb9eb4c5463b2d8dd4365b78c2dbe8e258d9" exitCode=0
Sep 18 13:59:05.562982 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.562822    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"
Sep 18 13:59:05.562982 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.562861    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b" event={"ID":"f75de0ea-5b81-4f12-a58c-38e752eaa14c","Type":"ContainerDied","Data":"a4c22f3e4c272c2a8c78f7af9400cb9eb4c5463b2d8dd4365b78c2dbe8e258d9"}
Sep 18 13:59:05.562982 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.562910    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b" event={"ID":"f75de0ea-5b81-4f12-a58c-38e752eaa14c","Type":"ContainerDied","Data":"db6dd71967eb9223406f048da6e7165acc85780bea23ba6228deee6a93e08133"}
Sep 18 13:59:05.562982 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.562932    2572 scope.go:117] "RemoveContainer" containerID="a4c22f3e4c272c2a8c78f7af9400cb9eb4c5463b2d8dd4365b78c2dbe8e258d9"
Sep 18 13:59:05.564393 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.564371    2572 generic.go:356] "Generic (PLEG): container finished" podID="c3f89f2d-89db-42ca-8153-229a22508c4d" containerID="756473ab971d0b32dd9cbf875e7bc4296fc5747de84966c87962febf2b1c947e" exitCode=0
Sep 18 13:59:05.564564 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.564399    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"
Sep 18 13:59:05.564564 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.564445    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck" event={"ID":"c3f89f2d-89db-42ca-8153-229a22508c4d","Type":"ContainerDied","Data":"756473ab971d0b32dd9cbf875e7bc4296fc5747de84966c87962febf2b1c947e"}
Sep 18 13:59:05.564564 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.564491    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck" event={"ID":"c3f89f2d-89db-42ca-8153-229a22508c4d","Type":"ContainerDied","Data":"00d473195d727d237f532a5a2a88ce505d2aafa210f303c4cd973842963d110d"}
Sep 18 13:59:05.565942 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.565882    2572 generic.go:356] "Generic (PLEG): container finished" podID="3bf11e73-504b-4815-a844-60e7afc4551d" containerID="ab49f84bd324a104702cf6e5d662600281c23713419b56592f377a2ad020680d" exitCode=0
Sep 18 13:59:05.565942 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.565911    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf" event={"ID":"3bf11e73-504b-4815-a844-60e7afc4551d","Type":"ContainerDied","Data":"ab49f84bd324a104702cf6e5d662600281c23713419b56592f377a2ad020680d"}
Sep 18 13:59:05.565942 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.565926    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"
Sep 18 13:59:05.566154 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.565959    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf" event={"ID":"3bf11e73-504b-4815-a844-60e7afc4551d","Type":"ContainerDied","Data":"862a5c34e18edb3764db8131073b1ad1035c6662970c21f8310121d42da1a149"}
Sep 18 13:59:05.574453 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.574435    2572 scope.go:117] "RemoveContainer" containerID="a4c22f3e4c272c2a8c78f7af9400cb9eb4c5463b2d8dd4365b78c2dbe8e258d9"
Sep 18 13:59:05.574768 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:59:05.574746    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c22f3e4c272c2a8c78f7af9400cb9eb4c5463b2d8dd4365b78c2dbe8e258d9\": container with ID starting with a4c22f3e4c272c2a8c78f7af9400cb9eb4c5463b2d8dd4365b78c2dbe8e258d9 not found: ID does not exist" containerID="a4c22f3e4c272c2a8c78f7af9400cb9eb4c5463b2d8dd4365b78c2dbe8e258d9"
Sep 18 13:59:05.574842 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.574779    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c22f3e4c272c2a8c78f7af9400cb9eb4c5463b2d8dd4365b78c2dbe8e258d9"} err="failed to get container status \"a4c22f3e4c272c2a8c78f7af9400cb9eb4c5463b2d8dd4365b78c2dbe8e258d9\": rpc error: code = NotFound desc = could not find container \"a4c22f3e4c272c2a8c78f7af9400cb9eb4c5463b2d8dd4365b78c2dbe8e258d9\": container with ID starting with a4c22f3e4c272c2a8c78f7af9400cb9eb4c5463b2d8dd4365b78c2dbe8e258d9 not found: ID does not exist"
Sep 18 13:59:05.574842 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.574802    2572 scope.go:117] "RemoveContainer" containerID="756473ab971d0b32dd9cbf875e7bc4296fc5747de84966c87962febf2b1c947e"
Sep 18 13:59:05.583172 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.583155    2572 scope.go:117] "RemoveContainer" containerID="756473ab971d0b32dd9cbf875e7bc4296fc5747de84966c87962febf2b1c947e"
Sep 18 13:59:05.583432 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:59:05.583410    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756473ab971d0b32dd9cbf875e7bc4296fc5747de84966c87962febf2b1c947e\": container with ID starting with 756473ab971d0b32dd9cbf875e7bc4296fc5747de84966c87962febf2b1c947e not found: ID does not exist" containerID="756473ab971d0b32dd9cbf875e7bc4296fc5747de84966c87962febf2b1c947e"
Sep 18 13:59:05.583527 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.583434    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756473ab971d0b32dd9cbf875e7bc4296fc5747de84966c87962febf2b1c947e"} err="failed to get container status \"756473ab971d0b32dd9cbf875e7bc4296fc5747de84966c87962febf2b1c947e\": rpc error: code = NotFound desc = could not find container \"756473ab971d0b32dd9cbf875e7bc4296fc5747de84966c87962febf2b1c947e\": container with ID starting with 756473ab971d0b32dd9cbf875e7bc4296fc5747de84966c87962febf2b1c947e not found: ID does not exist"
Sep 18 13:59:05.583527 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.583450    2572 scope.go:117] "RemoveContainer" containerID="ab49f84bd324a104702cf6e5d662600281c23713419b56592f377a2ad020680d"
Sep 18 13:59:05.591447 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.591422    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"]
Sep 18 13:59:05.593065 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.593040    2572 scope.go:117] "RemoveContainer" containerID="ab49f84bd324a104702cf6e5d662600281c23713419b56592f377a2ad020680d"
Sep 18 13:59:05.593357 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:59:05.593339    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab49f84bd324a104702cf6e5d662600281c23713419b56592f377a2ad020680d\": container with ID starting with ab49f84bd324a104702cf6e5d662600281c23713419b56592f377a2ad020680d not found: ID does not exist" containerID="ab49f84bd324a104702cf6e5d662600281c23713419b56592f377a2ad020680d"
Sep 18 13:59:05.593411 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.593362    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab49f84bd324a104702cf6e5d662600281c23713419b56592f377a2ad020680d"} err="failed to get container status \"ab49f84bd324a104702cf6e5d662600281c23713419b56592f377a2ad020680d\": rpc error: code = NotFound desc = could not find container \"ab49f84bd324a104702cf6e5d662600281c23713419b56592f377a2ad020680d\": container with ID starting with ab49f84bd324a104702cf6e5d662600281c23713419b56592f377a2ad020680d not found: ID does not exist"
Sep 18 13:59:05.594613 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.594590    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-57fc5a8f-6d8d577686-96d2b"]
Sep 18 13:59:05.604858 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.604834    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"]
Sep 18 13:59:05.609492 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.609466    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57fc5a8f-649978494d-8w4zf"]
Sep 18 13:59:05.621882 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.621858    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"]
Sep 18 13:59:05.627501 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:05.627481    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57fc5a8f-7b64f6895c-t8tck"]
Sep 18 13:59:06.197921 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:06.197844    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf11e73-504b-4815-a844-60e7afc4551d" path="/var/lib/kubelet/pods/3bf11e73-504b-4815-a844-60e7afc4551d/volumes"
Sep 18 13:59:06.198246 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:06.198195    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3f89f2d-89db-42ca-8153-229a22508c4d" path="/var/lib/kubelet/pods/c3f89f2d-89db-42ca-8153-229a22508c4d/volumes"
Sep 18 13:59:06.198506 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:06.198492    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f75de0ea-5b81-4f12-a58c-38e752eaa14c" path="/var/lib/kubelet/pods/f75de0ea-5b81-4f12-a58c-38e752eaa14c/volumes"
Sep 18 13:59:11.532964 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.532875    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"]
Sep 18 13:59:11.533521 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.533337    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f89f2d-89db-42ca-8153-229a22508c4d" containerName="payload-processing"
Sep 18 13:59:11.533521 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.533351    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f89f2d-89db-42ca-8153-229a22508c4d" containerName="payload-processing"
Sep 18 13:59:11.533521 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.533375    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf11e73-504b-4815-a844-60e7afc4551d" containerName="maas-api"
Sep 18 13:59:11.533521 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.533381    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf11e73-504b-4815-a844-60e7afc4551d" containerName="maas-api"
Sep 18 13:59:11.533521 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.533390    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f75de0ea-5b81-4f12-a58c-38e752eaa14c" containerName="payload-pre-processing"
Sep 18 13:59:11.533521 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.533396    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f75de0ea-5b81-4f12-a58c-38e752eaa14c" containerName="payload-pre-processing"
Sep 18 13:59:11.533521 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.533453    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f89f2d-89db-42ca-8153-229a22508c4d" containerName="payload-processing"
Sep 18 13:59:11.533521 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.533465    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf11e73-504b-4815-a844-60e7afc4551d" containerName="maas-api"
Sep 18 13:59:11.533521 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.533472    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f75de0ea-5b81-4f12-a58c-38e752eaa14c" containerName="payload-pre-processing"
Sep 18 13:59:11.538393 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.538372    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:11.541328 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.541306    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-850ecce6-serving-cert\""
Sep 18 13:59:11.553198 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.553177    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"]
Sep 18 13:59:11.708610 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.708575    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdrw2\" (UniqueName: \"kubernetes.io/projected/d05f757d-340f-49bf-b2a4-997317a63ae9-kube-api-access-wdrw2\") pod \"maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl\" (UID: \"d05f757d-340f-49bf-b2a4-997317a63ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:11.708800 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.708627    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05f757d-340f-49bf-b2a4-997317a63ae9-openshift-service-ca\") pod \"maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl\" (UID: \"d05f757d-340f-49bf-b2a4-997317a63ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:11.708800 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.708673    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05f757d-340f-49bf-b2a4-997317a63ae9-maas-api-tls\") pod \"maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl\" (UID: \"d05f757d-340f-49bf-b2a4-997317a63ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:11.708800 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.708782    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05f757d-340f-49bf-b2a4-997317a63ae9-metrics-certs\") pod \"maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl\" (UID: \"d05f757d-340f-49bf-b2a4-997317a63ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:11.810159 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.810074    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05f757d-340f-49bf-b2a4-997317a63ae9-metrics-certs\") pod \"maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl\" (UID: \"d05f757d-340f-49bf-b2a4-997317a63ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:11.810159 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.810145    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdrw2\" (UniqueName: \"kubernetes.io/projected/d05f757d-340f-49bf-b2a4-997317a63ae9-kube-api-access-wdrw2\") pod \"maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl\" (UID: \"d05f757d-340f-49bf-b2a4-997317a63ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:11.810361 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.810196    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05f757d-340f-49bf-b2a4-997317a63ae9-openshift-service-ca\") pod \"maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl\" (UID: \"d05f757d-340f-49bf-b2a4-997317a63ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:11.810361 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.810222    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05f757d-340f-49bf-b2a4-997317a63ae9-maas-api-tls\") pod \"maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl\" (UID: \"d05f757d-340f-49bf-b2a4-997317a63ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:11.810897 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.810869    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05f757d-340f-49bf-b2a4-997317a63ae9-openshift-service-ca\") pod \"maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl\" (UID: \"d05f757d-340f-49bf-b2a4-997317a63ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:11.812877 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.812849    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05f757d-340f-49bf-b2a4-997317a63ae9-metrics-certs\") pod \"maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl\" (UID: \"d05f757d-340f-49bf-b2a4-997317a63ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:11.813008 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.812921    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05f757d-340f-49bf-b2a4-997317a63ae9-maas-api-tls\") pod \"maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl\" (UID: \"d05f757d-340f-49bf-b2a4-997317a63ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:11.819366 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.819343    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdrw2\" (UniqueName: \"kubernetes.io/projected/d05f757d-340f-49bf-b2a4-997317a63ae9-kube-api-access-wdrw2\") pod \"maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl\" (UID: \"d05f757d-340f-49bf-b2a4-997317a63ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:11.849091 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:11.849066    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:12.128453 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.128416    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"]
Sep 18 13:59:12.293008 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.292976    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"]
Sep 18 13:59:12.296692 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.296671    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"
Sep 18 13:59:12.298976 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.298955    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-850ecce6\""
Sep 18 13:59:12.299586 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.299570    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-850ecce6-dockercfg-m86d2\""
Sep 18 13:59:12.310604 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.310578    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"]
Sep 18 13:59:12.415022 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.414934    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52587b9-021a-4b9a-a395-4aa86abd712a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn\" (UID: \"d52587b9-021a-4b9a-a395-4aa86abd712a\") " pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"
Sep 18 13:59:12.415022 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.414987    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj5ph\" (UniqueName: \"kubernetes.io/projected/d52587b9-021a-4b9a-a395-4aa86abd712a-kube-api-access-xj5ph\") pod \"payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn\" (UID: \"d52587b9-021a-4b9a-a395-4aa86abd712a\") " pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"
Sep 18 13:59:12.515894 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.515774    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52587b9-021a-4b9a-a395-4aa86abd712a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn\" (UID: \"d52587b9-021a-4b9a-a395-4aa86abd712a\") " pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"
Sep 18 13:59:12.515894 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.515833    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj5ph\" (UniqueName: \"kubernetes.io/projected/d52587b9-021a-4b9a-a395-4aa86abd712a-kube-api-access-xj5ph\") pod \"payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn\" (UID: \"d52587b9-021a-4b9a-a395-4aa86abd712a\") " pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"
Sep 18 13:59:12.516586 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.516563    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52587b9-021a-4b9a-a395-4aa86abd712a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn\" (UID: \"d52587b9-021a-4b9a-a395-4aa86abd712a\") " pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"
Sep 18 13:59:12.525345 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.525319    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj5ph\" (UniqueName: \"kubernetes.io/projected/d52587b9-021a-4b9a-a395-4aa86abd712a-kube-api-access-xj5ph\") pod \"payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn\" (UID: \"d52587b9-021a-4b9a-a395-4aa86abd712a\") " pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"
Sep 18 13:59:12.609880 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.609847    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"
Sep 18 13:59:12.613058 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.613027    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl" event={"ID":"d05f757d-340f-49bf-b2a4-997317a63ae9","Type":"ContainerStarted","Data":"36db6e0e369b2df84aa66b045d1af379269f89794743d2f28cef34a9860e9250"}
Sep 18 13:59:12.613160 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.613090    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl" event={"ID":"d05f757d-340f-49bf-b2a4-997317a63ae9","Type":"ContainerStarted","Data":"9948f4d4a6f5cc99518edbf9b177c78dad3878982713b85069125d1991cb8053"}
Sep 18 13:59:12.613870 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.613848    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:12.634925 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.634875    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl" podStartSLOduration=1.310923235 podStartE2EDuration="1.634854654s" totalImagesPullingTime="323.931419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:12.632323877 +0000 UTC m=+888.567289023" watchObservedRunningTime="2026-09-18 13:59:12.634854654 +0000 UTC m=+888.569819801"
Sep 18 13:59:12.888755 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:12.888731    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"]
Sep 18 13:59:12.890634 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:59:12.890604    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52587b9_021a_4b9a_a395_4aa86abd712a.slice/crio-40b833f3f41921e2618eb14078b083fefd3c905d2f1cf51560561e24e6e88f45 WatchSource:0}: Error finding container 40b833f3f41921e2618eb14078b083fefd3c905d2f1cf51560561e24e6e88f45: Status 404 returned error can't find the container with id 40b833f3f41921e2618eb14078b083fefd3c905d2f1cf51560561e24e6e88f45
Sep 18 13:59:13.621829 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:13.621777    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn" event={"ID":"d52587b9-021a-4b9a-a395-4aa86abd712a","Type":"ContainerStarted","Data":"76ed3169ef9af5376b5e6c100b691e1da9aee4e221f6a89cdfa9619b4c7fbdf7"}
Sep 18 13:59:13.622308 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:13.621845    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn" event={"ID":"d52587b9-021a-4b9a-a395-4aa86abd712a","Type":"ContainerStarted","Data":"40b833f3f41921e2618eb14078b083fefd3c905d2f1cf51560561e24e6e88f45"}
Sep 18 13:59:13.622308 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:13.621888    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"
Sep 18 13:59:13.644781 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:13.644736    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn" podStartSLOduration=1.6447238739999999 podStartE2EDuration="1.644723874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:13.643784865 +0000 UTC m=+889.578750002" watchObservedRunningTime="2026-09-18 13:59:13.644723874 +0000 UTC m=+889.579689048"
Sep 18 13:59:19.633353 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:19.633324    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:24.629670 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:24.629637    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"
Sep 18 13:59:26.201577 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:26.201549    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 13:59:26.204411 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:26.204387    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 13:59:41.298673 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.298641    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"]
Sep 18 13:59:41.301269 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.298909    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl" podUID="d05f757d-340f-49bf-b2a4-997317a63ae9" containerName="maas-api" containerID="cri-o://36db6e0e369b2df84aa66b045d1af379269f89794743d2f28cef34a9860e9250" gracePeriod=30
Sep 18 13:59:41.360349 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.360320    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"]
Sep 18 13:59:41.360619 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.360558    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn" podUID="d52587b9-021a-4b9a-a395-4aa86abd712a" containerName="payload-processing" containerID="cri-o://76ed3169ef9af5376b5e6c100b691e1da9aee4e221f6a89cdfa9619b4c7fbdf7" gracePeriod=30
Sep 18 13:59:41.627888 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.627856    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:41.668199 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.668174    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"
Sep 18 13:59:41.787381 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.787343    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05f757d-340f-49bf-b2a4-997317a63ae9-openshift-service-ca\") pod \"d05f757d-340f-49bf-b2a4-997317a63ae9\" (UID: \"d05f757d-340f-49bf-b2a4-997317a63ae9\") "
Sep 18 13:59:41.787585 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.787403    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52587b9-021a-4b9a-a395-4aa86abd712a-plugins-config-volume\") pod \"d52587b9-021a-4b9a-a395-4aa86abd712a\" (UID: \"d52587b9-021a-4b9a-a395-4aa86abd712a\") "
Sep 18 13:59:41.787585 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.787456    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05f757d-340f-49bf-b2a4-997317a63ae9-metrics-certs\") pod \"d05f757d-340f-49bf-b2a4-997317a63ae9\" (UID: \"d05f757d-340f-49bf-b2a4-997317a63ae9\") "
Sep 18 13:59:41.787723 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.787697    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05f757d-340f-49bf-b2a4-997317a63ae9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d05f757d-340f-49bf-b2a4-997317a63ae9" (UID: "d05f757d-340f-49bf-b2a4-997317a63ae9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:59:41.787876 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.787849    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05f757d-340f-49bf-b2a4-997317a63ae9-maas-api-tls\") pod \"d05f757d-340f-49bf-b2a4-997317a63ae9\" (UID: \"d05f757d-340f-49bf-b2a4-997317a63ae9\") "
Sep 18 13:59:41.788010 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.787927    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj5ph\" (UniqueName: \"kubernetes.io/projected/d52587b9-021a-4b9a-a395-4aa86abd712a-kube-api-access-xj5ph\") pod \"d52587b9-021a-4b9a-a395-4aa86abd712a\" (UID: \"d52587b9-021a-4b9a-a395-4aa86abd712a\") "
Sep 18 13:59:41.788010 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.787855    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52587b9-021a-4b9a-a395-4aa86abd712a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d52587b9-021a-4b9a-a395-4aa86abd712a" (UID: "d52587b9-021a-4b9a-a395-4aa86abd712a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:59:41.788010 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.787966    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdrw2\" (UniqueName: \"kubernetes.io/projected/d05f757d-340f-49bf-b2a4-997317a63ae9-kube-api-access-wdrw2\") pod \"d05f757d-340f-49bf-b2a4-997317a63ae9\" (UID: \"d05f757d-340f-49bf-b2a4-997317a63ae9\") "
Sep 18 13:59:41.788288 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.788262    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05f757d-340f-49bf-b2a4-997317a63ae9-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:59:41.788376 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.788294    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52587b9-021a-4b9a-a395-4aa86abd712a-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:59:41.790041 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.790012    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05f757d-340f-49bf-b2a4-997317a63ae9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d05f757d-340f-49bf-b2a4-997317a63ae9" (UID: "d05f757d-340f-49bf-b2a4-997317a63ae9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:59:41.790316 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.790280    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52587b9-021a-4b9a-a395-4aa86abd712a-kube-api-access-xj5ph" (OuterVolumeSpecName: "kube-api-access-xj5ph") pod "d52587b9-021a-4b9a-a395-4aa86abd712a" (UID: "d52587b9-021a-4b9a-a395-4aa86abd712a"). InnerVolumeSpecName "kube-api-access-xj5ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:59:41.790415 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.790320    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05f757d-340f-49bf-b2a4-997317a63ae9-kube-api-access-wdrw2" (OuterVolumeSpecName: "kube-api-access-wdrw2") pod "d05f757d-340f-49bf-b2a4-997317a63ae9" (UID: "d05f757d-340f-49bf-b2a4-997317a63ae9"). InnerVolumeSpecName "kube-api-access-wdrw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:59:41.790722 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.790695    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05f757d-340f-49bf-b2a4-997317a63ae9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d05f757d-340f-49bf-b2a4-997317a63ae9" (UID: "d05f757d-340f-49bf-b2a4-997317a63ae9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:59:41.791747 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.791671    2572 generic.go:356] "Generic (PLEG): container finished" podID="d52587b9-021a-4b9a-a395-4aa86abd712a" containerID="76ed3169ef9af5376b5e6c100b691e1da9aee4e221f6a89cdfa9619b4c7fbdf7" exitCode=0
Sep 18 13:59:41.791884 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.791760    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"
Sep 18 13:59:41.791884 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.791835    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn" event={"ID":"d52587b9-021a-4b9a-a395-4aa86abd712a","Type":"ContainerDied","Data":"76ed3169ef9af5376b5e6c100b691e1da9aee4e221f6a89cdfa9619b4c7fbdf7"}
Sep 18 13:59:41.792004 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.791883    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn" event={"ID":"d52587b9-021a-4b9a-a395-4aa86abd712a","Type":"ContainerDied","Data":"40b833f3f41921e2618eb14078b083fefd3c905d2f1cf51560561e24e6e88f45"}
Sep 18 13:59:41.792004 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.791951    2572 scope.go:117] "RemoveContainer" containerID="76ed3169ef9af5376b5e6c100b691e1da9aee4e221f6a89cdfa9619b4c7fbdf7"
Sep 18 13:59:41.794328 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.794305    2572 generic.go:356] "Generic (PLEG): container finished" podID="d05f757d-340f-49bf-b2a4-997317a63ae9" containerID="36db6e0e369b2df84aa66b045d1af379269f89794743d2f28cef34a9860e9250" exitCode=0
Sep 18 13:59:41.794456 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.794337    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl" event={"ID":"d05f757d-340f-49bf-b2a4-997317a63ae9","Type":"ContainerDied","Data":"36db6e0e369b2df84aa66b045d1af379269f89794743d2f28cef34a9860e9250"}
Sep 18 13:59:41.794456 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.794354    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"
Sep 18 13:59:41.794456 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.794386    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl" event={"ID":"d05f757d-340f-49bf-b2a4-997317a63ae9","Type":"ContainerDied","Data":"9948f4d4a6f5cc99518edbf9b177c78dad3878982713b85069125d1991cb8053"}
Sep 18 13:59:41.803607 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.803480    2572 scope.go:117] "RemoveContainer" containerID="76ed3169ef9af5376b5e6c100b691e1da9aee4e221f6a89cdfa9619b4c7fbdf7"
Sep 18 13:59:41.803919 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:59:41.803838    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ed3169ef9af5376b5e6c100b691e1da9aee4e221f6a89cdfa9619b4c7fbdf7\": container with ID starting with 76ed3169ef9af5376b5e6c100b691e1da9aee4e221f6a89cdfa9619b4c7fbdf7 not found: ID does not exist" containerID="76ed3169ef9af5376b5e6c100b691e1da9aee4e221f6a89cdfa9619b4c7fbdf7"
Sep 18 13:59:41.803919 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.803882    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ed3169ef9af5376b5e6c100b691e1da9aee4e221f6a89cdfa9619b4c7fbdf7"} err="failed to get container status \"76ed3169ef9af5376b5e6c100b691e1da9aee4e221f6a89cdfa9619b4c7fbdf7\": rpc error: code = NotFound desc = could not find container \"76ed3169ef9af5376b5e6c100b691e1da9aee4e221f6a89cdfa9619b4c7fbdf7\": container with ID starting with 76ed3169ef9af5376b5e6c100b691e1da9aee4e221f6a89cdfa9619b4c7fbdf7 not found: ID does not exist"
Sep 18 13:59:41.803919 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.803907    2572 scope.go:117] "RemoveContainer" containerID="36db6e0e369b2df84aa66b045d1af379269f89794743d2f28cef34a9860e9250"
Sep 18 13:59:41.815722 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.815640    2572 scope.go:117] "RemoveContainer" containerID="36db6e0e369b2df84aa66b045d1af379269f89794743d2f28cef34a9860e9250"
Sep 18 13:59:41.815986 ip-10-0-128-141 kubenswrapper[2572]: E0918 13:59:41.815961    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36db6e0e369b2df84aa66b045d1af379269f89794743d2f28cef34a9860e9250\": container with ID starting with 36db6e0e369b2df84aa66b045d1af379269f89794743d2f28cef34a9860e9250 not found: ID does not exist" containerID="36db6e0e369b2df84aa66b045d1af379269f89794743d2f28cef34a9860e9250"
Sep 18 13:59:41.816135 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.815994    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36db6e0e369b2df84aa66b045d1af379269f89794743d2f28cef34a9860e9250"} err="failed to get container status \"36db6e0e369b2df84aa66b045d1af379269f89794743d2f28cef34a9860e9250\": rpc error: code = NotFound desc = could not find container \"36db6e0e369b2df84aa66b045d1af379269f89794743d2f28cef34a9860e9250\": container with ID starting with 36db6e0e369b2df84aa66b045d1af379269f89794743d2f28cef34a9860e9250 not found: ID does not exist"
Sep 18 13:59:41.822628 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.822596    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"]
Sep 18 13:59:41.826708 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.826658    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-850ecce6-754d9b9478-q2bdn"]
Sep 18 13:59:41.834127 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.834106    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"]
Sep 18 13:59:41.838053 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.838028    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-850ecce6-68fcdd5f77-6jkwl"]
Sep 18 13:59:41.889795 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.889772    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05f757d-340f-49bf-b2a4-997317a63ae9-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:59:41.889795 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.889794    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj5ph\" (UniqueName: \"kubernetes.io/projected/d52587b9-021a-4b9a-a395-4aa86abd712a-kube-api-access-xj5ph\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:59:41.889930 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.889804    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdrw2\" (UniqueName: \"kubernetes.io/projected/d05f757d-340f-49bf-b2a4-997317a63ae9-kube-api-access-wdrw2\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:59:41.889930 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:41.889814    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05f757d-340f-49bf-b2a4-997317a63ae9-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 13:59:42.198436 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:42.198404    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d05f757d-340f-49bf-b2a4-997317a63ae9" path="/var/lib/kubelet/pods/d05f757d-340f-49bf-b2a4-997317a63ae9/volumes"
Sep 18 13:59:42.198808 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:42.198794    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d52587b9-021a-4b9a-a395-4aa86abd712a" path="/var/lib/kubelet/pods/d52587b9-021a-4b9a-a395-4aa86abd712a/volumes"
Sep 18 13:59:47.792623 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:47.792590    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"]
Sep 18 13:59:47.793069 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:47.792982    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52587b9-021a-4b9a-a395-4aa86abd712a" containerName="payload-processing"
Sep 18 13:59:47.793069 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:47.792993    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52587b9-021a-4b9a-a395-4aa86abd712a" containerName="payload-processing"
Sep 18 13:59:47.793069 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:47.793012    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d05f757d-340f-49bf-b2a4-997317a63ae9" containerName="maas-api"
Sep 18 13:59:47.793069 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:47.793017    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d05f757d-340f-49bf-b2a4-997317a63ae9" containerName="maas-api"
Sep 18 13:59:47.793198 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:47.793084    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52587b9-021a-4b9a-a395-4aa86abd712a" containerName="payload-processing"
Sep 18 13:59:47.793198 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:47.793095    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d05f757d-340f-49bf-b2a4-997317a63ae9" containerName="maas-api"
Sep 18 13:59:47.796134 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:47.796119    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 13:59:47.801473 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:47.801454    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8a84bc4e-serving-cert\""
Sep 18 13:59:47.841753 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:47.841716    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"]
Sep 18 13:59:47.951794 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:47.951757    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-maas-api-tls\") pod \"maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq\" (UID: \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 13:59:47.951986 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:47.951875    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcqpl\" (UniqueName: \"kubernetes.io/projected/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-kube-api-access-wcqpl\") pod \"maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq\" (UID: \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 13:59:47.951986 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:47.951933    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-metrics-certs\") pod \"maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq\" (UID: \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 13:59:47.951986 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:47.951974    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-openshift-service-ca\") pod \"maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq\" (UID: \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 13:59:48.053091 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.052993    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-maas-api-tls\") pod \"maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq\" (UID: \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 13:59:48.053091 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.053051    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcqpl\" (UniqueName: \"kubernetes.io/projected/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-kube-api-access-wcqpl\") pod \"maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq\" (UID: \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 13:59:48.053091 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.053090    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-metrics-certs\") pod \"maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq\" (UID: \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 13:59:48.053388 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.053122    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-openshift-service-ca\") pod \"maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq\" (UID: \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 13:59:48.053897 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.053876    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-openshift-service-ca\") pod \"maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq\" (UID: \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 13:59:48.055510 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.055484    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-metrics-certs\") pod \"maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq\" (UID: \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 13:59:48.055695 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.055673    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-maas-api-tls\") pod \"maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq\" (UID: \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 13:59:48.062339 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.062313    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcqpl\" (UniqueName: \"kubernetes.io/projected/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-kube-api-access-wcqpl\") pod \"maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq\" (UID: \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 13:59:48.106562 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.106503    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 13:59:48.397041 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.397014    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"]
Sep 18 13:59:48.398714 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:59:48.398679    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ec390b_f5c5_4d7e_9ac4_c60489fab5fe.slice/crio-a3a239465eb5ee0cd6b85ba6edf39639fd64de35c2354f936453b3cde08e4c8f WatchSource:0}: Error finding container a3a239465eb5ee0cd6b85ba6edf39639fd64de35c2354f936453b3cde08e4c8f: Status 404 returned error can't find the container with id a3a239465eb5ee0cd6b85ba6edf39639fd64de35c2354f936453b3cde08e4c8f
Sep 18 13:59:48.400120 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.400100    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:59:48.551361 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.551328    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"]
Sep 18 13:59:48.556061 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.556039    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"
Sep 18 13:59:48.558449 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.558429    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8a84bc4e\""
Sep 18 13:59:48.569229 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.569202    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"]
Sep 18 13:59:48.657315 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.657224    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7\" (UID: \"3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"
Sep 18 13:59:48.657315 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.657308    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6wpm\" (UniqueName: \"kubernetes.io/projected/3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5-kube-api-access-d6wpm\") pod \"payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7\" (UID: \"3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"
Sep 18 13:59:48.758295 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.758261    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6wpm\" (UniqueName: \"kubernetes.io/projected/3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5-kube-api-access-d6wpm\") pod \"payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7\" (UID: \"3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"
Sep 18 13:59:48.758480 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.758360    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7\" (UID: \"3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"
Sep 18 13:59:48.759154 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.759129    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7\" (UID: \"3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"
Sep 18 13:59:48.774677 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.774651    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6wpm\" (UniqueName: \"kubernetes.io/projected/3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5-kube-api-access-d6wpm\") pod \"payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7\" (UID: \"3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"
Sep 18 13:59:48.844065 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.844037    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq" event={"ID":"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe","Type":"ContainerStarted","Data":"a3a239465eb5ee0cd6b85ba6edf39639fd64de35c2354f936453b3cde08e4c8f"}
Sep 18 13:59:48.868119 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:48.868080    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"
Sep 18 13:59:49.163769 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:49.163741    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"]
Sep 18 13:59:49.165683 ip-10-0-128-141 kubenswrapper[2572]: W0918 13:59:49.165653    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3e5ef6_3700_4b0b_ac4f_d83184e0fdd5.slice/crio-ab29312b45b6cbdae4d23ed18ce8a1acb0bcc2db2c2845ead71bb5008692af7e WatchSource:0}: Error finding container ab29312b45b6cbdae4d23ed18ce8a1acb0bcc2db2c2845ead71bb5008692af7e: Status 404 returned error can't find the container with id ab29312b45b6cbdae4d23ed18ce8a1acb0bcc2db2c2845ead71bb5008692af7e
Sep 18 13:59:49.852709 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:49.852666    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq" event={"ID":"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe","Type":"ContainerStarted","Data":"8bd7a78e1d012f5e6b1e3beb0706f777ad2729c8c41cb958c922f5aabb9f81a0"}
Sep 18 13:59:49.853231 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:49.852869    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 13:59:49.854492 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:49.854465    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7" event={"ID":"3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5","Type":"ContainerStarted","Data":"9cd1949257aa82a46d2a68c8b6e37378fb82ef7cc3fa3b0f697afe44e7f88f89"}
Sep 18 13:59:49.854639 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:49.854510    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7" event={"ID":"3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5","Type":"ContainerStarted","Data":"ab29312b45b6cbdae4d23ed18ce8a1acb0bcc2db2c2845ead71bb5008692af7e"}
Sep 18 13:59:49.854639 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:49.854552    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"
Sep 18 13:59:49.873057 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:49.873005    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq" podStartSLOduration=2.437985585 podStartE2EDuration="2.872989721s" totalImagesPullingTime="435.004136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:49.870065363 +0000 UTC m=+925.805030507" watchObservedRunningTime="2026-09-18 13:59:49.872989721 +0000 UTC m=+925.807954866"
Sep 18 13:59:49.889642 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:49.889589    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7" podStartSLOduration=1.889571981 podStartE2EDuration="1.889571981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:49.88807902 +0000 UTC m=+925.823044168" watchObservedRunningTime="2026-09-18 13:59:49.889571981 +0000 UTC m=+925.824537126"
Sep 18 13:59:55.866484 ip-10-0-128-141 kubenswrapper[2572]: I0918 13:59:55.866452    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 14:00:00.862779 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:00.862747    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"
Sep 18 14:00:26.492159 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:26.492121    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"]
Sep 18 14:00:26.492655 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:26.492367    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq" podUID="90ec390b-f5c5-4d7e-9ac4-c60489fab5fe" containerName="maas-api" containerID="cri-o://8bd7a78e1d012f5e6b1e3beb0706f777ad2729c8c41cb958c922f5aabb9f81a0" gracePeriod=30
Sep 18 14:00:26.583400 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:26.583365    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"]
Sep 18 14:00:26.583599 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:26.583581    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7" podUID="3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5" containerName="payload-pre-processing" containerID="cri-o://9cd1949257aa82a46d2a68c8b6e37378fb82ef7cc3fa3b0f697afe44e7f88f89" gracePeriod=30
Sep 18 14:00:26.847195 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:26.847130    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 14:00:26.880488 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:26.880460    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"
Sep 18 14:00:26.918742 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:26.918709    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-maas-api-tls\") pod \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\" (UID: \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\") "
Sep 18 14:00:26.918935 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:26.918866    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-openshift-service-ca\") pod \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\" (UID: \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\") "
Sep 18 14:00:26.918935 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:26.918904    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-metrics-certs\") pod \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\" (UID: \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\") "
Sep 18 14:00:26.919067 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:26.918946    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcqpl\" (UniqueName: \"kubernetes.io/projected/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-kube-api-access-wcqpl\") pod \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\" (UID: \"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe\") "
Sep 18 14:00:26.919454 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:26.919414    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90ec390b-f5c5-4d7e-9ac4-c60489fab5fe" (UID: "90ec390b-f5c5-4d7e-9ac4-c60489fab5fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:00:26.922144 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:26.922089    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90ec390b-f5c5-4d7e-9ac4-c60489fab5fe" (UID: "90ec390b-f5c5-4d7e-9ac4-c60489fab5fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:00:26.922254 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:26.922141    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "90ec390b-f5c5-4d7e-9ac4-c60489fab5fe" (UID: "90ec390b-f5c5-4d7e-9ac4-c60489fab5fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:00:26.922485 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:26.922464    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-kube-api-access-wcqpl" (OuterVolumeSpecName: "kube-api-access-wcqpl") pod "90ec390b-f5c5-4d7e-9ac4-c60489fab5fe" (UID: "90ec390b-f5c5-4d7e-9ac4-c60489fab5fe"). InnerVolumeSpecName "kube-api-access-wcqpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:00:27.020448 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.020405    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6wpm\" (UniqueName: \"kubernetes.io/projected/3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5-kube-api-access-d6wpm\") pod \"3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5\" (UID: \"3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5\") "
Sep 18 14:00:27.020448 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.020473    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5-plugins-config-volume\") pod \"3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5\" (UID: \"3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5\") "
Sep 18 14:00:27.020887 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.020863    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:00:27.020956 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.020892    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:00:27.020956 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.020894    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5" (UID: "3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:00:27.020956 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.020906    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcqpl\" (UniqueName: \"kubernetes.io/projected/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-kube-api-access-wcqpl\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:00:27.020956 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.020952    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:00:27.023397 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.023362    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5-kube-api-access-d6wpm" (OuterVolumeSpecName: "kube-api-access-d6wpm") pod "3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5" (UID: "3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5"). InnerVolumeSpecName "kube-api-access-d6wpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:00:27.076075 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.075987    2572 generic.go:356] "Generic (PLEG): container finished" podID="90ec390b-f5c5-4d7e-9ac4-c60489fab5fe" containerID="8bd7a78e1d012f5e6b1e3beb0706f777ad2729c8c41cb958c922f5aabb9f81a0" exitCode=0
Sep 18 14:00:27.076075 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.076040    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"
Sep 18 14:00:27.076271 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.076075    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq" event={"ID":"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe","Type":"ContainerDied","Data":"8bd7a78e1d012f5e6b1e3beb0706f777ad2729c8c41cb958c922f5aabb9f81a0"}
Sep 18 14:00:27.076271 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.076167    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq" event={"ID":"90ec390b-f5c5-4d7e-9ac4-c60489fab5fe","Type":"ContainerDied","Data":"a3a239465eb5ee0cd6b85ba6edf39639fd64de35c2354f936453b3cde08e4c8f"}
Sep 18 14:00:27.076271 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.076206    2572 scope.go:117] "RemoveContainer" containerID="8bd7a78e1d012f5e6b1e3beb0706f777ad2729c8c41cb958c922f5aabb9f81a0"
Sep 18 14:00:27.077574 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.077550    2572 generic.go:356] "Generic (PLEG): container finished" podID="3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5" containerID="9cd1949257aa82a46d2a68c8b6e37378fb82ef7cc3fa3b0f697afe44e7f88f89" exitCode=0
Sep 18 14:00:27.077702 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.077586    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7" event={"ID":"3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5","Type":"ContainerDied","Data":"9cd1949257aa82a46d2a68c8b6e37378fb82ef7cc3fa3b0f697afe44e7f88f89"}
Sep 18 14:00:27.077702 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.077606    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"
Sep 18 14:00:27.077702 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.077626    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7" event={"ID":"3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5","Type":"ContainerDied","Data":"ab29312b45b6cbdae4d23ed18ce8a1acb0bcc2db2c2845ead71bb5008692af7e"}
Sep 18 14:00:27.086775 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.086755    2572 scope.go:117] "RemoveContainer" containerID="8bd7a78e1d012f5e6b1e3beb0706f777ad2729c8c41cb958c922f5aabb9f81a0"
Sep 18 14:00:27.087049 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:00:27.087029    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd7a78e1d012f5e6b1e3beb0706f777ad2729c8c41cb958c922f5aabb9f81a0\": container with ID starting with 8bd7a78e1d012f5e6b1e3beb0706f777ad2729c8c41cb958c922f5aabb9f81a0 not found: ID does not exist" containerID="8bd7a78e1d012f5e6b1e3beb0706f777ad2729c8c41cb958c922f5aabb9f81a0"
Sep 18 14:00:27.087101 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.087058    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd7a78e1d012f5e6b1e3beb0706f777ad2729c8c41cb958c922f5aabb9f81a0"} err="failed to get container status \"8bd7a78e1d012f5e6b1e3beb0706f777ad2729c8c41cb958c922f5aabb9f81a0\": rpc error: code = NotFound desc = could not find container \"8bd7a78e1d012f5e6b1e3beb0706f777ad2729c8c41cb958c922f5aabb9f81a0\": container with ID starting with 8bd7a78e1d012f5e6b1e3beb0706f777ad2729c8c41cb958c922f5aabb9f81a0 not found: ID does not exist"
Sep 18 14:00:27.087101 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.087076    2572 scope.go:117] "RemoveContainer" containerID="9cd1949257aa82a46d2a68c8b6e37378fb82ef7cc3fa3b0f697afe44e7f88f89"
Sep 18 14:00:27.095384 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.095357    2572 scope.go:117] "RemoveContainer" containerID="9cd1949257aa82a46d2a68c8b6e37378fb82ef7cc3fa3b0f697afe44e7f88f89"
Sep 18 14:00:27.095754 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:00:27.095731    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd1949257aa82a46d2a68c8b6e37378fb82ef7cc3fa3b0f697afe44e7f88f89\": container with ID starting with 9cd1949257aa82a46d2a68c8b6e37378fb82ef7cc3fa3b0f697afe44e7f88f89 not found: ID does not exist" containerID="9cd1949257aa82a46d2a68c8b6e37378fb82ef7cc3fa3b0f697afe44e7f88f89"
Sep 18 14:00:27.095836 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.095759    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd1949257aa82a46d2a68c8b6e37378fb82ef7cc3fa3b0f697afe44e7f88f89"} err="failed to get container status \"9cd1949257aa82a46d2a68c8b6e37378fb82ef7cc3fa3b0f697afe44e7f88f89\": rpc error: code = NotFound desc = could not find container \"9cd1949257aa82a46d2a68c8b6e37378fb82ef7cc3fa3b0f697afe44e7f88f89\": container with ID starting with 9cd1949257aa82a46d2a68c8b6e37378fb82ef7cc3fa3b0f697afe44e7f88f89 not found: ID does not exist"
Sep 18 14:00:27.105370 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.105348    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"]
Sep 18 14:00:27.107780 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.107762    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a84bc4e-ccb9cf9b-fldhq"]
Sep 18 14:00:27.121957 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.121928    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6wpm\" (UniqueName: \"kubernetes.io/projected/3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5-kube-api-access-d6wpm\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:00:27.122086 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.121960    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:00:27.127562 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.127521    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"]
Sep 18 14:00:27.131463 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:27.131443    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a84bc4e-5978c48b5d-mplc7"]
Sep 18 14:00:28.198962 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:28.198927    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5" path="/var/lib/kubelet/pods/3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5/volumes"
Sep 18 14:00:28.199543 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:28.199508    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90ec390b-f5c5-4d7e-9ac4-c60489fab5fe" path="/var/lib/kubelet/pods/90ec390b-f5c5-4d7e-9ac4-c60489fab5fe/volumes"
Sep 18 14:00:36.173944 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.173912    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"]
Sep 18 14:00:36.174413 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.174311    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5" containerName="payload-pre-processing"
Sep 18 14:00:36.174413 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.174322    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5" containerName="payload-pre-processing"
Sep 18 14:00:36.174413 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.174339    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90ec390b-f5c5-4d7e-9ac4-c60489fab5fe" containerName="maas-api"
Sep 18 14:00:36.174413 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.174344    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="90ec390b-f5c5-4d7e-9ac4-c60489fab5fe" containerName="maas-api"
Sep 18 14:00:36.174413 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.174415    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="90ec390b-f5c5-4d7e-9ac4-c60489fab5fe" containerName="maas-api"
Sep 18 14:00:36.174723 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.174424    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b3e5ef6-3700-4b0b-ac4f-d83184e0fdd5" containerName="payload-pre-processing"
Sep 18 14:00:36.179524 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.179501    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:00:36.181935 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.181911    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-21049c60-serving-cert\""
Sep 18 14:00:36.190790 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.190764    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"]
Sep 18 14:00:36.310777 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.310743    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b994fc2b-3a45-466a-9e4d-abaccf71bf79-metrics-certs\") pod \"maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5\" (UID: \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:00:36.310932 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.310797    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b994fc2b-3a45-466a-9e4d-abaccf71bf79-maas-api-tls\") pod \"maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5\" (UID: \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:00:36.310932 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.310863    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b994fc2b-3a45-466a-9e4d-abaccf71bf79-openshift-service-ca\") pod \"maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5\" (UID: \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:00:36.310932 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.310892    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drdqc\" (UniqueName: \"kubernetes.io/projected/b994fc2b-3a45-466a-9e4d-abaccf71bf79-kube-api-access-drdqc\") pod \"maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5\" (UID: \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:00:36.412156 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.412120    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b994fc2b-3a45-466a-9e4d-abaccf71bf79-metrics-certs\") pod \"maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5\" (UID: \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:00:36.412364 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.412180    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b994fc2b-3a45-466a-9e4d-abaccf71bf79-maas-api-tls\") pod \"maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5\" (UID: \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:00:36.412364 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.412233    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b994fc2b-3a45-466a-9e4d-abaccf71bf79-openshift-service-ca\") pod \"maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5\" (UID: \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:00:36.412364 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.412274    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drdqc\" (UniqueName: \"kubernetes.io/projected/b994fc2b-3a45-466a-9e4d-abaccf71bf79-kube-api-access-drdqc\") pod \"maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5\" (UID: \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:00:36.413190 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.413160    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b994fc2b-3a45-466a-9e4d-abaccf71bf79-openshift-service-ca\") pod \"maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5\" (UID: \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:00:36.415056 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.415029    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b994fc2b-3a45-466a-9e4d-abaccf71bf79-maas-api-tls\") pod \"maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5\" (UID: \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:00:36.416382 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.416356    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b994fc2b-3a45-466a-9e4d-abaccf71bf79-metrics-certs\") pod \"maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5\" (UID: \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:00:36.421483 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.421460    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drdqc\" (UniqueName: \"kubernetes.io/projected/b994fc2b-3a45-466a-9e4d-abaccf71bf79-kube-api-access-drdqc\") pod \"maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5\" (UID: \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:00:36.491605 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.491578    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:00:36.767861 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.767833    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"]
Sep 18 14:00:36.768636 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:00:36.768611    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb994fc2b_3a45_466a_9e4d_abaccf71bf79.slice/crio-3f7f78d09a8962f253105a5e640b9e4f1b9a2ae6e4177ec39ab28388f74f1ce7 WatchSource:0}: Error finding container 3f7f78d09a8962f253105a5e640b9e4f1b9a2ae6e4177ec39ab28388f74f1ce7: Status 404 returned error can't find the container with id 3f7f78d09a8962f253105a5e640b9e4f1b9a2ae6e4177ec39ab28388f74f1ce7
Sep 18 14:00:36.827353 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.827327    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"]
Sep 18 14:00:36.832380 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.832359    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"
Sep 18 14:00:36.835457 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.835426    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-21049c60\""
Sep 18 14:00:36.835654 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.835633    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-21049c60-dockercfg-jwgjs\""
Sep 18 14:00:36.852122 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:36.852091    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"]
Sep 18 14:00:37.018925 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:37.018837    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b59fc88-47da-4e6e-805f-afbde78b98f3-plugins-config-volume\") pod \"payload-processing-e2e-derive-21049c60-8855d48b5-lt89j\" (UID: \"7b59fc88-47da-4e6e-805f-afbde78b98f3\") " pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"
Sep 18 14:00:37.018925 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:37.018886    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xtr6\" (UniqueName: \"kubernetes.io/projected/7b59fc88-47da-4e6e-805f-afbde78b98f3-kube-api-access-5xtr6\") pod \"payload-processing-e2e-derive-21049c60-8855d48b5-lt89j\" (UID: \"7b59fc88-47da-4e6e-805f-afbde78b98f3\") " pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"
Sep 18 14:00:37.120092 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:37.120043    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b59fc88-47da-4e6e-805f-afbde78b98f3-plugins-config-volume\") pod \"payload-processing-e2e-derive-21049c60-8855d48b5-lt89j\" (UID: \"7b59fc88-47da-4e6e-805f-afbde78b98f3\") " pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"
Sep 18 14:00:37.120291 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:37.120102    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xtr6\" (UniqueName: \"kubernetes.io/projected/7b59fc88-47da-4e6e-805f-afbde78b98f3-kube-api-access-5xtr6\") pod \"payload-processing-e2e-derive-21049c60-8855d48b5-lt89j\" (UID: \"7b59fc88-47da-4e6e-805f-afbde78b98f3\") " pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"
Sep 18 14:00:37.120826 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:37.120806    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b59fc88-47da-4e6e-805f-afbde78b98f3-plugins-config-volume\") pod \"payload-processing-e2e-derive-21049c60-8855d48b5-lt89j\" (UID: \"7b59fc88-47da-4e6e-805f-afbde78b98f3\") " pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"
Sep 18 14:00:37.128704 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:37.128674    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xtr6\" (UniqueName: \"kubernetes.io/projected/7b59fc88-47da-4e6e-805f-afbde78b98f3-kube-api-access-5xtr6\") pod \"payload-processing-e2e-derive-21049c60-8855d48b5-lt89j\" (UID: \"7b59fc88-47da-4e6e-805f-afbde78b98f3\") " pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"
Sep 18 14:00:37.138018 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:37.137978    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5" event={"ID":"b994fc2b-3a45-466a-9e4d-abaccf71bf79","Type":"ContainerStarted","Data":"3f7f78d09a8962f253105a5e640b9e4f1b9a2ae6e4177ec39ab28388f74f1ce7"}
Sep 18 14:00:37.154766 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:37.154728    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"
Sep 18 14:00:37.430363 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:37.430331    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"]
Sep 18 14:00:37.431252 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:00:37.431228    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b59fc88_47da_4e6e_805f_afbde78b98f3.slice/crio-22880571eb4f68cc780a2bc6381210524c7d8583764beb10c46d067eed3215f8 WatchSource:0}: Error finding container 22880571eb4f68cc780a2bc6381210524c7d8583764beb10c46d067eed3215f8: Status 404 returned error can't find the container with id 22880571eb4f68cc780a2bc6381210524c7d8583764beb10c46d067eed3215f8
Sep 18 14:00:38.146300 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:38.146197    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j" event={"ID":"7b59fc88-47da-4e6e-805f-afbde78b98f3","Type":"ContainerStarted","Data":"30f5c5f4858abe894dc058cea1dd8c7afef40aaec57751ade352054469150e35"}
Sep 18 14:00:38.146300 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:38.146279    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"
Sep 18 14:00:38.146641 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:38.146304    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j" event={"ID":"7b59fc88-47da-4e6e-805f-afbde78b98f3","Type":"ContainerStarted","Data":"22880571eb4f68cc780a2bc6381210524c7d8583764beb10c46d067eed3215f8"}
Sep 18 14:00:38.148050 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:38.148022    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5" event={"ID":"b994fc2b-3a45-466a-9e4d-abaccf71bf79","Type":"ContainerStarted","Data":"7d813d11f993ca2fd78005e68ae5008b690a723fd09e05d54ac69d56fc7de920"}
Sep 18 14:00:38.148167 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:38.148098    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:00:38.163807 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:38.163756    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j" podStartSLOduration=2.163735153 podStartE2EDuration="2.163735153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:00:38.162661913 +0000 UTC m=+974.097627059" watchObservedRunningTime="2026-09-18 14:00:38.163735153 +0000 UTC m=+974.098700300"
Sep 18 14:00:38.180365 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:38.180312    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5" podStartSLOduration=1.6518951469999998 podStartE2EDuration="2.18029526s" totalImagesPullingTime="528.400113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:00:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:00:38.178978881 +0000 UTC m=+974.113944026" watchObservedRunningTime="2026-09-18 14:00:38.18029526 +0000 UTC m=+974.115260404"
Sep 18 14:00:44.159994 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:44.159963    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:00:49.155938 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:00:49.155902    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"
Sep 18 14:01:08.721554 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:08.721432    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"]
Sep 18 14:01:08.722465 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:08.722404    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv" podUID="2b790eec-5aea-4bff-b4ef-c5fa802e29cd" containerName="maas-api" containerID="cri-o://b438cbff94166461630d659b05fe5cff69d8fb70b8e25b90f839224b29aa2c36" gracePeriod=30
Sep 18 14:01:08.787128 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:08.787092    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"]
Sep 18 14:01:08.787375 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:08.787348    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht" podUID="e3ef3b5e-2d17-41cf-8b44-1892903435e2" containerName="payload-processing" containerID="cri-o://188bebe98bbd22c9988e635567c8739c152fd1fdc4a9574df5f8dea5d584f278" gracePeriod=30
Sep 18 14:01:08.813007 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:08.812968    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"]
Sep 18 14:01:08.818450 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:08.813475    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc" podUID="a06c46fb-97e2-4ffd-8d04-ef2252a23030" containerName="payload-pre-processing" containerID="cri-o://a1dd5e15485f29e186e8b7ff5beb929f7336e746af7755a021bfcd5cdfa33cba" gracePeriod=30
Sep 18 14:01:09.056916 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.056688    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 14:01:09.133596 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.133471    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"
Sep 18 14:01:09.141135 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.141108    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"
Sep 18 14:01:09.246627 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.246585    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjrdh\" (UniqueName: \"kubernetes.io/projected/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-kube-api-access-cjrdh\") pod \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\" (UID: \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\") "
Sep 18 14:01:09.246818 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.246641    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-metrics-certs\") pod \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\" (UID: \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\") "
Sep 18 14:01:09.246818 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.246683    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3ef3b5e-2d17-41cf-8b44-1892903435e2-plugins-config-volume\") pod \"e3ef3b5e-2d17-41cf-8b44-1892903435e2\" (UID: \"e3ef3b5e-2d17-41cf-8b44-1892903435e2\") "
Sep 18 14:01:09.246818 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.246712    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-openshift-service-ca\") pod \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\" (UID: \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\") "
Sep 18 14:01:09.246999 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.246838    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06c46fb-97e2-4ffd-8d04-ef2252a23030-plugins-config-volume\") pod \"a06c46fb-97e2-4ffd-8d04-ef2252a23030\" (UID: \"a06c46fb-97e2-4ffd-8d04-ef2252a23030\") "
Sep 18 14:01:09.246999 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.246876    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvsh5\" (UniqueName: \"kubernetes.io/projected/e3ef3b5e-2d17-41cf-8b44-1892903435e2-kube-api-access-mvsh5\") pod \"e3ef3b5e-2d17-41cf-8b44-1892903435e2\" (UID: \"e3ef3b5e-2d17-41cf-8b44-1892903435e2\") "
Sep 18 14:01:09.246999 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.246963    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddtmw\" (UniqueName: \"kubernetes.io/projected/a06c46fb-97e2-4ffd-8d04-ef2252a23030-kube-api-access-ddtmw\") pod \"a06c46fb-97e2-4ffd-8d04-ef2252a23030\" (UID: \"a06c46fb-97e2-4ffd-8d04-ef2252a23030\") "
Sep 18 14:01:09.247165 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.247017    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-maas-api-tls\") pod \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\" (UID: \"2b790eec-5aea-4bff-b4ef-c5fa802e29cd\") "
Sep 18 14:01:09.247165 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.247131    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2b790eec-5aea-4bff-b4ef-c5fa802e29cd" (UID: "2b790eec-5aea-4bff-b4ef-c5fa802e29cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:01:09.247268 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.247184    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ef3b5e-2d17-41cf-8b44-1892903435e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e3ef3b5e-2d17-41cf-8b44-1892903435e2" (UID: "e3ef3b5e-2d17-41cf-8b44-1892903435e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:01:09.247268 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.247207    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06c46fb-97e2-4ffd-8d04-ef2252a23030-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a06c46fb-97e2-4ffd-8d04-ef2252a23030" (UID: "a06c46fb-97e2-4ffd-8d04-ef2252a23030"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:01:09.247452 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.247425    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3ef3b5e-2d17-41cf-8b44-1892903435e2-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:01:09.247522 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.247451    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:01:09.247522 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.247466    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06c46fb-97e2-4ffd-8d04-ef2252a23030-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:01:09.249214 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.249187    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2b790eec-5aea-4bff-b4ef-c5fa802e29cd" (UID: "2b790eec-5aea-4bff-b4ef-c5fa802e29cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:01:09.249214 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.249208    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-kube-api-access-cjrdh" (OuterVolumeSpecName: "kube-api-access-cjrdh") pod "2b790eec-5aea-4bff-b4ef-c5fa802e29cd" (UID: "2b790eec-5aea-4bff-b4ef-c5fa802e29cd"). InnerVolumeSpecName "kube-api-access-cjrdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:01:09.249499 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.249476    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06c46fb-97e2-4ffd-8d04-ef2252a23030-kube-api-access-ddtmw" (OuterVolumeSpecName: "kube-api-access-ddtmw") pod "a06c46fb-97e2-4ffd-8d04-ef2252a23030" (UID: "a06c46fb-97e2-4ffd-8d04-ef2252a23030"). InnerVolumeSpecName "kube-api-access-ddtmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:01:09.249701 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.249570    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2b790eec-5aea-4bff-b4ef-c5fa802e29cd" (UID: "2b790eec-5aea-4bff-b4ef-c5fa802e29cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:01:09.249752 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.249692    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ef3b5e-2d17-41cf-8b44-1892903435e2-kube-api-access-mvsh5" (OuterVolumeSpecName: "kube-api-access-mvsh5") pod "e3ef3b5e-2d17-41cf-8b44-1892903435e2" (UID: "e3ef3b5e-2d17-41cf-8b44-1892903435e2"). InnerVolumeSpecName "kube-api-access-mvsh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:01:09.340065 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.339965    2572 generic.go:356] "Generic (PLEG): container finished" podID="2b790eec-5aea-4bff-b4ef-c5fa802e29cd" containerID="b438cbff94166461630d659b05fe5cff69d8fb70b8e25b90f839224b29aa2c36" exitCode=0
Sep 18 14:01:09.340065 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.340016    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"
Sep 18 14:01:09.340065 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.340050    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv" event={"ID":"2b790eec-5aea-4bff-b4ef-c5fa802e29cd","Type":"ContainerDied","Data":"b438cbff94166461630d659b05fe5cff69d8fb70b8e25b90f839224b29aa2c36"}
Sep 18 14:01:09.340320 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.340104    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv" event={"ID":"2b790eec-5aea-4bff-b4ef-c5fa802e29cd","Type":"ContainerDied","Data":"9a873ca39055f489963dc539657762eaeb023dff5156d886a1c97acfd50004b7"}
Sep 18 14:01:09.340320 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.340134    2572 scope.go:117] "RemoveContainer" containerID="b438cbff94166461630d659b05fe5cff69d8fb70b8e25b90f839224b29aa2c36"
Sep 18 14:01:09.341444 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.341419    2572 generic.go:356] "Generic (PLEG): container finished" podID="e3ef3b5e-2d17-41cf-8b44-1892903435e2" containerID="188bebe98bbd22c9988e635567c8739c152fd1fdc4a9574df5f8dea5d584f278" exitCode=0
Sep 18 14:01:09.341606 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.341474    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"
Sep 18 14:01:09.341606 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.341500    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht" event={"ID":"e3ef3b5e-2d17-41cf-8b44-1892903435e2","Type":"ContainerDied","Data":"188bebe98bbd22c9988e635567c8739c152fd1fdc4a9574df5f8dea5d584f278"}
Sep 18 14:01:09.341725 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.341611    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht" event={"ID":"e3ef3b5e-2d17-41cf-8b44-1892903435e2","Type":"ContainerDied","Data":"7f3ed4860dc2a20f3b7ef03c4d565eb8a793e0b0ceb17fd1d9acb9d576dfa1e0"}
Sep 18 14:01:09.343145 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.343121    2572 generic.go:356] "Generic (PLEG): container finished" podID="a06c46fb-97e2-4ffd-8d04-ef2252a23030" containerID="a1dd5e15485f29e186e8b7ff5beb929f7336e746af7755a021bfcd5cdfa33cba" exitCode=0
Sep 18 14:01:09.343271 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.343145    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc" event={"ID":"a06c46fb-97e2-4ffd-8d04-ef2252a23030","Type":"ContainerDied","Data":"a1dd5e15485f29e186e8b7ff5beb929f7336e746af7755a021bfcd5cdfa33cba"}
Sep 18 14:01:09.343271 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.343163    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"
Sep 18 14:01:09.343271 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.343194    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc" event={"ID":"a06c46fb-97e2-4ffd-8d04-ef2252a23030","Type":"ContainerDied","Data":"998fd31fd05eb356cb5312ad1289baf6fd898f581efd32d468026173050404c9"}
Sep 18 14:01:09.348592 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.348287    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddtmw\" (UniqueName: \"kubernetes.io/projected/a06c46fb-97e2-4ffd-8d04-ef2252a23030-kube-api-access-ddtmw\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:01:09.348592 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.348312    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:01:09.348592 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.348327    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjrdh\" (UniqueName: \"kubernetes.io/projected/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-kube-api-access-cjrdh\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:01:09.348592 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.348340    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b790eec-5aea-4bff-b4ef-c5fa802e29cd-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:01:09.348592 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.348354    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvsh5\" (UniqueName: \"kubernetes.io/projected/e3ef3b5e-2d17-41cf-8b44-1892903435e2-kube-api-access-mvsh5\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:01:09.353588 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.353567    2572 scope.go:117] "RemoveContainer" containerID="b438cbff94166461630d659b05fe5cff69d8fb70b8e25b90f839224b29aa2c36"
Sep 18 14:01:09.353931 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:01:09.353907    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b438cbff94166461630d659b05fe5cff69d8fb70b8e25b90f839224b29aa2c36\": container with ID starting with b438cbff94166461630d659b05fe5cff69d8fb70b8e25b90f839224b29aa2c36 not found: ID does not exist" containerID="b438cbff94166461630d659b05fe5cff69d8fb70b8e25b90f839224b29aa2c36"
Sep 18 14:01:09.354012 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.353940    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b438cbff94166461630d659b05fe5cff69d8fb70b8e25b90f839224b29aa2c36"} err="failed to get container status \"b438cbff94166461630d659b05fe5cff69d8fb70b8e25b90f839224b29aa2c36\": rpc error: code = NotFound desc = could not find container \"b438cbff94166461630d659b05fe5cff69d8fb70b8e25b90f839224b29aa2c36\": container with ID starting with b438cbff94166461630d659b05fe5cff69d8fb70b8e25b90f839224b29aa2c36 not found: ID does not exist"
Sep 18 14:01:09.354012 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.353956    2572 scope.go:117] "RemoveContainer" containerID="188bebe98bbd22c9988e635567c8739c152fd1fdc4a9574df5f8dea5d584f278"
Sep 18 14:01:09.362935 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.362917    2572 scope.go:117] "RemoveContainer" containerID="188bebe98bbd22c9988e635567c8739c152fd1fdc4a9574df5f8dea5d584f278"
Sep 18 14:01:09.363190 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:01:09.363175    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188bebe98bbd22c9988e635567c8739c152fd1fdc4a9574df5f8dea5d584f278\": container with ID starting with 188bebe98bbd22c9988e635567c8739c152fd1fdc4a9574df5f8dea5d584f278 not found: ID does not exist" containerID="188bebe98bbd22c9988e635567c8739c152fd1fdc4a9574df5f8dea5d584f278"
Sep 18 14:01:09.363238 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.363197    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188bebe98bbd22c9988e635567c8739c152fd1fdc4a9574df5f8dea5d584f278"} err="failed to get container status \"188bebe98bbd22c9988e635567c8739c152fd1fdc4a9574df5f8dea5d584f278\": rpc error: code = NotFound desc = could not find container \"188bebe98bbd22c9988e635567c8739c152fd1fdc4a9574df5f8dea5d584f278\": container with ID starting with 188bebe98bbd22c9988e635567c8739c152fd1fdc4a9574df5f8dea5d584f278 not found: ID does not exist"
Sep 18 14:01:09.363238 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.363214    2572 scope.go:117] "RemoveContainer" containerID="a1dd5e15485f29e186e8b7ff5beb929f7336e746af7755a021bfcd5cdfa33cba"
Sep 18 14:01:09.370255 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.370229    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"]
Sep 18 14:01:09.372080 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.372057    2572 scope.go:117] "RemoveContainer" containerID="a1dd5e15485f29e186e8b7ff5beb929f7336e746af7755a021bfcd5cdfa33cba"
Sep 18 14:01:09.372411 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:01:09.372389    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1dd5e15485f29e186e8b7ff5beb929f7336e746af7755a021bfcd5cdfa33cba\": container with ID starting with a1dd5e15485f29e186e8b7ff5beb929f7336e746af7755a021bfcd5cdfa33cba not found: ID does not exist" containerID="a1dd5e15485f29e186e8b7ff5beb929f7336e746af7755a021bfcd5cdfa33cba"
Sep 18 14:01:09.372486 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.372417    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1dd5e15485f29e186e8b7ff5beb929f7336e746af7755a021bfcd5cdfa33cba"} err="failed to get container status \"a1dd5e15485f29e186e8b7ff5beb929f7336e746af7755a021bfcd5cdfa33cba\": rpc error: code = NotFound desc = could not find container \"a1dd5e15485f29e186e8b7ff5beb929f7336e746af7755a021bfcd5cdfa33cba\": container with ID starting with a1dd5e15485f29e186e8b7ff5beb929f7336e746af7755a021bfcd5cdfa33cba not found: ID does not exist"
Sep 18 14:01:09.376913 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.376893    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-13658e-dd7d878fd-hr2xv"]
Sep 18 14:01:09.389649 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.389621    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"]
Sep 18 14:01:09.399917 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.399890    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-13658e-69d54bf9b5-j4bht"]
Sep 18 14:01:09.412382 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.412361    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"]
Sep 18 14:01:09.416224 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:09.416199    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-13658e-5577dc6946-z7mgc"]
Sep 18 14:01:10.198262 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:10.198227    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b790eec-5aea-4bff-b4ef-c5fa802e29cd" path="/var/lib/kubelet/pods/2b790eec-5aea-4bff-b4ef-c5fa802e29cd/volumes"
Sep 18 14:01:10.198662 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:10.198633    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a06c46fb-97e2-4ffd-8d04-ef2252a23030" path="/var/lib/kubelet/pods/a06c46fb-97e2-4ffd-8d04-ef2252a23030/volumes"
Sep 18 14:01:10.198952 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:10.198939    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3ef3b5e-2d17-41cf-8b44-1892903435e2" path="/var/lib/kubelet/pods/e3ef3b5e-2d17-41cf-8b44-1892903435e2/volumes"
Sep 18 14:01:17.617812 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:17.617773    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"]
Sep 18 14:01:17.618211 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:17.618045    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5" podUID="b994fc2b-3a45-466a-9e4d-abaccf71bf79" containerName="maas-api" containerID="cri-o://7d813d11f993ca2fd78005e68ae5008b690a723fd09e05d54ac69d56fc7de920" gracePeriod=30
Sep 18 14:01:17.691947 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:17.691911    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"]
Sep 18 14:01:17.692206 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:17.692163    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j" podUID="7b59fc88-47da-4e6e-805f-afbde78b98f3" containerName="payload-processing" containerID="cri-o://30f5c5f4858abe894dc058cea1dd8c7afef40aaec57751ade352054469150e35" gracePeriod=30
Sep 18 14:01:17.884834 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:17.884806    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:01:17.933592 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:17.933559    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b994fc2b-3a45-466a-9e4d-abaccf71bf79-openshift-service-ca\") pod \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\" (UID: \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\") "
Sep 18 14:01:17.933859 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:17.933839    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drdqc\" (UniqueName: \"kubernetes.io/projected/b994fc2b-3a45-466a-9e4d-abaccf71bf79-kube-api-access-drdqc\") pod \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\" (UID: \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\") "
Sep 18 14:01:17.933969 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:17.933938    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b994fc2b-3a45-466a-9e4d-abaccf71bf79-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b994fc2b-3a45-466a-9e4d-abaccf71bf79" (UID: "b994fc2b-3a45-466a-9e4d-abaccf71bf79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:01:17.934110 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:17.934094    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b994fc2b-3a45-466a-9e4d-abaccf71bf79-metrics-certs\") pod \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\" (UID: \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\") "
Sep 18 14:01:17.934242 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:17.934228    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b994fc2b-3a45-466a-9e4d-abaccf71bf79-maas-api-tls\") pod \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\" (UID: \"b994fc2b-3a45-466a-9e4d-abaccf71bf79\") "
Sep 18 14:01:17.934716 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:17.934697    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b994fc2b-3a45-466a-9e4d-abaccf71bf79-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:01:17.936865 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:17.936834    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b994fc2b-3a45-466a-9e4d-abaccf71bf79-kube-api-access-drdqc" (OuterVolumeSpecName: "kube-api-access-drdqc") pod "b994fc2b-3a45-466a-9e4d-abaccf71bf79" (UID: "b994fc2b-3a45-466a-9e4d-abaccf71bf79"). InnerVolumeSpecName "kube-api-access-drdqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:01:17.937458 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:17.937420    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b994fc2b-3a45-466a-9e4d-abaccf71bf79-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b994fc2b-3a45-466a-9e4d-abaccf71bf79" (UID: "b994fc2b-3a45-466a-9e4d-abaccf71bf79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:01:17.938378 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:17.938353    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b994fc2b-3a45-466a-9e4d-abaccf71bf79-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b994fc2b-3a45-466a-9e4d-abaccf71bf79" (UID: "b994fc2b-3a45-466a-9e4d-abaccf71bf79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:01:17.954062 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:17.954035    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"
Sep 18 14:01:18.038983 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.038939    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b59fc88-47da-4e6e-805f-afbde78b98f3-plugins-config-volume\") pod \"7b59fc88-47da-4e6e-805f-afbde78b98f3\" (UID: \"7b59fc88-47da-4e6e-805f-afbde78b98f3\") "
Sep 18 14:01:18.039200 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.038993    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xtr6\" (UniqueName: \"kubernetes.io/projected/7b59fc88-47da-4e6e-805f-afbde78b98f3-kube-api-access-5xtr6\") pod \"7b59fc88-47da-4e6e-805f-afbde78b98f3\" (UID: \"7b59fc88-47da-4e6e-805f-afbde78b98f3\") "
Sep 18 14:01:18.040253 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.040221    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drdqc\" (UniqueName: \"kubernetes.io/projected/b994fc2b-3a45-466a-9e4d-abaccf71bf79-kube-api-access-drdqc\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:01:18.040253 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.040255    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b994fc2b-3a45-466a-9e4d-abaccf71bf79-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:01:18.040489 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.040279    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b994fc2b-3a45-466a-9e4d-abaccf71bf79-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:01:18.042997 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.042961    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b59fc88-47da-4e6e-805f-afbde78b98f3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b59fc88-47da-4e6e-805f-afbde78b98f3" (UID: "7b59fc88-47da-4e6e-805f-afbde78b98f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:01:18.045307 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.045257    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b59fc88-47da-4e6e-805f-afbde78b98f3-kube-api-access-5xtr6" (OuterVolumeSpecName: "kube-api-access-5xtr6") pod "7b59fc88-47da-4e6e-805f-afbde78b98f3" (UID: "7b59fc88-47da-4e6e-805f-afbde78b98f3"). InnerVolumeSpecName "kube-api-access-5xtr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:01:18.141847 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.141757    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b59fc88-47da-4e6e-805f-afbde78b98f3-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:01:18.141847 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.141794    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xtr6\" (UniqueName: \"kubernetes.io/projected/7b59fc88-47da-4e6e-805f-afbde78b98f3-kube-api-access-5xtr6\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:01:18.410567 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.410450    2572 generic.go:356] "Generic (PLEG): container finished" podID="7b59fc88-47da-4e6e-805f-afbde78b98f3" containerID="30f5c5f4858abe894dc058cea1dd8c7afef40aaec57751ade352054469150e35" exitCode=0
Sep 18 14:01:18.410567 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.410528    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"
Sep 18 14:01:18.410567 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.410549    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j" event={"ID":"7b59fc88-47da-4e6e-805f-afbde78b98f3","Type":"ContainerDied","Data":"30f5c5f4858abe894dc058cea1dd8c7afef40aaec57751ade352054469150e35"}
Sep 18 14:01:18.411012 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.410609    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j" event={"ID":"7b59fc88-47da-4e6e-805f-afbde78b98f3","Type":"ContainerDied","Data":"22880571eb4f68cc780a2bc6381210524c7d8583764beb10c46d067eed3215f8"}
Sep 18 14:01:18.411012 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.410624    2572 scope.go:117] "RemoveContainer" containerID="30f5c5f4858abe894dc058cea1dd8c7afef40aaec57751ade352054469150e35"
Sep 18 14:01:18.412385 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.412359    2572 generic.go:356] "Generic (PLEG): container finished" podID="b994fc2b-3a45-466a-9e4d-abaccf71bf79" containerID="7d813d11f993ca2fd78005e68ae5008b690a723fd09e05d54ac69d56fc7de920" exitCode=0
Sep 18 14:01:18.412483 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.412467    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"
Sep 18 14:01:18.413143 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.413115    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5" event={"ID":"b994fc2b-3a45-466a-9e4d-abaccf71bf79","Type":"ContainerDied","Data":"7d813d11f993ca2fd78005e68ae5008b690a723fd09e05d54ac69d56fc7de920"}
Sep 18 14:01:18.413271 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.413171    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5" event={"ID":"b994fc2b-3a45-466a-9e4d-abaccf71bf79","Type":"ContainerDied","Data":"3f7f78d09a8962f253105a5e640b9e4f1b9a2ae6e4177ec39ab28388f74f1ce7"}
Sep 18 14:01:18.423058 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.423037    2572 scope.go:117] "RemoveContainer" containerID="30f5c5f4858abe894dc058cea1dd8c7afef40aaec57751ade352054469150e35"
Sep 18 14:01:18.423636 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:01:18.423471    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f5c5f4858abe894dc058cea1dd8c7afef40aaec57751ade352054469150e35\": container with ID starting with 30f5c5f4858abe894dc058cea1dd8c7afef40aaec57751ade352054469150e35 not found: ID does not exist" containerID="30f5c5f4858abe894dc058cea1dd8c7afef40aaec57751ade352054469150e35"
Sep 18 14:01:18.423636 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.423506    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f5c5f4858abe894dc058cea1dd8c7afef40aaec57751ade352054469150e35"} err="failed to get container status \"30f5c5f4858abe894dc058cea1dd8c7afef40aaec57751ade352054469150e35\": rpc error: code = NotFound desc = could not find container \"30f5c5f4858abe894dc058cea1dd8c7afef40aaec57751ade352054469150e35\": container with ID starting with 30f5c5f4858abe894dc058cea1dd8c7afef40aaec57751ade352054469150e35 not found: ID does not exist"
Sep 18 14:01:18.423636 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.423545    2572 scope.go:117] "RemoveContainer" containerID="7d813d11f993ca2fd78005e68ae5008b690a723fd09e05d54ac69d56fc7de920"
Sep 18 14:01:18.433318 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.433300    2572 scope.go:117] "RemoveContainer" containerID="7d813d11f993ca2fd78005e68ae5008b690a723fd09e05d54ac69d56fc7de920"
Sep 18 14:01:18.433612 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:01:18.433593    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d813d11f993ca2fd78005e68ae5008b690a723fd09e05d54ac69d56fc7de920\": container with ID starting with 7d813d11f993ca2fd78005e68ae5008b690a723fd09e05d54ac69d56fc7de920 not found: ID does not exist" containerID="7d813d11f993ca2fd78005e68ae5008b690a723fd09e05d54ac69d56fc7de920"
Sep 18 14:01:18.433673 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.433622    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d813d11f993ca2fd78005e68ae5008b690a723fd09e05d54ac69d56fc7de920"} err="failed to get container status \"7d813d11f993ca2fd78005e68ae5008b690a723fd09e05d54ac69d56fc7de920\": rpc error: code = NotFound desc = could not find container \"7d813d11f993ca2fd78005e68ae5008b690a723fd09e05d54ac69d56fc7de920\": container with ID starting with 7d813d11f993ca2fd78005e68ae5008b690a723fd09e05d54ac69d56fc7de920 not found: ID does not exist"
Sep 18 14:01:18.437934 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.437907    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"]
Sep 18 14:01:18.441397 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.441377    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-21049c60-8855d48b5-lt89j"]
Sep 18 14:01:18.456873 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.456849    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"]
Sep 18 14:01:18.459114 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:18.459094    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21049c60-5765dcc8fc-g88q5"]
Sep 18 14:01:20.198022 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:20.197987    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b59fc88-47da-4e6e-805f-afbde78b98f3" path="/var/lib/kubelet/pods/7b59fc88-47da-4e6e-805f-afbde78b98f3/volumes"
Sep 18 14:01:20.198484 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:20.198392    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b994fc2b-3a45-466a-9e4d-abaccf71bf79" path="/var/lib/kubelet/pods/b994fc2b-3a45-466a-9e4d-abaccf71bf79/volumes"
Sep 18 14:01:46.084854 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.084806    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"]
Sep 18 14:01:46.085417 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.085234    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b994fc2b-3a45-466a-9e4d-abaccf71bf79" containerName="maas-api"
Sep 18 14:01:46.085417 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.085246    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b994fc2b-3a45-466a-9e4d-abaccf71bf79" containerName="maas-api"
Sep 18 14:01:46.085417 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.085257    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b59fc88-47da-4e6e-805f-afbde78b98f3" containerName="payload-processing"
Sep 18 14:01:46.085417 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.085264    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b59fc88-47da-4e6e-805f-afbde78b98f3" containerName="payload-processing"
Sep 18 14:01:46.085417 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.085281    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b790eec-5aea-4bff-b4ef-c5fa802e29cd" containerName="maas-api"
Sep 18 14:01:46.085417 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.085291    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b790eec-5aea-4bff-b4ef-c5fa802e29cd" containerName="maas-api"
Sep 18 14:01:46.085417 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.085299    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3ef3b5e-2d17-41cf-8b44-1892903435e2" containerName="payload-processing"
Sep 18 14:01:46.085417 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.085304    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3ef3b5e-2d17-41cf-8b44-1892903435e2" containerName="payload-processing"
Sep 18 14:01:46.085417 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.085316    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a06c46fb-97e2-4ffd-8d04-ef2252a23030" containerName="payload-pre-processing"
Sep 18 14:01:46.085417 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.085323    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a06c46fb-97e2-4ffd-8d04-ef2252a23030" containerName="payload-pre-processing"
Sep 18 14:01:46.085417 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.085385    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b790eec-5aea-4bff-b4ef-c5fa802e29cd" containerName="maas-api"
Sep 18 14:01:46.085417 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.085392    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a06c46fb-97e2-4ffd-8d04-ef2252a23030" containerName="payload-pre-processing"
Sep 18 14:01:46.085417 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.085399    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3ef3b5e-2d17-41cf-8b44-1892903435e2" containerName="payload-processing"
Sep 18 14:01:46.085417 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.085404    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b994fc2b-3a45-466a-9e4d-abaccf71bf79" containerName="maas-api"
Sep 18 14:01:46.085417 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.085413    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b59fc88-47da-4e6e-805f-afbde78b98f3" containerName="payload-processing"
Sep 18 14:01:46.088493 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.088469    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.092175 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.092144    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-40970f\"/\"premium-w0-40970f-kserve-self-signed-certs\""
Sep 18 14:01:46.092175 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.092164    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-40970f\"/\"default-dockercfg-mzr67\""
Sep 18 14:01:46.092175 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.092174    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-40970f\"/\"openshift-service-ca.crt\""
Sep 18 14:01:46.092624 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.092610    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-40970f\"/\"kube-root-ca.crt\""
Sep 18 14:01:46.099718 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.099689    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"]
Sep 18 14:01:46.118779 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.118737    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-model-cache\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.118997 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.118812    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-dshm\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.118997 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.118858    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-home\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.118997 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.118880    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34192080-ca2c-431e-b7bb-988e5cf7ff8f-tls-certs\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.118997 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.118919    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-tmp-dir\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.118997 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.118941    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtxvh\" (UniqueName: \"kubernetes.io/projected/34192080-ca2c-431e-b7bb-988e5cf7ff8f-kube-api-access-xtxvh\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.220286 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.220251    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-home\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.220286 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.220287    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34192080-ca2c-431e-b7bb-988e5cf7ff8f-tls-certs\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.220617 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.220316    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-tmp-dir\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.220617 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.220337    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtxvh\" (UniqueName: \"kubernetes.io/projected/34192080-ca2c-431e-b7bb-988e5cf7ff8f-kube-api-access-xtxvh\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.220617 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.220402    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-model-cache\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.220617 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.220469    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-dshm\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.220840 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.220789    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-tmp-dir\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.220840 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.220821    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-model-cache\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.220915 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.220862    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-home\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.222726 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.222700    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-dshm\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.223072 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.223048    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34192080-ca2c-431e-b7bb-988e5cf7ff8f-tls-certs\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.229759 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.229733    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtxvh\" (UniqueName: \"kubernetes.io/projected/34192080-ca2c-431e-b7bb-988e5cf7ff8f-kube-api-access-xtxvh\") pod \"premium-w0-40970f-kserve-5f5f8777bb-txmv8\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") " pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.398732 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.398630    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:46.677312 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:46.677273    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"]
Sep 18 14:01:46.678994 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:01:46.678961    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34192080_ca2c_431e_b7bb_988e5cf7ff8f.slice/crio-008752a9d7031d0efdcb9e1157d4f209dd5cf3e51efb48a245f1fea7746cd9b1 WatchSource:0}: Error finding container 008752a9d7031d0efdcb9e1157d4f209dd5cf3e51efb48a245f1fea7746cd9b1: Status 404 returned error can't find the container with id 008752a9d7031d0efdcb9e1157d4f209dd5cf3e51efb48a245f1fea7746cd9b1
Sep 18 14:01:47.585762 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:47.585724    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8" event={"ID":"34192080-ca2c-431e-b7bb-988e5cf7ff8f","Type":"ContainerStarted","Data":"664b6522b7970086ad3a4abaca6886075d57d4c50f4a3056e670e2faaf28433a"}
Sep 18 14:01:47.586185 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:47.585779    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8" event={"ID":"34192080-ca2c-431e-b7bb-988e5cf7ff8f","Type":"ContainerStarted","Data":"008752a9d7031d0efdcb9e1157d4f209dd5cf3e51efb48a245f1fea7746cd9b1"}
Sep 18 14:01:47.606559 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:47.606471    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8" podStartSLOduration=1.606453618 podStartE2EDuration="1.606453618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:01:47.605101126 +0000 UTC m=+1043.540066270" watchObservedRunningTime="2026-09-18 14:01:47.606453618 +0000 UTC m=+1043.541418763"
Sep 18 14:01:56.399189 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:56.399150    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:56.399740 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:56.399204    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:56.412293 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:56.412264    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:56.652107 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:56.652016    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:01:59.602977 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:59.602938    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"]
Sep 18 14:01:59.608480 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:59.608440    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:01:59.613599 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:59.613568    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-ec85d5-serving-cert\""
Sep 18 14:01:59.622711 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:59.622142    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"]
Sep 18 14:01:59.649294 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:59.649262    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb363a3a-dc05-4815-bd88-31381b77538e-maas-api-tls\") pod \"maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:01:59.649294 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:59.649298    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb363a3a-dc05-4815-bd88-31381b77538e-metrics-certs\") pod \"maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:01:59.649561 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:59.649327    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghf5k\" (UniqueName: \"kubernetes.io/projected/bb363a3a-dc05-4815-bd88-31381b77538e-kube-api-access-ghf5k\") pod \"maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:01:59.649561 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:59.649443    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb363a3a-dc05-4815-bd88-31381b77538e-openshift-service-ca\") pod \"maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:01:59.750618 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:59.750577    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb363a3a-dc05-4815-bd88-31381b77538e-maas-api-tls\") pod \"maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:01:59.750816 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:59.750627    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb363a3a-dc05-4815-bd88-31381b77538e-metrics-certs\") pod \"maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:01:59.750816 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:59.750659    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghf5k\" (UniqueName: \"kubernetes.io/projected/bb363a3a-dc05-4815-bd88-31381b77538e-kube-api-access-ghf5k\") pod \"maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:01:59.750816 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:59.750721    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb363a3a-dc05-4815-bd88-31381b77538e-openshift-service-ca\") pod \"maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:01:59.750816 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:01:59.750727    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-serving-cert: secret "maas-api-e2e-explicit-ec85d5-serving-cert" not found
Sep 18 14:01:59.750988 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:01:59.750820    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb363a3a-dc05-4815-bd88-31381b77538e-maas-api-tls podName:bb363a3a-dc05-4815-bd88-31381b77538e nodeName:}" failed. No retries permitted until 2026-09-18 14:02:00.250797737 +0000 UTC m=+1056.185762877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bb363a3a-dc05-4815-bd88-31381b77538e-maas-api-tls") pod "maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h" (UID: "bb363a3a-dc05-4815-bd88-31381b77538e") : secret "maas-api-e2e-explicit-ec85d5-serving-cert" not found
Sep 18 14:01:59.751420 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:59.751403    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb363a3a-dc05-4815-bd88-31381b77538e-openshift-service-ca\") pod \"maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:01:59.753121 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:59.753104    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb363a3a-dc05-4815-bd88-31381b77538e-metrics-certs\") pod \"maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:01:59.767322 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:01:59.767291    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghf5k\" (UniqueName: \"kubernetes.io/projected/bb363a3a-dc05-4815-bd88-31381b77538e-kube-api-access-ghf5k\") pod \"maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:02:00.255876 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.255830    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb363a3a-dc05-4815-bd88-31381b77538e-maas-api-tls\") pod \"maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:02:00.258509 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.258480    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb363a3a-dc05-4815-bd88-31381b77538e-maas-api-tls\") pod \"maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:02:00.475347 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.475311    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"]
Sep 18 14:02:00.479360 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.479340    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"
Sep 18 14:02:00.482013 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.481989    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-ec85d5-dockercfg-n4pth\""
Sep 18 14:02:00.482313 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.482294    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-ec85d5\""
Sep 18 14:02:00.492660 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.492629    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"]
Sep 18 14:02:00.528906 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.528814    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:02:00.543080 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.543050    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"]
Sep 18 14:02:00.546840 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.546810    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"
Sep 18 14:02:00.558185 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.558149    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj8g7\" (UniqueName: \"kubernetes.io/projected/ecf52985-7740-4075-80ad-5266b1e6507d-kube-api-access-wj8g7\") pod \"payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg\" (UID: \"ecf52985-7740-4075-80ad-5266b1e6507d\") " pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"
Sep 18 14:02:00.558337 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.558240    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf52985-7740-4075-80ad-5266b1e6507d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg\" (UID: \"ecf52985-7740-4075-80ad-5266b1e6507d\") " pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"
Sep 18 14:02:00.563052 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.563024    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"]
Sep 18 14:02:00.658866 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.658829    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj8g7\" (UniqueName: \"kubernetes.io/projected/ecf52985-7740-4075-80ad-5266b1e6507d-kube-api-access-wj8g7\") pod \"payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg\" (UID: \"ecf52985-7740-4075-80ad-5266b1e6507d\") " pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"
Sep 18 14:02:00.659322 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.658891    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf52985-7740-4075-80ad-5266b1e6507d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg\" (UID: \"ecf52985-7740-4075-80ad-5266b1e6507d\") " pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"
Sep 18 14:02:00.659322 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.658933    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bd1384-e169-41a0-8c6c-db3225aea77a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt\" (UID: \"b4bd1384-e169-41a0-8c6c-db3225aea77a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"
Sep 18 14:02:00.659322 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.658962    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5dnr\" (UniqueName: \"kubernetes.io/projected/b4bd1384-e169-41a0-8c6c-db3225aea77a-kube-api-access-p5dnr\") pod \"payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt\" (UID: \"b4bd1384-e169-41a0-8c6c-db3225aea77a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"
Sep 18 14:02:00.659897 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.659872    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf52985-7740-4075-80ad-5266b1e6507d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg\" (UID: \"ecf52985-7740-4075-80ad-5266b1e6507d\") " pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"
Sep 18 14:02:00.667879 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.667852    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj8g7\" (UniqueName: \"kubernetes.io/projected/ecf52985-7740-4075-80ad-5266b1e6507d-kube-api-access-wj8g7\") pod \"payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg\" (UID: \"ecf52985-7740-4075-80ad-5266b1e6507d\") " pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"
Sep 18 14:02:00.760701 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.760657    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bd1384-e169-41a0-8c6c-db3225aea77a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt\" (UID: \"b4bd1384-e169-41a0-8c6c-db3225aea77a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"
Sep 18 14:02:00.760889 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.760723    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5dnr\" (UniqueName: \"kubernetes.io/projected/b4bd1384-e169-41a0-8c6c-db3225aea77a-kube-api-access-p5dnr\") pod \"payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt\" (UID: \"b4bd1384-e169-41a0-8c6c-db3225aea77a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"
Sep 18 14:02:00.761466 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.761446    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bd1384-e169-41a0-8c6c-db3225aea77a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt\" (UID: \"b4bd1384-e169-41a0-8c6c-db3225aea77a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"
Sep 18 14:02:00.769242 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.769211    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5dnr\" (UniqueName: \"kubernetes.io/projected/b4bd1384-e169-41a0-8c6c-db3225aea77a-kube-api-access-p5dnr\") pod \"payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt\" (UID: \"b4bd1384-e169-41a0-8c6c-db3225aea77a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"
Sep 18 14:02:00.791207 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.791102    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"
Sep 18 14:02:00.817511 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.817458    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"]
Sep 18 14:02:00.817675 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:02:00.817561    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb363a3a_dc05_4815_bd88_31381b77538e.slice/crio-b1a6bc7e84524ecfeee4e73007dd702dafaf44b8bef89c7dd3007a7a02b01a5f WatchSource:0}: Error finding container b1a6bc7e84524ecfeee4e73007dd702dafaf44b8bef89c7dd3007a7a02b01a5f: Status 404 returned error can't find the container with id b1a6bc7e84524ecfeee4e73007dd702dafaf44b8bef89c7dd3007a7a02b01a5f
Sep 18 14:02:00.869090 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:00.869057    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"
Sep 18 14:02:01.073648 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:01.073615    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"]
Sep 18 14:02:01.074737 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:02:01.074713    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf52985_7740_4075_80ad_5266b1e6507d.slice/crio-6017bdfafbbbbb9ed6f24bbe467b62e73a542d77373d6eb2ec0d9c62287d6bef WatchSource:0}: Error finding container 6017bdfafbbbbb9ed6f24bbe467b62e73a542d77373d6eb2ec0d9c62287d6bef: Status 404 returned error can't find the container with id 6017bdfafbbbbb9ed6f24bbe467b62e73a542d77373d6eb2ec0d9c62287d6bef
Sep 18 14:02:01.150355 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:01.150317    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"]
Sep 18 14:02:01.151336 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:02:01.151309    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bd1384_e169_41a0_8c6c_db3225aea77a.slice/crio-70d6b5c8bd5eb26e3f6fcb4caf4fc595452546b3dfd395973ca705dbfe1cb81e WatchSource:0}: Error finding container 70d6b5c8bd5eb26e3f6fcb4caf4fc595452546b3dfd395973ca705dbfe1cb81e: Status 404 returned error can't find the container with id 70d6b5c8bd5eb26e3f6fcb4caf4fc595452546b3dfd395973ca705dbfe1cb81e
Sep 18 14:02:01.671496 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:01.671455    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg" event={"ID":"ecf52985-7740-4075-80ad-5266b1e6507d","Type":"ContainerStarted","Data":"2f80b231203cbef468231f92e93c1e550598039e3c0c2d7bcc8e9485cc4dd7e4"}
Sep 18 14:02:01.672035 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:01.671509    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg" event={"ID":"ecf52985-7740-4075-80ad-5266b1e6507d","Type":"ContainerStarted","Data":"6017bdfafbbbbb9ed6f24bbe467b62e73a542d77373d6eb2ec0d9c62287d6bef"}
Sep 18 14:02:01.672035 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:01.671593    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"
Sep 18 14:02:01.673382 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:01.673355    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt" event={"ID":"b4bd1384-e169-41a0-8c6c-db3225aea77a","Type":"ContainerStarted","Data":"76749425df60b1c98ba2c2e2b73d38537b562232a2959243b9d80a36402f2166"}
Sep 18 14:02:01.673542 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:01.673406    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt" event={"ID":"b4bd1384-e169-41a0-8c6c-db3225aea77a","Type":"ContainerStarted","Data":"70d6b5c8bd5eb26e3f6fcb4caf4fc595452546b3dfd395973ca705dbfe1cb81e"}
Sep 18 14:02:01.673542 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:01.673441    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"
Sep 18 14:02:01.675069 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:01.675041    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h" event={"ID":"bb363a3a-dc05-4815-bd88-31381b77538e","Type":"ContainerStarted","Data":"ecf0aa0166578fd04401c0cde9f1217166d89b9d15f15d75e28f38a4ede5371c"}
Sep 18 14:02:01.675196 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:01.675088    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h" event={"ID":"bb363a3a-dc05-4815-bd88-31381b77538e","Type":"ContainerStarted","Data":"b1a6bc7e84524ecfeee4e73007dd702dafaf44b8bef89c7dd3007a7a02b01a5f"}
Sep 18 14:02:01.675196 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:01.675109    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:02:01.690156 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:01.690099    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg" podStartSLOduration=1.690081291 podStartE2EDuration="1.690081291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:02:01.687316567 +0000 UTC m=+1057.622281713" watchObservedRunningTime="2026-09-18 14:02:01.690081291 +0000 UTC m=+1057.625046435"
Sep 18 14:02:01.705410 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:01.705354    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt" podStartSLOduration=1.7053308 podStartE2EDuration="1.7053308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:02:01.702890549 +0000 UTC m=+1057.637855696" watchObservedRunningTime="2026-09-18 14:02:01.7053308 +0000 UTC m=+1057.640295997"
Sep 18 14:02:01.721980 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:01.721922    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h" podStartSLOduration=2.332028208 podStartE2EDuration="2.721903716s" totalImagesPullingTime="389.875508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:02:01.720253208 +0000 UTC m=+1057.655218355" watchObservedRunningTime="2026-09-18 14:02:01.721903716 +0000 UTC m=+1057.656868860"
Sep 18 14:02:07.686425 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:07.686395    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:02:12.682825 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:12.682739    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"
Sep 18 14:02:12.683201 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:02:12.682942    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"
Sep 18 14:03:06.993925 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:06.993888    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"]
Sep 18 14:03:06.997782 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:06.997761    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:06.999908 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:06.999885    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f17132d8-serving-cert\""
Sep 18 14:03:07.014489 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.014460    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"]
Sep 18 14:03:07.107761 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.107725    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/716d1066-7007-412d-95f0-c45bb44b7304-maas-api-tls\") pod \"maas-api-e2e-mt-f17132d8-c778cc674-v2h4c\" (UID: \"716d1066-7007-412d-95f0-c45bb44b7304\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:07.107934 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.107799    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v559b\" (UniqueName: \"kubernetes.io/projected/716d1066-7007-412d-95f0-c45bb44b7304-kube-api-access-v559b\") pod \"maas-api-e2e-mt-f17132d8-c778cc674-v2h4c\" (UID: \"716d1066-7007-412d-95f0-c45bb44b7304\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:07.107934 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.107844    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716d1066-7007-412d-95f0-c45bb44b7304-metrics-certs\") pod \"maas-api-e2e-mt-f17132d8-c778cc674-v2h4c\" (UID: \"716d1066-7007-412d-95f0-c45bb44b7304\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:07.107934 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.107865    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/716d1066-7007-412d-95f0-c45bb44b7304-openshift-service-ca\") pod \"maas-api-e2e-mt-f17132d8-c778cc674-v2h4c\" (UID: \"716d1066-7007-412d-95f0-c45bb44b7304\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:07.209867 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.209830    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/716d1066-7007-412d-95f0-c45bb44b7304-maas-api-tls\") pod \"maas-api-e2e-mt-f17132d8-c778cc674-v2h4c\" (UID: \"716d1066-7007-412d-95f0-c45bb44b7304\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:07.210084 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.209899    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v559b\" (UniqueName: \"kubernetes.io/projected/716d1066-7007-412d-95f0-c45bb44b7304-kube-api-access-v559b\") pod \"maas-api-e2e-mt-f17132d8-c778cc674-v2h4c\" (UID: \"716d1066-7007-412d-95f0-c45bb44b7304\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:07.210084 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.209953    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716d1066-7007-412d-95f0-c45bb44b7304-metrics-certs\") pod \"maas-api-e2e-mt-f17132d8-c778cc674-v2h4c\" (UID: \"716d1066-7007-412d-95f0-c45bb44b7304\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:07.210084 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.209977    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/716d1066-7007-412d-95f0-c45bb44b7304-openshift-service-ca\") pod \"maas-api-e2e-mt-f17132d8-c778cc674-v2h4c\" (UID: \"716d1066-7007-412d-95f0-c45bb44b7304\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:07.210763 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.210686    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/716d1066-7007-412d-95f0-c45bb44b7304-openshift-service-ca\") pod \"maas-api-e2e-mt-f17132d8-c778cc674-v2h4c\" (UID: \"716d1066-7007-412d-95f0-c45bb44b7304\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:07.212838 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.212808    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/716d1066-7007-412d-95f0-c45bb44b7304-maas-api-tls\") pod \"maas-api-e2e-mt-f17132d8-c778cc674-v2h4c\" (UID: \"716d1066-7007-412d-95f0-c45bb44b7304\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:07.212971 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.212871    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716d1066-7007-412d-95f0-c45bb44b7304-metrics-certs\") pod \"maas-api-e2e-mt-f17132d8-c778cc674-v2h4c\" (UID: \"716d1066-7007-412d-95f0-c45bb44b7304\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:07.219587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.219561    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v559b\" (UniqueName: \"kubernetes.io/projected/716d1066-7007-412d-95f0-c45bb44b7304-kube-api-access-v559b\") pod \"maas-api-e2e-mt-f17132d8-c778cc674-v2h4c\" (UID: \"716d1066-7007-412d-95f0-c45bb44b7304\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:07.309091 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.309006    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:07.586015 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.585989    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"]
Sep 18 14:03:07.587386 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:03:07.587356    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716d1066_7007_412d_95f0_c45bb44b7304.slice/crio-a643d10bb27f227252a9e4355cd304d723bc0eebf454fbdb0831259fd3c3e0d9 WatchSource:0}: Error finding container a643d10bb27f227252a9e4355cd304d723bc0eebf454fbdb0831259fd3c3e0d9: Status 404 returned error can't find the container with id a643d10bb27f227252a9e4355cd304d723bc0eebf454fbdb0831259fd3c3e0d9
Sep 18 14:03:07.653039 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.653004    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"]
Sep 18 14:03:07.657812 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.657790    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"
Sep 18 14:03:07.663403 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.663382    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f17132d8\""
Sep 18 14:03:07.670215 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.670187    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"]
Sep 18 14:03:07.729884 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.729848    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"]
Sep 18 14:03:07.733494 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.733466    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"
Sep 18 14:03:07.735706 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.735683    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f17132d8-dockercfg-9vl2x\""
Sep 18 14:03:07.745204 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.745177    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"]
Sep 18 14:03:07.816873 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.816839    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e776755d-20d6-4c46-9589-4926ce207809-plugins-config-volume\") pod \"payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg\" (UID: \"e776755d-20d6-4c46-9589-4926ce207809\") " pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"
Sep 18 14:03:07.816873 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.816878    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6cmq\" (UniqueName: \"kubernetes.io/projected/e776755d-20d6-4c46-9589-4926ce207809-kube-api-access-s6cmq\") pod \"payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg\" (UID: \"e776755d-20d6-4c46-9589-4926ce207809\") " pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"
Sep 18 14:03:07.918484 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.918398    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stj55\" (UniqueName: \"kubernetes.io/projected/428d445e-a2aa-47ea-9283-c26edaaea3d2-kube-api-access-stj55\") pod \"payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7\" (UID: \"428d445e-a2aa-47ea-9283-c26edaaea3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"
Sep 18 14:03:07.918484 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.918465    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428d445e-a2aa-47ea-9283-c26edaaea3d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7\" (UID: \"428d445e-a2aa-47ea-9283-c26edaaea3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"
Sep 18 14:03:07.918718 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.918518    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e776755d-20d6-4c46-9589-4926ce207809-plugins-config-volume\") pod \"payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg\" (UID: \"e776755d-20d6-4c46-9589-4926ce207809\") " pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"
Sep 18 14:03:07.918718 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.918578    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6cmq\" (UniqueName: \"kubernetes.io/projected/e776755d-20d6-4c46-9589-4926ce207809-kube-api-access-s6cmq\") pod \"payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg\" (UID: \"e776755d-20d6-4c46-9589-4926ce207809\") " pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"
Sep 18 14:03:07.919195 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.919176    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e776755d-20d6-4c46-9589-4926ce207809-plugins-config-volume\") pod \"payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg\" (UID: \"e776755d-20d6-4c46-9589-4926ce207809\") " pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"
Sep 18 14:03:07.927285 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.927260    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6cmq\" (UniqueName: \"kubernetes.io/projected/e776755d-20d6-4c46-9589-4926ce207809-kube-api-access-s6cmq\") pod \"payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg\" (UID: \"e776755d-20d6-4c46-9589-4926ce207809\") " pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"
Sep 18 14:03:07.970351 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:07.970317    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"
Sep 18 14:03:08.019473 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:08.019450    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stj55\" (UniqueName: \"kubernetes.io/projected/428d445e-a2aa-47ea-9283-c26edaaea3d2-kube-api-access-stj55\") pod \"payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7\" (UID: \"428d445e-a2aa-47ea-9283-c26edaaea3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"
Sep 18 14:03:08.019773 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:08.019574    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428d445e-a2aa-47ea-9283-c26edaaea3d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7\" (UID: \"428d445e-a2aa-47ea-9283-c26edaaea3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"
Sep 18 14:03:08.020308 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:08.020262    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428d445e-a2aa-47ea-9283-c26edaaea3d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7\" (UID: \"428d445e-a2aa-47ea-9283-c26edaaea3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"
Sep 18 14:03:08.032233 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:08.032150    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stj55\" (UniqueName: \"kubernetes.io/projected/428d445e-a2aa-47ea-9283-c26edaaea3d2-kube-api-access-stj55\") pod \"payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7\" (UID: \"428d445e-a2aa-47ea-9283-c26edaaea3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"
Sep 18 14:03:08.045237 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:08.045210    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"
Sep 18 14:03:08.078062 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:08.078031    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c" event={"ID":"716d1066-7007-412d-95f0-c45bb44b7304","Type":"ContainerStarted","Data":"a643d10bb27f227252a9e4355cd304d723bc0eebf454fbdb0831259fd3c3e0d9"}
Sep 18 14:03:08.275998 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:08.275968    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"]
Sep 18 14:03:08.276989 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:03:08.276968    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode776755d_20d6_4c46_9589_4926ce207809.slice/crio-4bce26919ec5897c85698c2e0a49ba5e0bf0ae288823d824f71cafdc4b6c6358 WatchSource:0}: Error finding container 4bce26919ec5897c85698c2e0a49ba5e0bf0ae288823d824f71cafdc4b6c6358: Status 404 returned error can't find the container with id 4bce26919ec5897c85698c2e0a49ba5e0bf0ae288823d824f71cafdc4b6c6358
Sep 18 14:03:08.326221 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:08.326199    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"]
Sep 18 14:03:08.333901 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:03:08.333860    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428d445e_a2aa_47ea_9283_c26edaaea3d2.slice/crio-804f06b848f234591bf43c390266350da0d0ed82b46d9f09e9cc95ec58849f2d WatchSource:0}: Error finding container 804f06b848f234591bf43c390266350da0d0ed82b46d9f09e9cc95ec58849f2d: Status 404 returned error can't find the container with id 804f06b848f234591bf43c390266350da0d0ed82b46d9f09e9cc95ec58849f2d
Sep 18 14:03:09.090154 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:09.090112    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg" event={"ID":"e776755d-20d6-4c46-9589-4926ce207809","Type":"ContainerStarted","Data":"c29d42837282ba7846649930e1393b309406c7cbcdd851e8556064eccab0e16a"}
Sep 18 14:03:09.090665 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:09.090178    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg" event={"ID":"e776755d-20d6-4c46-9589-4926ce207809","Type":"ContainerStarted","Data":"4bce26919ec5897c85698c2e0a49ba5e0bf0ae288823d824f71cafdc4b6c6358"}
Sep 18 14:03:09.090665 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:09.090296    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"
Sep 18 14:03:09.091876 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:09.091848    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c" event={"ID":"716d1066-7007-412d-95f0-c45bb44b7304","Type":"ContainerStarted","Data":"bb3cf35d3a91f61b0906f74bdda72dd9a8571381569218f20c6843e76b9ce6dd"}
Sep 18 14:03:09.092007 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:09.091903    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:09.093350 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:09.093323    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7" event={"ID":"428d445e-a2aa-47ea-9283-c26edaaea3d2","Type":"ContainerStarted","Data":"c0650d61cba1ece85974f7a39e9c291d788d7d7ea80d811f57b2aa5cb69c1587"}
Sep 18 14:03:09.093426 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:09.093367    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7" event={"ID":"428d445e-a2aa-47ea-9283-c26edaaea3d2","Type":"ContainerStarted","Data":"804f06b848f234591bf43c390266350da0d0ed82b46d9f09e9cc95ec58849f2d"}
Sep 18 14:03:09.093426 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:09.093404    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"
Sep 18 14:03:09.109662 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:09.109618    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg" podStartSLOduration=2.109604311 podStartE2EDuration="2.109604311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:03:09.106663998 +0000 UTC m=+1125.041629142" watchObservedRunningTime="2026-09-18 14:03:09.109604311 +0000 UTC m=+1125.044569481"
Sep 18 14:03:09.130370 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:09.130315    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7" podStartSLOduration=2.130295012 podStartE2EDuration="2.130295012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:03:09.126894873 +0000 UTC m=+1125.061860019" watchObservedRunningTime="2026-09-18 14:03:09.130295012 +0000 UTC m=+1125.065260161"
Sep 18 14:03:15.105238 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:15.105203    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:15.126282 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:15.126227    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c" podStartSLOduration=8.70056676 podStartE2EDuration="9.126209226s" totalImagesPullingTime="425.642466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:03:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:03:09.158301643 +0000 UTC m=+1125.093266792" watchObservedRunningTime="2026-09-18 14:03:15.126209226 +0000 UTC m=+1131.061174371"
Sep 18 14:03:20.102073 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:20.102040    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"
Sep 18 14:03:20.102491 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:20.102115    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"
Sep 18 14:03:51.022516 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.022430    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"]
Sep 18 14:03:51.023060 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.022690    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c" podUID="716d1066-7007-412d-95f0-c45bb44b7304" containerName="maas-api" containerID="cri-o://bb3cf35d3a91f61b0906f74bdda72dd9a8571381569218f20c6843e76b9ce6dd" gracePeriod=30
Sep 18 14:03:51.069806 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.069770    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"]
Sep 18 14:03:51.070051 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.070000    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg" podUID="e776755d-20d6-4c46-9589-4926ce207809" containerName="payload-processing" containerID="cri-o://c29d42837282ba7846649930e1393b309406c7cbcdd851e8556064eccab0e16a" gracePeriod=30
Sep 18 14:03:51.087319 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.087279    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"]
Sep 18 14:03:51.087577 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.087514    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7" podUID="428d445e-a2aa-47ea-9283-c26edaaea3d2" containerName="payload-pre-processing" containerID="cri-o://c0650d61cba1ece85974f7a39e9c291d788d7d7ea80d811f57b2aa5cb69c1587" gracePeriod=30
Sep 18 14:03:51.356085 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.355295    2572 generic.go:356] "Generic (PLEG): container finished" podID="e776755d-20d6-4c46-9589-4926ce207809" containerID="c29d42837282ba7846649930e1393b309406c7cbcdd851e8556064eccab0e16a" exitCode=0
Sep 18 14:03:51.356085 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.355406    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg" event={"ID":"e776755d-20d6-4c46-9589-4926ce207809","Type":"ContainerDied","Data":"c29d42837282ba7846649930e1393b309406c7cbcdd851e8556064eccab0e16a"}
Sep 18 14:03:51.358031 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.357811    2572 generic.go:356] "Generic (PLEG): container finished" podID="716d1066-7007-412d-95f0-c45bb44b7304" containerID="bb3cf35d3a91f61b0906f74bdda72dd9a8571381569218f20c6843e76b9ce6dd" exitCode=0
Sep 18 14:03:51.358031 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.357964    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c" event={"ID":"716d1066-7007-412d-95f0-c45bb44b7304","Type":"ContainerDied","Data":"bb3cf35d3a91f61b0906f74bdda72dd9a8571381569218f20c6843e76b9ce6dd"}
Sep 18 14:03:51.360673 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.360575    2572 generic.go:356] "Generic (PLEG): container finished" podID="428d445e-a2aa-47ea-9283-c26edaaea3d2" containerID="c0650d61cba1ece85974f7a39e9c291d788d7d7ea80d811f57b2aa5cb69c1587" exitCode=0
Sep 18 14:03:51.360673 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.360648    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7" event={"ID":"428d445e-a2aa-47ea-9283-c26edaaea3d2","Type":"ContainerDied","Data":"c0650d61cba1ece85974f7a39e9c291d788d7d7ea80d811f57b2aa5cb69c1587"}
Sep 18 14:03:51.412129 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.412090    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"
Sep 18 14:03:51.415713 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.415691    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:51.420009 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.419985    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"
Sep 18 14:03:51.454174 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.454148    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/716d1066-7007-412d-95f0-c45bb44b7304-maas-api-tls\") pod \"716d1066-7007-412d-95f0-c45bb44b7304\" (UID: \"716d1066-7007-412d-95f0-c45bb44b7304\") "
Sep 18 14:03:51.454344 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.454201    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v559b\" (UniqueName: \"kubernetes.io/projected/716d1066-7007-412d-95f0-c45bb44b7304-kube-api-access-v559b\") pod \"716d1066-7007-412d-95f0-c45bb44b7304\" (UID: \"716d1066-7007-412d-95f0-c45bb44b7304\") "
Sep 18 14:03:51.454344 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.454221    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e776755d-20d6-4c46-9589-4926ce207809-plugins-config-volume\") pod \"e776755d-20d6-4c46-9589-4926ce207809\" (UID: \"e776755d-20d6-4c46-9589-4926ce207809\") "
Sep 18 14:03:51.454344 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.454243    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/716d1066-7007-412d-95f0-c45bb44b7304-openshift-service-ca\") pod \"716d1066-7007-412d-95f0-c45bb44b7304\" (UID: \"716d1066-7007-412d-95f0-c45bb44b7304\") "
Sep 18 14:03:51.454523 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.454364    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428d445e-a2aa-47ea-9283-c26edaaea3d2-plugins-config-volume\") pod \"428d445e-a2aa-47ea-9283-c26edaaea3d2\" (UID: \"428d445e-a2aa-47ea-9283-c26edaaea3d2\") "
Sep 18 14:03:51.454523 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.454424    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6cmq\" (UniqueName: \"kubernetes.io/projected/e776755d-20d6-4c46-9589-4926ce207809-kube-api-access-s6cmq\") pod \"e776755d-20d6-4c46-9589-4926ce207809\" (UID: \"e776755d-20d6-4c46-9589-4926ce207809\") "
Sep 18 14:03:51.454523 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.454478    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stj55\" (UniqueName: \"kubernetes.io/projected/428d445e-a2aa-47ea-9283-c26edaaea3d2-kube-api-access-stj55\") pod \"428d445e-a2aa-47ea-9283-c26edaaea3d2\" (UID: \"428d445e-a2aa-47ea-9283-c26edaaea3d2\") "
Sep 18 14:03:51.454707 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.454580    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716d1066-7007-412d-95f0-c45bb44b7304-metrics-certs\") pod \"716d1066-7007-412d-95f0-c45bb44b7304\" (UID: \"716d1066-7007-412d-95f0-c45bb44b7304\") "
Sep 18 14:03:51.454707 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.454674    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716d1066-7007-412d-95f0-c45bb44b7304-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "716d1066-7007-412d-95f0-c45bb44b7304" (UID: "716d1066-7007-412d-95f0-c45bb44b7304"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:03:51.454810 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.454721    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e776755d-20d6-4c46-9589-4926ce207809-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e776755d-20d6-4c46-9589-4926ce207809" (UID: "e776755d-20d6-4c46-9589-4926ce207809"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:03:51.454810 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.454743    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428d445e-a2aa-47ea-9283-c26edaaea3d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "428d445e-a2aa-47ea-9283-c26edaaea3d2" (UID: "428d445e-a2aa-47ea-9283-c26edaaea3d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:03:51.454996 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.454978    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e776755d-20d6-4c46-9589-4926ce207809-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:03:51.455078 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.455000    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/716d1066-7007-412d-95f0-c45bb44b7304-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:03:51.455078 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.455016    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428d445e-a2aa-47ea-9283-c26edaaea3d2-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:03:51.456582 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.456557    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716d1066-7007-412d-95f0-c45bb44b7304-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "716d1066-7007-412d-95f0-c45bb44b7304" (UID: "716d1066-7007-412d-95f0-c45bb44b7304"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:03:51.457121 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.457097    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716d1066-7007-412d-95f0-c45bb44b7304-kube-api-access-v559b" (OuterVolumeSpecName: "kube-api-access-v559b") pod "716d1066-7007-412d-95f0-c45bb44b7304" (UID: "716d1066-7007-412d-95f0-c45bb44b7304"). InnerVolumeSpecName "kube-api-access-v559b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:03:51.457199 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.457130    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e776755d-20d6-4c46-9589-4926ce207809-kube-api-access-s6cmq" (OuterVolumeSpecName: "kube-api-access-s6cmq") pod "e776755d-20d6-4c46-9589-4926ce207809" (UID: "e776755d-20d6-4c46-9589-4926ce207809"). InnerVolumeSpecName "kube-api-access-s6cmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:03:51.457248 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.457206    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716d1066-7007-412d-95f0-c45bb44b7304-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "716d1066-7007-412d-95f0-c45bb44b7304" (UID: "716d1066-7007-412d-95f0-c45bb44b7304"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:03:51.457515 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.457500    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428d445e-a2aa-47ea-9283-c26edaaea3d2-kube-api-access-stj55" (OuterVolumeSpecName: "kube-api-access-stj55") pod "428d445e-a2aa-47ea-9283-c26edaaea3d2" (UID: "428d445e-a2aa-47ea-9283-c26edaaea3d2"). InnerVolumeSpecName "kube-api-access-stj55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:03:51.555718 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.555682    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/716d1066-7007-412d-95f0-c45bb44b7304-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:03:51.555718 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.555714    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v559b\" (UniqueName: \"kubernetes.io/projected/716d1066-7007-412d-95f0-c45bb44b7304-kube-api-access-v559b\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:03:51.555917 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.555732    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6cmq\" (UniqueName: \"kubernetes.io/projected/e776755d-20d6-4c46-9589-4926ce207809-kube-api-access-s6cmq\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:03:51.555917 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.555742    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stj55\" (UniqueName: \"kubernetes.io/projected/428d445e-a2aa-47ea-9283-c26edaaea3d2-kube-api-access-stj55\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:03:51.555917 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:51.555752    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716d1066-7007-412d-95f0-c45bb44b7304-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:03:52.369154 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:52.369068    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"
Sep 18 14:03:52.369154 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:52.369111    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c" event={"ID":"716d1066-7007-412d-95f0-c45bb44b7304","Type":"ContainerDied","Data":"a643d10bb27f227252a9e4355cd304d723bc0eebf454fbdb0831259fd3c3e0d9"}
Sep 18 14:03:52.369719 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:52.369187    2572 scope.go:117] "RemoveContainer" containerID="bb3cf35d3a91f61b0906f74bdda72dd9a8571381569218f20c6843e76b9ce6dd"
Sep 18 14:03:52.370830 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:52.370755    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"
Sep 18 14:03:52.370913 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:52.370838    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7" event={"ID":"428d445e-a2aa-47ea-9283-c26edaaea3d2","Type":"ContainerDied","Data":"804f06b848f234591bf43c390266350da0d0ed82b46d9f09e9cc95ec58849f2d"}
Sep 18 14:03:52.372462 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:52.372436    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg" event={"ID":"e776755d-20d6-4c46-9589-4926ce207809","Type":"ContainerDied","Data":"4bce26919ec5897c85698c2e0a49ba5e0bf0ae288823d824f71cafdc4b6c6358"}
Sep 18 14:03:52.372462 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:52.372470    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"
Sep 18 14:03:52.380637 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:52.380604    2572 scope.go:117] "RemoveContainer" containerID="c0650d61cba1ece85974f7a39e9c291d788d7d7ea80d811f57b2aa5cb69c1587"
Sep 18 14:03:52.389476 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:52.389448    2572 scope.go:117] "RemoveContainer" containerID="c29d42837282ba7846649930e1393b309406c7cbcdd851e8556064eccab0e16a"
Sep 18 14:03:52.391909 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:52.391889    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"]
Sep 18 14:03:52.395753 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:52.395732    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f17132d8-c778cc674-v2h4c"]
Sep 18 14:03:52.405903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:52.405878    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"]
Sep 18 14:03:52.410132 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:52.410109    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f17132d8-78bbbf999c-t9jvg"]
Sep 18 14:03:52.419216 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:52.419195    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"]
Sep 18 14:03:52.423718 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:52.423697    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f17132d8-587ff8cf98-np7p7"]
Sep 18 14:03:54.197856 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:54.197822    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="428d445e-a2aa-47ea-9283-c26edaaea3d2" path="/var/lib/kubelet/pods/428d445e-a2aa-47ea-9283-c26edaaea3d2/volumes"
Sep 18 14:03:54.198252 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:54.198228    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="716d1066-7007-412d-95f0-c45bb44b7304" path="/var/lib/kubelet/pods/716d1066-7007-412d-95f0-c45bb44b7304/volumes"
Sep 18 14:03:54.198558 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:03:54.198528    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e776755d-20d6-4c46-9589-4926ce207809" path="/var/lib/kubelet/pods/e776755d-20d6-4c46-9589-4926ce207809/volumes"
Sep 18 14:04:08.720979 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:08.720938    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"]
Sep 18 14:04:08.721581 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:08.721197    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h" podUID="bb363a3a-dc05-4815-bd88-31381b77538e" containerName="maas-api" containerID="cri-o://ecf0aa0166578fd04401c0cde9f1217166d89b9d15f15d75e28f38a4ede5371c" gracePeriod=30
Sep 18 14:04:08.784189 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:08.784156    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"]
Sep 18 14:04:08.784446 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:08.784421    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg" podUID="ecf52985-7740-4075-80ad-5266b1e6507d" containerName="payload-processing" containerID="cri-o://2f80b231203cbef468231f92e93c1e550598039e3c0c2d7bcc8e9485cc4dd7e4" gracePeriod=30
Sep 18 14:04:08.792410 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:08.792376    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"]
Sep 18 14:04:08.792704 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:08.792672    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt" podUID="b4bd1384-e169-41a0-8c6c-db3225aea77a" containerName="payload-pre-processing" containerID="cri-o://76749425df60b1c98ba2c2e2b73d38537b562232a2959243b9d80a36402f2166" gracePeriod=30
Sep 18 14:04:09.109750 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.109720    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"
Sep 18 14:04:09.120655 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.120623    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:04:09.122693 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.122670    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"
Sep 18 14:04:09.235111 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.235077    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb363a3a-dc05-4815-bd88-31381b77538e-openshift-service-ca\") pod \"bb363a3a-dc05-4815-bd88-31381b77538e\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") "
Sep 18 14:04:09.235111 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.235124    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf52985-7740-4075-80ad-5266b1e6507d-plugins-config-volume\") pod \"ecf52985-7740-4075-80ad-5266b1e6507d\" (UID: \"ecf52985-7740-4075-80ad-5266b1e6507d\") "
Sep 18 14:04:09.235397 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.235158    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb363a3a-dc05-4815-bd88-31381b77538e-metrics-certs\") pod \"bb363a3a-dc05-4815-bd88-31381b77538e\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") "
Sep 18 14:04:09.235397 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.235186    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb363a3a-dc05-4815-bd88-31381b77538e-maas-api-tls\") pod \"bb363a3a-dc05-4815-bd88-31381b77538e\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") "
Sep 18 14:04:09.235397 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.235220    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wj8g7\" (UniqueName: \"kubernetes.io/projected/ecf52985-7740-4075-80ad-5266b1e6507d-kube-api-access-wj8g7\") pod \"ecf52985-7740-4075-80ad-5266b1e6507d\" (UID: \"ecf52985-7740-4075-80ad-5266b1e6507d\") "
Sep 18 14:04:09.235397 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.235270    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghf5k\" (UniqueName: \"kubernetes.io/projected/bb363a3a-dc05-4815-bd88-31381b77538e-kube-api-access-ghf5k\") pod \"bb363a3a-dc05-4815-bd88-31381b77538e\" (UID: \"bb363a3a-dc05-4815-bd88-31381b77538e\") "
Sep 18 14:04:09.235397 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.235307    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bd1384-e169-41a0-8c6c-db3225aea77a-plugins-config-volume\") pod \"b4bd1384-e169-41a0-8c6c-db3225aea77a\" (UID: \"b4bd1384-e169-41a0-8c6c-db3225aea77a\") "
Sep 18 14:04:09.235715 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.235410    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5dnr\" (UniqueName: \"kubernetes.io/projected/b4bd1384-e169-41a0-8c6c-db3225aea77a-kube-api-access-p5dnr\") pod \"b4bd1384-e169-41a0-8c6c-db3225aea77a\" (UID: \"b4bd1384-e169-41a0-8c6c-db3225aea77a\") "
Sep 18 14:04:09.235715 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.235592    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb363a3a-dc05-4815-bd88-31381b77538e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bb363a3a-dc05-4815-bd88-31381b77538e" (UID: "bb363a3a-dc05-4815-bd88-31381b77538e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:04:09.236046 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.235904    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb363a3a-dc05-4815-bd88-31381b77538e-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:04:09.236046 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.236006    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf52985-7740-4075-80ad-5266b1e6507d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecf52985-7740-4075-80ad-5266b1e6507d" (UID: "ecf52985-7740-4075-80ad-5266b1e6507d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:04:09.236442 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.236350    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4bd1384-e169-41a0-8c6c-db3225aea77a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b4bd1384-e169-41a0-8c6c-db3225aea77a" (UID: "b4bd1384-e169-41a0-8c6c-db3225aea77a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:04:09.238045 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.238005    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb363a3a-dc05-4815-bd88-31381b77538e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bb363a3a-dc05-4815-bd88-31381b77538e" (UID: "bb363a3a-dc05-4815-bd88-31381b77538e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:04:09.238129 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.238108    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb363a3a-dc05-4815-bd88-31381b77538e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bb363a3a-dc05-4815-bd88-31381b77538e" (UID: "bb363a3a-dc05-4815-bd88-31381b77538e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:04:09.238248 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.238229    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4bd1384-e169-41a0-8c6c-db3225aea77a-kube-api-access-p5dnr" (OuterVolumeSpecName: "kube-api-access-p5dnr") pod "b4bd1384-e169-41a0-8c6c-db3225aea77a" (UID: "b4bd1384-e169-41a0-8c6c-db3225aea77a"). InnerVolumeSpecName "kube-api-access-p5dnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:04:09.238304 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.238271    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb363a3a-dc05-4815-bd88-31381b77538e-kube-api-access-ghf5k" (OuterVolumeSpecName: "kube-api-access-ghf5k") pod "bb363a3a-dc05-4815-bd88-31381b77538e" (UID: "bb363a3a-dc05-4815-bd88-31381b77538e"). InnerVolumeSpecName "kube-api-access-ghf5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:04:09.238468 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.238444    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf52985-7740-4075-80ad-5266b1e6507d-kube-api-access-wj8g7" (OuterVolumeSpecName: "kube-api-access-wj8g7") pod "ecf52985-7740-4075-80ad-5266b1e6507d" (UID: "ecf52985-7740-4075-80ad-5266b1e6507d"). InnerVolumeSpecName "kube-api-access-wj8g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:04:09.336439 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.336350    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5dnr\" (UniqueName: \"kubernetes.io/projected/b4bd1384-e169-41a0-8c6c-db3225aea77a-kube-api-access-p5dnr\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:04:09.336439 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.336380    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf52985-7740-4075-80ad-5266b1e6507d-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:04:09.336439 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.336393    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb363a3a-dc05-4815-bd88-31381b77538e-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:04:09.336439 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.336405    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb363a3a-dc05-4815-bd88-31381b77538e-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:04:09.336439 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.336416    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj8g7\" (UniqueName: \"kubernetes.io/projected/ecf52985-7740-4075-80ad-5266b1e6507d-kube-api-access-wj8g7\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:04:09.336439 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.336428    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghf5k\" (UniqueName: \"kubernetes.io/projected/bb363a3a-dc05-4815-bd88-31381b77538e-kube-api-access-ghf5k\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:04:09.336439 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.336440    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bd1384-e169-41a0-8c6c-db3225aea77a-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:04:09.477723 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.477689    2572 generic.go:356] "Generic (PLEG): container finished" podID="bb363a3a-dc05-4815-bd88-31381b77538e" containerID="ecf0aa0166578fd04401c0cde9f1217166d89b9d15f15d75e28f38a4ede5371c" exitCode=0
Sep 18 14:04:09.477913 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.477735    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"
Sep 18 14:04:09.477913 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.477777    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h" event={"ID":"bb363a3a-dc05-4815-bd88-31381b77538e","Type":"ContainerDied","Data":"ecf0aa0166578fd04401c0cde9f1217166d89b9d15f15d75e28f38a4ede5371c"}
Sep 18 14:04:09.477913 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.477837    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h" event={"ID":"bb363a3a-dc05-4815-bd88-31381b77538e","Type":"ContainerDied","Data":"b1a6bc7e84524ecfeee4e73007dd702dafaf44b8bef89c7dd3007a7a02b01a5f"}
Sep 18 14:04:09.477913 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.477864    2572 scope.go:117] "RemoveContainer" containerID="ecf0aa0166578fd04401c0cde9f1217166d89b9d15f15d75e28f38a4ede5371c"
Sep 18 14:04:09.479225 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.479204    2572 generic.go:356] "Generic (PLEG): container finished" podID="ecf52985-7740-4075-80ad-5266b1e6507d" containerID="2f80b231203cbef468231f92e93c1e550598039e3c0c2d7bcc8e9485cc4dd7e4" exitCode=0
Sep 18 14:04:09.479357 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.479302    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"
Sep 18 14:04:09.479357 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.479300    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg" event={"ID":"ecf52985-7740-4075-80ad-5266b1e6507d","Type":"ContainerDied","Data":"2f80b231203cbef468231f92e93c1e550598039e3c0c2d7bcc8e9485cc4dd7e4"}
Sep 18 14:04:09.479466 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.479359    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg" event={"ID":"ecf52985-7740-4075-80ad-5266b1e6507d","Type":"ContainerDied","Data":"6017bdfafbbbbb9ed6f24bbe467b62e73a542d77373d6eb2ec0d9c62287d6bef"}
Sep 18 14:04:09.480935 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.480914    2572 generic.go:356] "Generic (PLEG): container finished" podID="b4bd1384-e169-41a0-8c6c-db3225aea77a" containerID="76749425df60b1c98ba2c2e2b73d38537b562232a2959243b9d80a36402f2166" exitCode=0
Sep 18 14:04:09.481044 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.480946    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt" event={"ID":"b4bd1384-e169-41a0-8c6c-db3225aea77a","Type":"ContainerDied","Data":"76749425df60b1c98ba2c2e2b73d38537b562232a2959243b9d80a36402f2166"}
Sep 18 14:04:09.481044 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.480987    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt" event={"ID":"b4bd1384-e169-41a0-8c6c-db3225aea77a","Type":"ContainerDied","Data":"70d6b5c8bd5eb26e3f6fcb4caf4fc595452546b3dfd395973ca705dbfe1cb81e"}
Sep 18 14:04:09.481044 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.480997    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"
Sep 18 14:04:09.489480 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.489365    2572 scope.go:117] "RemoveContainer" containerID="ecf0aa0166578fd04401c0cde9f1217166d89b9d15f15d75e28f38a4ede5371c"
Sep 18 14:04:09.489756 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:04:09.489728    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf0aa0166578fd04401c0cde9f1217166d89b9d15f15d75e28f38a4ede5371c\": container with ID starting with ecf0aa0166578fd04401c0cde9f1217166d89b9d15f15d75e28f38a4ede5371c not found: ID does not exist" containerID="ecf0aa0166578fd04401c0cde9f1217166d89b9d15f15d75e28f38a4ede5371c"
Sep 18 14:04:09.489836 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.489759    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf0aa0166578fd04401c0cde9f1217166d89b9d15f15d75e28f38a4ede5371c"} err="failed to get container status \"ecf0aa0166578fd04401c0cde9f1217166d89b9d15f15d75e28f38a4ede5371c\": rpc error: code = NotFound desc = could not find container \"ecf0aa0166578fd04401c0cde9f1217166d89b9d15f15d75e28f38a4ede5371c\": container with ID starting with ecf0aa0166578fd04401c0cde9f1217166d89b9d15f15d75e28f38a4ede5371c not found: ID does not exist"
Sep 18 14:04:09.489836 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.489780    2572 scope.go:117] "RemoveContainer" containerID="2f80b231203cbef468231f92e93c1e550598039e3c0c2d7bcc8e9485cc4dd7e4"
Sep 18 14:04:09.498307 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.498290    2572 scope.go:117] "RemoveContainer" containerID="2f80b231203cbef468231f92e93c1e550598039e3c0c2d7bcc8e9485cc4dd7e4"
Sep 18 14:04:09.498634 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:04:09.498609    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f80b231203cbef468231f92e93c1e550598039e3c0c2d7bcc8e9485cc4dd7e4\": container with ID starting with 2f80b231203cbef468231f92e93c1e550598039e3c0c2d7bcc8e9485cc4dd7e4 not found: ID does not exist" containerID="2f80b231203cbef468231f92e93c1e550598039e3c0c2d7bcc8e9485cc4dd7e4"
Sep 18 14:04:09.498741 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.498634    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f80b231203cbef468231f92e93c1e550598039e3c0c2d7bcc8e9485cc4dd7e4"} err="failed to get container status \"2f80b231203cbef468231f92e93c1e550598039e3c0c2d7bcc8e9485cc4dd7e4\": rpc error: code = NotFound desc = could not find container \"2f80b231203cbef468231f92e93c1e550598039e3c0c2d7bcc8e9485cc4dd7e4\": container with ID starting with 2f80b231203cbef468231f92e93c1e550598039e3c0c2d7bcc8e9485cc4dd7e4 not found: ID does not exist"
Sep 18 14:04:09.498741 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.498651    2572 scope.go:117] "RemoveContainer" containerID="76749425df60b1c98ba2c2e2b73d38537b562232a2959243b9d80a36402f2166"
Sep 18 14:04:09.505370 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.505343    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"]
Sep 18 14:04:09.507940 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.507917    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ec85d5-77df596b79-fjd6h"]
Sep 18 14:04:09.508011 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.507974    2572 scope.go:117] "RemoveContainer" containerID="76749425df60b1c98ba2c2e2b73d38537b562232a2959243b9d80a36402f2166"
Sep 18 14:04:09.508274 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:04:09.508255    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76749425df60b1c98ba2c2e2b73d38537b562232a2959243b9d80a36402f2166\": container with ID starting with 76749425df60b1c98ba2c2e2b73d38537b562232a2959243b9d80a36402f2166 not found: ID does not exist" containerID="76749425df60b1c98ba2c2e2b73d38537b562232a2959243b9d80a36402f2166"
Sep 18 14:04:09.508317 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.508280    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76749425df60b1c98ba2c2e2b73d38537b562232a2959243b9d80a36402f2166"} err="failed to get container status \"76749425df60b1c98ba2c2e2b73d38537b562232a2959243b9d80a36402f2166\": rpc error: code = NotFound desc = could not find container \"76749425df60b1c98ba2c2e2b73d38537b562232a2959243b9d80a36402f2166\": container with ID starting with 76749425df60b1c98ba2c2e2b73d38537b562232a2959243b9d80a36402f2166 not found: ID does not exist"
Sep 18 14:04:09.519780 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.519754    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"]
Sep 18 14:04:09.524978 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.524955    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ec85d5-7d8b4d9845-4qltg"]
Sep 18 14:04:09.535766 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.535743    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"]
Sep 18 14:04:09.539987 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:09.539967    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ec85d5-5f685474fd-g8cbt"]
Sep 18 14:04:10.198082 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:10.198050    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4bd1384-e169-41a0-8c6c-db3225aea77a" path="/var/lib/kubelet/pods/b4bd1384-e169-41a0-8c6c-db3225aea77a/volumes"
Sep 18 14:04:10.198491 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:10.198477    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb363a3a-dc05-4815-bd88-31381b77538e" path="/var/lib/kubelet/pods/bb363a3a-dc05-4815-bd88-31381b77538e/volumes"
Sep 18 14:04:10.198820 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:10.198808    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecf52985-7740-4075-80ad-5266b1e6507d" path="/var/lib/kubelet/pods/ecf52985-7740-4075-80ad-5266b1e6507d/volumes"
Sep 18 14:04:19.869137 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869095    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"]
Sep 18 14:04:19.869545 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869518    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb363a3a-dc05-4815-bd88-31381b77538e" containerName="maas-api"
Sep 18 14:04:19.869592 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869548    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb363a3a-dc05-4815-bd88-31381b77538e" containerName="maas-api"
Sep 18 14:04:19.869592 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869559    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="716d1066-7007-412d-95f0-c45bb44b7304" containerName="maas-api"
Sep 18 14:04:19.869592 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869564    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="716d1066-7007-412d-95f0-c45bb44b7304" containerName="maas-api"
Sep 18 14:04:19.869592 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869570    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e776755d-20d6-4c46-9589-4926ce207809" containerName="payload-processing"
Sep 18 14:04:19.869592 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869576    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e776755d-20d6-4c46-9589-4926ce207809" containerName="payload-processing"
Sep 18 14:04:19.869592 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869583    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4bd1384-e169-41a0-8c6c-db3225aea77a" containerName="payload-pre-processing"
Sep 18 14:04:19.869592 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869588    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4bd1384-e169-41a0-8c6c-db3225aea77a" containerName="payload-pre-processing"
Sep 18 14:04:19.869828 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869607    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf52985-7740-4075-80ad-5266b1e6507d" containerName="payload-processing"
Sep 18 14:04:19.869828 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869612    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf52985-7740-4075-80ad-5266b1e6507d" containerName="payload-processing"
Sep 18 14:04:19.869828 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869625    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="428d445e-a2aa-47ea-9283-c26edaaea3d2" containerName="payload-pre-processing"
Sep 18 14:04:19.869828 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869630    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="428d445e-a2aa-47ea-9283-c26edaaea3d2" containerName="payload-pre-processing"
Sep 18 14:04:19.869828 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869692    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecf52985-7740-4075-80ad-5266b1e6507d" containerName="payload-processing"
Sep 18 14:04:19.869828 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869702    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb363a3a-dc05-4815-bd88-31381b77538e" containerName="maas-api"
Sep 18 14:04:19.869828 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869709    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="716d1066-7007-412d-95f0-c45bb44b7304" containerName="maas-api"
Sep 18 14:04:19.869828 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869715    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4bd1384-e169-41a0-8c6c-db3225aea77a" containerName="payload-pre-processing"
Sep 18 14:04:19.869828 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869721    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e776755d-20d6-4c46-9589-4926ce207809" containerName="payload-processing"
Sep 18 14:04:19.869828 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.869725    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="428d445e-a2aa-47ea-9283-c26edaaea3d2" containerName="payload-pre-processing"
Sep 18 14:04:19.878719 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.878689    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:19.882726 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.882701    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-d4e72b-serving-cert\""
Sep 18 14:04:19.886830 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:19.886799    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"]
Sep 18 14:04:20.045934 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.045891    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/214b90fb-cc46-4298-99f3-8c81dfb90e80-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:20.045934 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.045934    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214b90fb-cc46-4298-99f3-8c81dfb90e80-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:20.046184 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.045990    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq5rn\" (UniqueName: \"kubernetes.io/projected/214b90fb-cc46-4298-99f3-8c81dfb90e80-kube-api-access-hq5rn\") pod \"maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:20.046184 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.046033    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/214b90fb-cc46-4298-99f3-8c81dfb90e80-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:20.147131 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.147036    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/214b90fb-cc46-4298-99f3-8c81dfb90e80-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:20.147131 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.147075    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214b90fb-cc46-4298-99f3-8c81dfb90e80-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:20.147131 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.147116    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5rn\" (UniqueName: \"kubernetes.io/projected/214b90fb-cc46-4298-99f3-8c81dfb90e80-kube-api-access-hq5rn\") pod \"maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:20.147411 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.147154    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/214b90fb-cc46-4298-99f3-8c81dfb90e80-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:20.147411 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:04:20.147342    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-serving-cert: secret "maas-api-e2e-shared-a-w2-d4e72b-serving-cert" not found
Sep 18 14:04:20.147513 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:04:20.147449    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/214b90fb-cc46-4298-99f3-8c81dfb90e80-maas-api-tls podName:214b90fb-cc46-4298-99f3-8c81dfb90e80 nodeName:}" failed. No retries permitted until 2026-09-18 14:04:20.647423797 +0000 UTC m=+1196.582388925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/214b90fb-cc46-4298-99f3-8c81dfb90e80-maas-api-tls") pod "maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz" (UID: "214b90fb-cc46-4298-99f3-8c81dfb90e80") : secret "maas-api-e2e-shared-a-w2-d4e72b-serving-cert" not found
Sep 18 14:04:20.147798 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.147780    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/214b90fb-cc46-4298-99f3-8c81dfb90e80-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:20.149640 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.149623    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214b90fb-cc46-4298-99f3-8c81dfb90e80-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:20.157359 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.157337    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq5rn\" (UniqueName: \"kubernetes.io/projected/214b90fb-cc46-4298-99f3-8c81dfb90e80-kube-api-access-hq5rn\") pod \"maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:20.562592 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.562561    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"]
Sep 18 14:04:20.566291 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.566273    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"
Sep 18 14:04:20.569611 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.569578    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d4e72b-dockercfg-kxh77\""
Sep 18 14:04:20.569758 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.569720    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d4e72b\""
Sep 18 14:04:20.578698 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.578674    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"]
Sep 18 14:04:20.609961 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.609922    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"]
Sep 18 14:04:20.619998 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.619965    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"
Sep 18 14:04:20.628876 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.628846    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"]
Sep 18 14:04:20.651647 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.651609    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/214b90fb-cc46-4298-99f3-8c81dfb90e80-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:20.655353 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.655272    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/214b90fb-cc46-4298-99f3-8c81dfb90e80-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:20.752281 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.752236    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea32efc-eb86-433a-b6fb-ce0dba10de6d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb\" (UID: \"1ea32efc-eb86-433a-b6fb-ce0dba10de6d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"
Sep 18 14:04:20.752494 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.752300    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjsn5\" (UniqueName: \"kubernetes.io/projected/1ea32efc-eb86-433a-b6fb-ce0dba10de6d-kube-api-access-qjsn5\") pod \"payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb\" (UID: \"1ea32efc-eb86-433a-b6fb-ce0dba10de6d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"
Sep 18 14:04:20.752494 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.752324    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7wrf\" (UniqueName: \"kubernetes.io/projected/8e854df5-adb6-414e-a465-43bc706df627-kube-api-access-w7wrf\") pod \"payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g\" (UID: \"8e854df5-adb6-414e-a465-43bc706df627\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"
Sep 18 14:04:20.752494 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.752404    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e854df5-adb6-414e-a465-43bc706df627-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g\" (UID: \"8e854df5-adb6-414e-a465-43bc706df627\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"
Sep 18 14:04:20.793342 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.793298    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:20.854336 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.853147    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e854df5-adb6-414e-a465-43bc706df627-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g\" (UID: \"8e854df5-adb6-414e-a465-43bc706df627\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"
Sep 18 14:04:20.854336 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.853263    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea32efc-eb86-433a-b6fb-ce0dba10de6d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb\" (UID: \"1ea32efc-eb86-433a-b6fb-ce0dba10de6d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"
Sep 18 14:04:20.854336 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.853331    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjsn5\" (UniqueName: \"kubernetes.io/projected/1ea32efc-eb86-433a-b6fb-ce0dba10de6d-kube-api-access-qjsn5\") pod \"payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb\" (UID: \"1ea32efc-eb86-433a-b6fb-ce0dba10de6d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"
Sep 18 14:04:20.854336 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.853363    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wrf\" (UniqueName: \"kubernetes.io/projected/8e854df5-adb6-414e-a465-43bc706df627-kube-api-access-w7wrf\") pod \"payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g\" (UID: \"8e854df5-adb6-414e-a465-43bc706df627\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"
Sep 18 14:04:20.854336 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.854028    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e854df5-adb6-414e-a465-43bc706df627-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g\" (UID: \"8e854df5-adb6-414e-a465-43bc706df627\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"
Sep 18 14:04:20.854749 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.854461    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea32efc-eb86-433a-b6fb-ce0dba10de6d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb\" (UID: \"1ea32efc-eb86-433a-b6fb-ce0dba10de6d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"
Sep 18 14:04:20.862834 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.862738    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjsn5\" (UniqueName: \"kubernetes.io/projected/1ea32efc-eb86-433a-b6fb-ce0dba10de6d-kube-api-access-qjsn5\") pod \"payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb\" (UID: \"1ea32efc-eb86-433a-b6fb-ce0dba10de6d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"
Sep 18 14:04:20.863726 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.863555    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7wrf\" (UniqueName: \"kubernetes.io/projected/8e854df5-adb6-414e-a465-43bc706df627-kube-api-access-w7wrf\") pod \"payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g\" (UID: \"8e854df5-adb6-414e-a465-43bc706df627\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"
Sep 18 14:04:20.878450 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.878299    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"
Sep 18 14:04:20.935609 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:20.935556    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"
Sep 18 14:04:21.070484 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:21.070457    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"]
Sep 18 14:04:21.072236 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:04:21.072211    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214b90fb_cc46_4298_99f3_8c81dfb90e80.slice/crio-6eff8f3eee9085a2b412d9dcb34c6d3a2de8bc075ade012bccb4f2623a4637e6 WatchSource:0}: Error finding container 6eff8f3eee9085a2b412d9dcb34c6d3a2de8bc075ade012bccb4f2623a4637e6: Status 404 returned error can't find the container with id 6eff8f3eee9085a2b412d9dcb34c6d3a2de8bc075ade012bccb4f2623a4637e6
Sep 18 14:04:21.155492 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:21.155458    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"]
Sep 18 14:04:21.157243 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:04:21.157211    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea32efc_eb86_433a_b6fb_ce0dba10de6d.slice/crio-72c85f7a4926412262e84e4380c286785d33edcf034759725eb3263fe58f2ec9 WatchSource:0}: Error finding container 72c85f7a4926412262e84e4380c286785d33edcf034759725eb3263fe58f2ec9: Status 404 returned error can't find the container with id 72c85f7a4926412262e84e4380c286785d33edcf034759725eb3263fe58f2ec9
Sep 18 14:04:21.221955 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:21.221920    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"]
Sep 18 14:04:21.223663 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:04:21.223632    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e854df5_adb6_414e_a465_43bc706df627.slice/crio-89a5afe73332da4e6a57a89323920295f0b5822a53bec400b2fc5e6f82e21763 WatchSource:0}: Error finding container 89a5afe73332da4e6a57a89323920295f0b5822a53bec400b2fc5e6f82e21763: Status 404 returned error can't find the container with id 89a5afe73332da4e6a57a89323920295f0b5822a53bec400b2fc5e6f82e21763
Sep 18 14:04:21.561864 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:21.561773    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g" event={"ID":"8e854df5-adb6-414e-a465-43bc706df627","Type":"ContainerStarted","Data":"afb02be9c1ea48a35fdb0476489ae80d08e9e48dcc84d5407fca05649d120d53"}
Sep 18 14:04:21.561864 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:21.561862    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g" event={"ID":"8e854df5-adb6-414e-a465-43bc706df627","Type":"ContainerStarted","Data":"89a5afe73332da4e6a57a89323920295f0b5822a53bec400b2fc5e6f82e21763"}
Sep 18 14:04:21.562121 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:21.561911    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"
Sep 18 14:04:21.563653 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:21.563625    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb" event={"ID":"1ea32efc-eb86-433a-b6fb-ce0dba10de6d","Type":"ContainerStarted","Data":"8dbb0f2894e0eb8584bb077375b07fb594a7d1ff6f6f8df20ca4950c3160b26d"}
Sep 18 14:04:21.563792 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:21.563677    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb" event={"ID":"1ea32efc-eb86-433a-b6fb-ce0dba10de6d","Type":"ContainerStarted","Data":"72c85f7a4926412262e84e4380c286785d33edcf034759725eb3263fe58f2ec9"}
Sep 18 14:04:21.563792 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:21.563709    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"
Sep 18 14:04:21.565332 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:21.565309    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz" event={"ID":"214b90fb-cc46-4298-99f3-8c81dfb90e80","Type":"ContainerStarted","Data":"93e797444529a02d0c4baec57a68ae4c64116ef8db3683cae247e20029c7dfd0"}
Sep 18 14:04:21.565438 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:21.565351    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz" event={"ID":"214b90fb-cc46-4298-99f3-8c81dfb90e80","Type":"ContainerStarted","Data":"6eff8f3eee9085a2b412d9dcb34c6d3a2de8bc075ade012bccb4f2623a4637e6"}
Sep 18 14:04:21.565438 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:21.565372    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:21.581616 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:21.581559    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g" podStartSLOduration=1.581523593 podStartE2EDuration="1.581523593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:04:21.579125132 +0000 UTC m=+1197.514090276" watchObservedRunningTime="2026-09-18 14:04:21.581523593 +0000 UTC m=+1197.516488736"
Sep 18 14:04:21.601753 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:21.601688    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz" podStartSLOduration=2.227624599 podStartE2EDuration="2.601666337s" totalImagesPullingTime="374.041738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:04:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:04:21.598654496 +0000 UTC m=+1197.533619638" watchObservedRunningTime="2026-09-18 14:04:21.601666337 +0000 UTC m=+1197.536631485"
Sep 18 14:04:21.620593 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:21.620516    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb" podStartSLOduration=1.6204979819999998 podStartE2EDuration="1.620497982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:04:21.615783497 +0000 UTC m=+1197.550748652" watchObservedRunningTime="2026-09-18 14:04:21.620497982 +0000 UTC m=+1197.555463127"
Sep 18 14:04:26.244094 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.244051    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 14:04:26.246054 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.246031    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 14:04:26.458326 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.458288    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"]
Sep 18 14:04:26.463755 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.463736    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.466055 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.466030    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-40970f\"/\"distinct-w0-40970f-kserve-self-signed-certs\""
Sep 18 14:04:26.470032 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.470008    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"]
Sep 18 14:04:26.506854 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.506759    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r2r5\" (UniqueName: \"kubernetes.io/projected/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-kube-api-access-5r2r5\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.506854 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.506801    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-dshm\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.506854 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.506826    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-tls-certs\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.507148 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.506947    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-model-cache\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.507148 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.506984    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-home\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.507148 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.507053    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-tmp-dir\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.608083 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.608047    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r2r5\" (UniqueName: \"kubernetes.io/projected/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-kube-api-access-5r2r5\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.608083 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.608084    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-dshm\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.608352 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.608104    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-tls-certs\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.608352 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.608159    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-model-cache\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.608352 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.608182    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-home\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.608352 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.608210    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-tmp-dir\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.608654 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.608629    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-tmp-dir\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.608762 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.608638    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-model-cache\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.608762 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.608697    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-home\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.610397 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.610377    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-dshm\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.610718 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.610700    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-tls-certs\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.617579 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.617527    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r2r5\" (UniqueName: \"kubernetes.io/projected/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-kube-api-access-5r2r5\") pod \"distinct-w0-40970f-kserve-7499dd4449-z4q96\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") " pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:26.774841 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:26.774751    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:27.074354 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:27.074324    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"]
Sep 18 14:04:27.075978 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:04:27.075950    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod049791ec_ffa8_4a2f_a1ea_1f07361a4d1a.slice/crio-da5e0a5bfc3e1128ed7d0388754cefcc7eb8f28f9c2d1d1df7f5e800ad94b496 WatchSource:0}: Error finding container da5e0a5bfc3e1128ed7d0388754cefcc7eb8f28f9c2d1d1df7f5e800ad94b496: Status 404 returned error can't find the container with id da5e0a5bfc3e1128ed7d0388754cefcc7eb8f28f9c2d1d1df7f5e800ad94b496
Sep 18 14:04:27.577582 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:27.577549    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:04:27.602598 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:27.602556    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96" event={"ID":"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a","Type":"ContainerStarted","Data":"663bacb2a92d024dcc2c3e104ea6bf5e56d5e514795c57950ef833d85cb5052e"}
Sep 18 14:04:27.602768 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:27.602608    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96" event={"ID":"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a","Type":"ContainerStarted","Data":"da5e0a5bfc3e1128ed7d0388754cefcc7eb8f28f9c2d1d1df7f5e800ad94b496"}
Sep 18 14:04:27.631312 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:27.631262    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96" podStartSLOduration=1.631243732 podStartE2EDuration="1.631243732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:04:27.62917381 +0000 UTC m=+1203.564138956" watchObservedRunningTime="2026-09-18 14:04:27.631243732 +0000 UTC m=+1203.566208912"
Sep 18 14:04:32.573743 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:32.573703    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"
Sep 18 14:04:32.574167 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:32.573778    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"
Sep 18 14:04:36.775473 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:36.775425    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:36.775998 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:36.775489    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:36.787969 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:36.787936    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:37.673656 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:37.673627    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:04:54.374228 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:54.374190    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"]
Sep 18 14:04:54.379194 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:54.379172    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:04:54.381423 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:54.381399    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-75dcf1-serving-cert\""
Sep 18 14:04:54.392384 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:54.392356    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"]
Sep 18 14:04:54.488389 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:54.488301    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnlpw\" (UniqueName: \"kubernetes.io/projected/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-kube-api-access-pnlpw\") pod \"maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:04:54.488626 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:54.488414    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:04:54.488626 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:54.488589    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:04:54.488761 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:54.488659    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:04:54.590057 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:54.590016    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnlpw\" (UniqueName: \"kubernetes.io/projected/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-kube-api-access-pnlpw\") pod \"maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:04:54.590057 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:54.590057    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:04:54.590281 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:54.590113    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:04:54.590281 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:54.590147    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:04:54.590281 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:04:54.590260    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-serving-cert: secret "maas-api-e2e-shared-b-w2-75dcf1-serving-cert" not found
Sep 18 14:04:54.590389 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:04:54.590323    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-maas-api-tls podName:7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186 nodeName:}" failed. No retries permitted until 2026-09-18 14:04:55.090307126 +0000 UTC m=+1231.025272262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-maas-api-tls") pod "maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4" (UID: "7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186") : secret "maas-api-e2e-shared-b-w2-75dcf1-serving-cert" not found
Sep 18 14:04:54.590874 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:54.590855    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:04:54.592629 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:54.592610    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:04:54.597994 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:54.597969    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnlpw\" (UniqueName: \"kubernetes.io/projected/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-kube-api-access-pnlpw\") pod \"maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:04:55.052815 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.052775    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"]
Sep 18 14:04:55.057173 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.057150    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"
Sep 18 14:04:55.059376 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.059355    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-75dcf1-dockercfg-47jmd\""
Sep 18 14:04:55.059488 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.059377    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-75dcf1\""
Sep 18 14:04:55.067780 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.067747    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"]
Sep 18 14:04:55.094512 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.094479    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:04:55.096975 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.096952    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:04:55.195811 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.195768    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w82wr\" (UniqueName: \"kubernetes.io/projected/24e558a8-4d01-44cd-aeb0-9e98d2879e95-kube-api-access-w82wr\") pod \"payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz\" (UID: \"24e558a8-4d01-44cd-aeb0-9e98d2879e95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"
Sep 18 14:04:55.196010 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.195944    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e558a8-4d01-44cd-aeb0-9e98d2879e95-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz\" (UID: \"24e558a8-4d01-44cd-aeb0-9e98d2879e95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"
Sep 18 14:04:55.294082 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.294043    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:04:55.297696 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.297170    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e558a8-4d01-44cd-aeb0-9e98d2879e95-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz\" (UID: \"24e558a8-4d01-44cd-aeb0-9e98d2879e95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"
Sep 18 14:04:55.297696 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.297229    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w82wr\" (UniqueName: \"kubernetes.io/projected/24e558a8-4d01-44cd-aeb0-9e98d2879e95-kube-api-access-w82wr\") pod \"payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz\" (UID: \"24e558a8-4d01-44cd-aeb0-9e98d2879e95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"
Sep 18 14:04:55.298186 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.298162    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e558a8-4d01-44cd-aeb0-9e98d2879e95-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz\" (UID: \"24e558a8-4d01-44cd-aeb0-9e98d2879e95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"
Sep 18 14:04:55.308718 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.308692    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w82wr\" (UniqueName: \"kubernetes.io/projected/24e558a8-4d01-44cd-aeb0-9e98d2879e95-kube-api-access-w82wr\") pod \"payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz\" (UID: \"24e558a8-4d01-44cd-aeb0-9e98d2879e95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"
Sep 18 14:04:55.372678 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.372502    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"
Sep 18 14:04:55.575296 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.575266    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"]
Sep 18 14:04:55.576707 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:04:55.576676    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc27cda_c5ee_4c6c_a154_ea6b8fe5a186.slice/crio-33077757b49263104fa2b70f6119dff61e8009957a85aada7fe9a53774919534 WatchSource:0}: Error finding container 33077757b49263104fa2b70f6119dff61e8009957a85aada7fe9a53774919534: Status 404 returned error can't find the container with id 33077757b49263104fa2b70f6119dff61e8009957a85aada7fe9a53774919534
Sep 18 14:04:55.578164 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.578145    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:04:55.664298 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.664265    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"]
Sep 18 14:04:55.666334 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:04:55.666298    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e558a8_4d01_44cd_aeb0_9e98d2879e95.slice/crio-5575b4e7e6ea33f32f6c4a20c7e8422f7e559c66f736f1e11a0ecd9f1c6689d9 WatchSource:0}: Error finding container 5575b4e7e6ea33f32f6c4a20c7e8422f7e559c66f736f1e11a0ecd9f1c6689d9: Status 404 returned error can't find the container with id 5575b4e7e6ea33f32f6c4a20c7e8422f7e559c66f736f1e11a0ecd9f1c6689d9
Sep 18 14:04:55.782615 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.782574    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz" event={"ID":"24e558a8-4d01-44cd-aeb0-9e98d2879e95","Type":"ContainerStarted","Data":"d112cc1bad1ca64156d2110d3b23ec585a1eecb5815daebf8810f332cf53b7bf"}
Sep 18 14:04:55.782806 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.782638    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz" event={"ID":"24e558a8-4d01-44cd-aeb0-9e98d2879e95","Type":"ContainerStarted","Data":"5575b4e7e6ea33f32f6c4a20c7e8422f7e559c66f736f1e11a0ecd9f1c6689d9"}
Sep 18 14:04:55.782806 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.782709    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"
Sep 18 14:04:55.784040 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.784013    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4" event={"ID":"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186","Type":"ContainerStarted","Data":"33077757b49263104fa2b70f6119dff61e8009957a85aada7fe9a53774919534"}
Sep 18 14:04:55.802587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:55.802504    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz" podStartSLOduration=0.802482312 podStartE2EDuration="802.482312ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:04:55.800467015 +0000 UTC m=+1231.735432164" watchObservedRunningTime="2026-09-18 14:04:55.802482312 +0000 UTC m=+1231.737447459"
Sep 18 14:04:56.791620 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:56.791577    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4" event={"ID":"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186","Type":"ContainerStarted","Data":"58d6f2a241fde94f96724e0297d808ce593daadc55ec5f6448ea9d1214440b60"}
Sep 18 14:04:56.792109 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:56.791731    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:04:56.811866 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:04:56.811806    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4" podStartSLOduration=2.3034752960000002 podStartE2EDuration="2.81178573s" totalImagesPullingTime="508.310434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:04:56.810354128 +0000 UTC m=+1232.745319274" watchObservedRunningTime="2026-09-18 14:04:56.81178573 +0000 UTC m=+1232.746750878"
Sep 18 14:05:02.803025 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:02.802996    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:05:06.794290 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:06.794258    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"
Sep 18 14:05:38.174723 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.174505    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"]
Sep 18 14:05:38.180347 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.180321    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:05:38.182977 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.182952    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-31cca2d1-serving-cert\""
Sep 18 14:05:38.190551 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.190495    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"]
Sep 18 14:05:38.325697 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.325657    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/460bfda5-43dd-4d0f-8234-380ad7ab3916-metrics-certs\") pod \"maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps\" (UID: \"460bfda5-43dd-4d0f-8234-380ad7ab3916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:05:38.325881 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.325703    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r6vv\" (UniqueName: \"kubernetes.io/projected/460bfda5-43dd-4d0f-8234-380ad7ab3916-kube-api-access-2r6vv\") pod \"maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps\" (UID: \"460bfda5-43dd-4d0f-8234-380ad7ab3916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:05:38.325881 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.325787    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/460bfda5-43dd-4d0f-8234-380ad7ab3916-openshift-service-ca\") pod \"maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps\" (UID: \"460bfda5-43dd-4d0f-8234-380ad7ab3916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:05:38.326017 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.325950    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/460bfda5-43dd-4d0f-8234-380ad7ab3916-maas-api-tls\") pod \"maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps\" (UID: \"460bfda5-43dd-4d0f-8234-380ad7ab3916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:05:38.427174 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.427066    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/460bfda5-43dd-4d0f-8234-380ad7ab3916-metrics-certs\") pod \"maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps\" (UID: \"460bfda5-43dd-4d0f-8234-380ad7ab3916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:05:38.427174 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.427112    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r6vv\" (UniqueName: \"kubernetes.io/projected/460bfda5-43dd-4d0f-8234-380ad7ab3916-kube-api-access-2r6vv\") pod \"maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps\" (UID: \"460bfda5-43dd-4d0f-8234-380ad7ab3916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:05:38.427410 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.427231    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/460bfda5-43dd-4d0f-8234-380ad7ab3916-openshift-service-ca\") pod \"maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps\" (UID: \"460bfda5-43dd-4d0f-8234-380ad7ab3916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:05:38.427410 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.427363    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/460bfda5-43dd-4d0f-8234-380ad7ab3916-maas-api-tls\") pod \"maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps\" (UID: \"460bfda5-43dd-4d0f-8234-380ad7ab3916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:05:38.427988 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.427968    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/460bfda5-43dd-4d0f-8234-380ad7ab3916-openshift-service-ca\") pod \"maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps\" (UID: \"460bfda5-43dd-4d0f-8234-380ad7ab3916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:05:38.429708 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.429684    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/460bfda5-43dd-4d0f-8234-380ad7ab3916-metrics-certs\") pod \"maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps\" (UID: \"460bfda5-43dd-4d0f-8234-380ad7ab3916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:05:38.429871 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.429853    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/460bfda5-43dd-4d0f-8234-380ad7ab3916-maas-api-tls\") pod \"maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps\" (UID: \"460bfda5-43dd-4d0f-8234-380ad7ab3916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:05:38.435577 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.435551    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r6vv\" (UniqueName: \"kubernetes.io/projected/460bfda5-43dd-4d0f-8234-380ad7ab3916-kube-api-access-2r6vv\") pod \"maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps\" (UID: \"460bfda5-43dd-4d0f-8234-380ad7ab3916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:05:38.496026 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.495980    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:05:38.776876 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:38.776847    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"]
Sep 18 14:05:38.778660 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:05:38.778626    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460bfda5_43dd_4d0f_8234_380ad7ab3916.slice/crio-fc84fc66a27e49532c094746ad4f54248c4f73ca58344a6faee5935c9dbad5d2 WatchSource:0}: Error finding container fc84fc66a27e49532c094746ad4f54248c4f73ca58344a6faee5935c9dbad5d2: Status 404 returned error can't find the container with id fc84fc66a27e49532c094746ad4f54248c4f73ca58344a6faee5935c9dbad5d2
Sep 18 14:05:39.059848 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:39.059757    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps" event={"ID":"460bfda5-43dd-4d0f-8234-380ad7ab3916","Type":"ContainerStarted","Data":"fc84fc66a27e49532c094746ad4f54248c4f73ca58344a6faee5935c9dbad5d2"}
Sep 18 14:05:40.068329 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:40.068286    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps" event={"ID":"460bfda5-43dd-4d0f-8234-380ad7ab3916","Type":"ContainerStarted","Data":"845ad8331ad4e70fe18fd003a638d0937b4ba6da70312787d834762f9c3a5a91"}
Sep 18 14:05:40.068764 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:40.068381    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:05:40.089840 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:40.089783    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps" podStartSLOduration=1.652204503 podStartE2EDuration="2.089765135s" totalImagesPullingTime="437.560632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:05:40.087943351 +0000 UTC m=+1276.022908497" watchObservedRunningTime="2026-09-18 14:05:40.089765135 +0000 UTC m=+1276.024730280"
Sep 18 14:05:46.080880 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:05:46.080852    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:06:04.617647 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:04.617610    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"]
Sep 18 14:06:04.618402 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:04.618368    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh" podUID="a88ecf2b-8437-4a65-9eb1-1c80054a9ae4" containerName="maas-api" containerID="cri-o://84ab4fe908e5fb22d19ec1e1ed30bcf222ea46b9009eeab4160417016e7b1aac" gracePeriod=30
Sep 18 14:06:04.883618 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:04.883548    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 14:06:04.997078 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:04.997044    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-maas-api-tls\") pod \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") "
Sep 18 14:06:04.997289 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:04.997128    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rzp4\" (UniqueName: \"kubernetes.io/projected/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-kube-api-access-2rzp4\") pod \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") "
Sep 18 14:06:04.997289 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:04.997245    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-openshift-service-ca\") pod \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") "
Sep 18 14:06:04.997634 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:04.997608    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-metrics-certs\") pod \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\" (UID: \"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4\") "
Sep 18 14:06:04.997769 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:04.997626    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a88ecf2b-8437-4a65-9eb1-1c80054a9ae4" (UID: "a88ecf2b-8437-4a65-9eb1-1c80054a9ae4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:06:04.997953 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:04.997933    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:06:04.999992 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:04.999935    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a88ecf2b-8437-4a65-9eb1-1c80054a9ae4" (UID: "a88ecf2b-8437-4a65-9eb1-1c80054a9ae4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:06:05.000102 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:05.000053    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a88ecf2b-8437-4a65-9eb1-1c80054a9ae4" (UID: "a88ecf2b-8437-4a65-9eb1-1c80054a9ae4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:06:05.000102 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:05.000088    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-kube-api-access-2rzp4" (OuterVolumeSpecName: "kube-api-access-2rzp4") pod "a88ecf2b-8437-4a65-9eb1-1c80054a9ae4" (UID: "a88ecf2b-8437-4a65-9eb1-1c80054a9ae4"). InnerVolumeSpecName "kube-api-access-2rzp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:06:05.099293 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:05.099252    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:06:05.099293 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:05.099285    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:06:05.099293 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:05.099294    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rzp4\" (UniqueName: \"kubernetes.io/projected/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4-kube-api-access-2rzp4\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:06:05.218246 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:05.218209    2572 generic.go:356] "Generic (PLEG): container finished" podID="a88ecf2b-8437-4a65-9eb1-1c80054a9ae4" containerID="84ab4fe908e5fb22d19ec1e1ed30bcf222ea46b9009eeab4160417016e7b1aac" exitCode=0
Sep 18 14:06:05.218430 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:05.218276    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"
Sep 18 14:06:05.218430 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:05.218281    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh" event={"ID":"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4","Type":"ContainerDied","Data":"84ab4fe908e5fb22d19ec1e1ed30bcf222ea46b9009eeab4160417016e7b1aac"}
Sep 18 14:06:05.218430 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:05.218403    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh" event={"ID":"a88ecf2b-8437-4a65-9eb1-1c80054a9ae4","Type":"ContainerDied","Data":"165ab52cccc4d09363a8ec94a8e231197c5ba7c7499d7b8de6f31bbf533371db"}
Sep 18 14:06:05.218430 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:05.218427    2572 scope.go:117] "RemoveContainer" containerID="84ab4fe908e5fb22d19ec1e1ed30bcf222ea46b9009eeab4160417016e7b1aac"
Sep 18 14:06:05.229231 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:05.229209    2572 scope.go:117] "RemoveContainer" containerID="84ab4fe908e5fb22d19ec1e1ed30bcf222ea46b9009eeab4160417016e7b1aac"
Sep 18 14:06:05.229632 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:06:05.229611    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ab4fe908e5fb22d19ec1e1ed30bcf222ea46b9009eeab4160417016e7b1aac\": container with ID starting with 84ab4fe908e5fb22d19ec1e1ed30bcf222ea46b9009eeab4160417016e7b1aac not found: ID does not exist" containerID="84ab4fe908e5fb22d19ec1e1ed30bcf222ea46b9009eeab4160417016e7b1aac"
Sep 18 14:06:05.229734 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:05.229642    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ab4fe908e5fb22d19ec1e1ed30bcf222ea46b9009eeab4160417016e7b1aac"} err="failed to get container status \"84ab4fe908e5fb22d19ec1e1ed30bcf222ea46b9009eeab4160417016e7b1aac\": rpc error: code = NotFound desc = could not find container \"84ab4fe908e5fb22d19ec1e1ed30bcf222ea46b9009eeab4160417016e7b1aac\": container with ID starting with 84ab4fe908e5fb22d19ec1e1ed30bcf222ea46b9009eeab4160417016e7b1aac not found: ID does not exist"
Sep 18 14:06:05.240847 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:05.240820    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"]
Sep 18 14:06:05.245905 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:05.245877    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-036fc2-68bc85d658-rw7fh"]
Sep 18 14:06:06.199202 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:06:06.199164    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a88ecf2b-8437-4a65-9eb1-1c80054a9ae4" path="/var/lib/kubelet/pods/a88ecf2b-8437-4a65-9eb1-1c80054a9ae4/volumes"
Sep 18 14:07:48.684152 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.684098    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"]
Sep 18 14:07:48.684795 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.684632    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a88ecf2b-8437-4a65-9eb1-1c80054a9ae4" containerName="maas-api"
Sep 18 14:07:48.684795 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.684652    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a88ecf2b-8437-4a65-9eb1-1c80054a9ae4" containerName="maas-api"
Sep 18 14:07:48.684795 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.684772    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a88ecf2b-8437-4a65-9eb1-1c80054a9ae4" containerName="maas-api"
Sep 18 14:07:48.688008 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.687989    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:07:48.690411 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.690388    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-d634d2-serving-cert\""
Sep 18 14:07:48.700151 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.700099    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"]
Sep 18 14:07:48.735855 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.735811    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/976fdf61-e598-4e60-b736-fc42cea248e6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:07:48.736041 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.735864    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976fdf61-e598-4e60-b736-fc42cea248e6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:07:48.736041 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.735924    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/976fdf61-e598-4e60-b736-fc42cea248e6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:07:48.736150 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.736048    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffzpf\" (UniqueName: \"kubernetes.io/projected/976fdf61-e598-4e60-b736-fc42cea248e6-kube-api-access-ffzpf\") pod \"maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:07:48.837221 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.837174    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976fdf61-e598-4e60-b736-fc42cea248e6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:07:48.837221 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.837229    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/976fdf61-e598-4e60-b736-fc42cea248e6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:07:48.837477 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.837301    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffzpf\" (UniqueName: \"kubernetes.io/projected/976fdf61-e598-4e60-b736-fc42cea248e6-kube-api-access-ffzpf\") pod \"maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:07:48.837477 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:07:48.837325    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-serving-cert: secret "maas-api-e2e-shared-b-w3-d634d2-serving-cert" not found
Sep 18 14:07:48.837477 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.837366    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/976fdf61-e598-4e60-b736-fc42cea248e6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:07:48.837477 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:07:48.837390    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/976fdf61-e598-4e60-b736-fc42cea248e6-maas-api-tls podName:976fdf61-e598-4e60-b736-fc42cea248e6 nodeName:}" failed. No retries permitted until 2026-09-18 14:07:49.337372645 +0000 UTC m=+1405.272337786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/976fdf61-e598-4e60-b736-fc42cea248e6-maas-api-tls") pod "maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl" (UID: "976fdf61-e598-4e60-b736-fc42cea248e6") : secret "maas-api-e2e-shared-b-w3-d634d2-serving-cert" not found
Sep 18 14:07:48.838128 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.838102    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/976fdf61-e598-4e60-b736-fc42cea248e6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:07:48.839878 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.839860    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/976fdf61-e598-4e60-b736-fc42cea248e6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:07:48.845889 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:48.845865    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffzpf\" (UniqueName: \"kubernetes.io/projected/976fdf61-e598-4e60-b736-fc42cea248e6-kube-api-access-ffzpf\") pod \"maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:07:49.343422 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:49.343384    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976fdf61-e598-4e60-b736-fc42cea248e6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:07:49.345815 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:49.345785    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976fdf61-e598-4e60-b736-fc42cea248e6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:07:49.601029 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:49.600939    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:07:49.887253 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:49.887218    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"]
Sep 18 14:07:49.888735 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:07:49.888703    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976fdf61_e598_4e60_b736_fc42cea248e6.slice/crio-fdf18ea9f480fb787b44c953f069023cfecdb974118b54c6ccccf0b7102bdd14 WatchSource:0}: Error finding container fdf18ea9f480fb787b44c953f069023cfecdb974118b54c6ccccf0b7102bdd14: Status 404 returned error can't find the container with id fdf18ea9f480fb787b44c953f069023cfecdb974118b54c6ccccf0b7102bdd14
Sep 18 14:07:50.863921 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:50.863874    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl" event={"ID":"976fdf61-e598-4e60-b736-fc42cea248e6","Type":"ContainerStarted","Data":"e24f110b57f5465fd97f1db2d2bbecdf7011641d4a7ab36733f1c19bfd01e2ce"}
Sep 18 14:07:50.864088 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:50.863944    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:07:50.864088 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:50.863961    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl" event={"ID":"976fdf61-e598-4e60-b736-fc42cea248e6","Type":"ContainerStarted","Data":"fdf18ea9f480fb787b44c953f069023cfecdb974118b54c6ccccf0b7102bdd14"}
Sep 18 14:07:50.881316 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:50.881260    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl" podStartSLOduration=2.477946997 podStartE2EDuration="2.881241317s" totalImagesPullingTime="403.29432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:07:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:07:50.879182177 +0000 UTC m=+1406.814147323" watchObservedRunningTime="2026-09-18 14:07:50.881241317 +0000 UTC m=+1406.816206465"
Sep 18 14:07:56.875842 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:07:56.875809    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:08:09.368093 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.367995    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"]
Sep 18 14:08:09.368597 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.368241    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps" podUID="460bfda5-43dd-4d0f-8234-380ad7ab3916" containerName="maas-api" containerID="cri-o://845ad8331ad4e70fe18fd003a638d0937b4ba6da70312787d834762f9c3a5a91" gracePeriod=30
Sep 18 14:08:09.648030 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.648001    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:08:09.755061 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.755021    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/460bfda5-43dd-4d0f-8234-380ad7ab3916-openshift-service-ca\") pod \"460bfda5-43dd-4d0f-8234-380ad7ab3916\" (UID: \"460bfda5-43dd-4d0f-8234-380ad7ab3916\") "
Sep 18 14:08:09.755261 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.755080    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2r6vv\" (UniqueName: \"kubernetes.io/projected/460bfda5-43dd-4d0f-8234-380ad7ab3916-kube-api-access-2r6vv\") pod \"460bfda5-43dd-4d0f-8234-380ad7ab3916\" (UID: \"460bfda5-43dd-4d0f-8234-380ad7ab3916\") "
Sep 18 14:08:09.755261 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.755146    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/460bfda5-43dd-4d0f-8234-380ad7ab3916-metrics-certs\") pod \"460bfda5-43dd-4d0f-8234-380ad7ab3916\" (UID: \"460bfda5-43dd-4d0f-8234-380ad7ab3916\") "
Sep 18 14:08:09.755261 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.755209    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/460bfda5-43dd-4d0f-8234-380ad7ab3916-maas-api-tls\") pod \"460bfda5-43dd-4d0f-8234-380ad7ab3916\" (UID: \"460bfda5-43dd-4d0f-8234-380ad7ab3916\") "
Sep 18 14:08:09.755427 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.755392    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460bfda5-43dd-4d0f-8234-380ad7ab3916-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "460bfda5-43dd-4d0f-8234-380ad7ab3916" (UID: "460bfda5-43dd-4d0f-8234-380ad7ab3916"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:08:09.755629 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.755611    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/460bfda5-43dd-4d0f-8234-380ad7ab3916-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:08:09.757470 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.757440    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/460bfda5-43dd-4d0f-8234-380ad7ab3916-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "460bfda5-43dd-4d0f-8234-380ad7ab3916" (UID: "460bfda5-43dd-4d0f-8234-380ad7ab3916"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:08:09.757596 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.757495    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460bfda5-43dd-4d0f-8234-380ad7ab3916-kube-api-access-2r6vv" (OuterVolumeSpecName: "kube-api-access-2r6vv") pod "460bfda5-43dd-4d0f-8234-380ad7ab3916" (UID: "460bfda5-43dd-4d0f-8234-380ad7ab3916"). InnerVolumeSpecName "kube-api-access-2r6vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:08:09.757596 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.757502    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/460bfda5-43dd-4d0f-8234-380ad7ab3916-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "460bfda5-43dd-4d0f-8234-380ad7ab3916" (UID: "460bfda5-43dd-4d0f-8234-380ad7ab3916"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:08:09.856190 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.856148    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/460bfda5-43dd-4d0f-8234-380ad7ab3916-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:08:09.856190 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.856182    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/460bfda5-43dd-4d0f-8234-380ad7ab3916-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:08:09.856190 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.856192    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r6vv\" (UniqueName: \"kubernetes.io/projected/460bfda5-43dd-4d0f-8234-380ad7ab3916-kube-api-access-2r6vv\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:08:09.980793 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.980752    2572 generic.go:356] "Generic (PLEG): container finished" podID="460bfda5-43dd-4d0f-8234-380ad7ab3916" containerID="845ad8331ad4e70fe18fd003a638d0937b4ba6da70312787d834762f9c3a5a91" exitCode=0
Sep 18 14:08:09.981000 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.980811    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"
Sep 18 14:08:09.981000 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.980838    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps" event={"ID":"460bfda5-43dd-4d0f-8234-380ad7ab3916","Type":"ContainerDied","Data":"845ad8331ad4e70fe18fd003a638d0937b4ba6da70312787d834762f9c3a5a91"}
Sep 18 14:08:09.981000 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.980887    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps" event={"ID":"460bfda5-43dd-4d0f-8234-380ad7ab3916","Type":"ContainerDied","Data":"fc84fc66a27e49532c094746ad4f54248c4f73ca58344a6faee5935c9dbad5d2"}
Sep 18 14:08:09.981000 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.980909    2572 scope.go:117] "RemoveContainer" containerID="845ad8331ad4e70fe18fd003a638d0937b4ba6da70312787d834762f9c3a5a91"
Sep 18 14:08:09.991558 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.991523    2572 scope.go:117] "RemoveContainer" containerID="845ad8331ad4e70fe18fd003a638d0937b4ba6da70312787d834762f9c3a5a91"
Sep 18 14:08:09.991871 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:08:09.991842    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845ad8331ad4e70fe18fd003a638d0937b4ba6da70312787d834762f9c3a5a91\": container with ID starting with 845ad8331ad4e70fe18fd003a638d0937b4ba6da70312787d834762f9c3a5a91 not found: ID does not exist" containerID="845ad8331ad4e70fe18fd003a638d0937b4ba6da70312787d834762f9c3a5a91"
Sep 18 14:08:09.991972 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:09.991877    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845ad8331ad4e70fe18fd003a638d0937b4ba6da70312787d834762f9c3a5a91"} err="failed to get container status \"845ad8331ad4e70fe18fd003a638d0937b4ba6da70312787d834762f9c3a5a91\": rpc error: code = NotFound desc = could not find container \"845ad8331ad4e70fe18fd003a638d0937b4ba6da70312787d834762f9c3a5a91\": container with ID starting with 845ad8331ad4e70fe18fd003a638d0937b4ba6da70312787d834762f9c3a5a91 not found: ID does not exist"
Sep 18 14:08:10.008947 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:10.008921    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"]
Sep 18 14:08:10.015312 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:10.015273    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-31cca2d1-74d6d5656-pjmps"]
Sep 18 14:08:10.197701 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:08:10.197663    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="460bfda5-43dd-4d0f-8234-380ad7ab3916" path="/var/lib/kubelet/pods/460bfda5-43dd-4d0f-8234-380ad7ab3916/volumes"
Sep 18 14:09:26.285013 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:09:26.284983    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 14:09:26.289304 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:09:26.289280    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 14:10:46.713747 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.713709    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"]
Sep 18 14:10:46.714356 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.714333    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="460bfda5-43dd-4d0f-8234-380ad7ab3916" containerName="maas-api"
Sep 18 14:10:46.714356 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.714348    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="460bfda5-43dd-4d0f-8234-380ad7ab3916" containerName="maas-api"
Sep 18 14:10:46.714567 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.714471    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="460bfda5-43dd-4d0f-8234-380ad7ab3916" containerName="maas-api"
Sep 18 14:10:46.718850 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.718813    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:10:46.721129 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.721103    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7b5a6c-serving-cert\""
Sep 18 14:10:46.726784 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.726758    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"]
Sep 18 14:10:46.820836 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.820793    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-metrics-certs\") pod \"maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf\" (UID: \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:10:46.821170 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.821143    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-openshift-service-ca\") pod \"maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf\" (UID: \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:10:46.821469 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.821449    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-maas-api-tls\") pod \"maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf\" (UID: \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:10:46.821722 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.821659    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gbq7\" (UniqueName: \"kubernetes.io/projected/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-kube-api-access-4gbq7\") pod \"maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf\" (UID: \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:10:46.922924 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.922886    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-maas-api-tls\") pod \"maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf\" (UID: \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:10:46.922924 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.922933    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gbq7\" (UniqueName: \"kubernetes.io/projected/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-kube-api-access-4gbq7\") pod \"maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf\" (UID: \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:10:46.923191 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.922965    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-metrics-certs\") pod \"maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf\" (UID: \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:10:46.923191 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.923099    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-openshift-service-ca\") pod \"maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf\" (UID: \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:10:46.923785 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.923747    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-openshift-service-ca\") pod \"maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf\" (UID: \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:10:46.925474 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.925446    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-maas-api-tls\") pod \"maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf\" (UID: \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:10:46.925630 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.925480    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-metrics-certs\") pod \"maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf\" (UID: \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:10:46.933547 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:46.933516    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gbq7\" (UniqueName: \"kubernetes.io/projected/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-kube-api-access-4gbq7\") pod \"maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf\" (UID: \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:10:47.033619 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:47.033474    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:10:47.329584 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:47.329548    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"]
Sep 18 14:10:47.331192 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:10:47.331166    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963f2b89_0b7c_40d5_b96b_d7d1cb72ea6f.slice/crio-872c5f41902e833416706dbd44162b31bbd4ee6a068752395a1d6d1310945306 WatchSource:0}: Error finding container 872c5f41902e833416706dbd44162b31bbd4ee6a068752395a1d6d1310945306: Status 404 returned error can't find the container with id 872c5f41902e833416706dbd44162b31bbd4ee6a068752395a1d6d1310945306
Sep 18 14:10:47.332551 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:47.332520    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:10:47.588892 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:47.588808    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"]
Sep 18 14:10:47.593631 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:47.593597    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"
Sep 18 14:10:47.596425 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:47.596400    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7b5a6c-dockercfg-9595n\""
Sep 18 14:10:47.596896 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:47.596876    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7b5a6c\""
Sep 18 14:10:47.605938 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:47.605911    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"]
Sep 18 14:10:47.733453 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:47.733417    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d16b854d-505d-4355-8fd8-5b5d27741794-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb\" (UID: \"d16b854d-505d-4355-8fd8-5b5d27741794\") " pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"
Sep 18 14:10:47.733895 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:47.733612    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftdz5\" (UniqueName: \"kubernetes.io/projected/d16b854d-505d-4355-8fd8-5b5d27741794-kube-api-access-ftdz5\") pod \"payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb\" (UID: \"d16b854d-505d-4355-8fd8-5b5d27741794\") " pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"
Sep 18 14:10:47.834851 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:47.834812    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftdz5\" (UniqueName: \"kubernetes.io/projected/d16b854d-505d-4355-8fd8-5b5d27741794-kube-api-access-ftdz5\") pod \"payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb\" (UID: \"d16b854d-505d-4355-8fd8-5b5d27741794\") " pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"
Sep 18 14:10:47.835028 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:47.834865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d16b854d-505d-4355-8fd8-5b5d27741794-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb\" (UID: \"d16b854d-505d-4355-8fd8-5b5d27741794\") " pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"
Sep 18 14:10:47.835665 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:47.835646    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d16b854d-505d-4355-8fd8-5b5d27741794-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb\" (UID: \"d16b854d-505d-4355-8fd8-5b5d27741794\") " pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"
Sep 18 14:10:47.843932 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:47.843879    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftdz5\" (UniqueName: \"kubernetes.io/projected/d16b854d-505d-4355-8fd8-5b5d27741794-kube-api-access-ftdz5\") pod \"payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb\" (UID: \"d16b854d-505d-4355-8fd8-5b5d27741794\") " pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"
Sep 18 14:10:47.906585 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:47.906548    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"
Sep 18 14:10:47.962132 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:47.962042    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf" event={"ID":"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f","Type":"ContainerStarted","Data":"872c5f41902e833416706dbd44162b31bbd4ee6a068752395a1d6d1310945306"}
Sep 18 14:10:48.196879 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:10:48.196849    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16b854d_505d_4355_8fd8_5b5d27741794.slice/crio-085965082de84df9ada6120946285e39b1cfc43de8d516fa62729bdf0788f929 WatchSource:0}: Error finding container 085965082de84df9ada6120946285e39b1cfc43de8d516fa62729bdf0788f929: Status 404 returned error can't find the container with id 085965082de84df9ada6120946285e39b1cfc43de8d516fa62729bdf0788f929
Sep 18 14:10:48.199891 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:48.199863    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"]
Sep 18 14:10:48.970586 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:48.970513    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf" event={"ID":"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f","Type":"ContainerStarted","Data":"242b3e065c2fd356b1319a97f69b23ad5a969cffbbc7d335fc78c0ebfbd946ee"}
Sep 18 14:10:48.971101 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:48.970726    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:10:48.972277 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:48.972256    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb" event={"ID":"d16b854d-505d-4355-8fd8-5b5d27741794","Type":"ContainerStarted","Data":"85ece69ad951f70f26c6fb30877b6eb201e5c4fe1e0b1ec00e225357ce499d21"}
Sep 18 14:10:48.972388 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:48.972291    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb" event={"ID":"d16b854d-505d-4355-8fd8-5b5d27741794","Type":"ContainerStarted","Data":"085965082de84df9ada6120946285e39b1cfc43de8d516fa62729bdf0788f929"}
Sep 18 14:10:48.972388 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:48.972340    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"
Sep 18 14:10:48.993066 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:48.993002    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf" podStartSLOduration=2.411236003 podStartE2EDuration="2.992980996s" totalImagesPullingTime="581.744993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:10:48.989580623 +0000 UTC m=+1584.924545750" watchObservedRunningTime="2026-09-18 14:10:48.992980996 +0000 UTC m=+1584.927946147"
Sep 18 14:10:49.009429 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:49.009373    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb" podStartSLOduration=2.009353776 podStartE2EDuration="2.009353776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:10:49.006609368 +0000 UTC m=+1584.941574515" watchObservedRunningTime="2026-09-18 14:10:49.009353776 +0000 UTC m=+1584.944318989"
Sep 18 14:10:54.985725 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:54.985695    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:10:59.981949 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:10:59.981917    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"
Sep 18 14:11:23.437669 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:23.437585    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"]
Sep 18 14:11:23.442650 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:23.442623    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"
Sep 18 14:11:23.445394 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:23.445366    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3c1586-dockercfg-nnc5k\""
Sep 18 14:11:23.445587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:23.445404    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3c1586\""
Sep 18 14:11:23.451696 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:23.451668    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"]
Sep 18 14:11:23.500208 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:23.500165    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76817456-fb85-4dec-8cca-c0e741359e60-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs\" (UID: \"76817456-fb85-4dec-8cca-c0e741359e60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"
Sep 18 14:11:23.500208 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:23.500221    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfgcp\" (UniqueName: \"kubernetes.io/projected/76817456-fb85-4dec-8cca-c0e741359e60-kube-api-access-nfgcp\") pod \"payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs\" (UID: \"76817456-fb85-4dec-8cca-c0e741359e60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"
Sep 18 14:11:23.601049 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:23.601001    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76817456-fb85-4dec-8cca-c0e741359e60-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs\" (UID: \"76817456-fb85-4dec-8cca-c0e741359e60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"
Sep 18 14:11:23.601234 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:23.601062    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfgcp\" (UniqueName: \"kubernetes.io/projected/76817456-fb85-4dec-8cca-c0e741359e60-kube-api-access-nfgcp\") pod \"payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs\" (UID: \"76817456-fb85-4dec-8cca-c0e741359e60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"
Sep 18 14:11:23.601777 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:23.601758    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76817456-fb85-4dec-8cca-c0e741359e60-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs\" (UID: \"76817456-fb85-4dec-8cca-c0e741359e60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"
Sep 18 14:11:23.609936 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:23.609916    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfgcp\" (UniqueName: \"kubernetes.io/projected/76817456-fb85-4dec-8cca-c0e741359e60-kube-api-access-nfgcp\") pod \"payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs\" (UID: \"76817456-fb85-4dec-8cca-c0e741359e60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"
Sep 18 14:11:23.755492 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:23.755451    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"
Sep 18 14:11:24.032442 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:24.032405    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"]
Sep 18 14:11:24.033472 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:11:24.033447    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76817456_fb85_4dec_8cca_c0e741359e60.slice/crio-3bf0eb62e3eb0a4d799708cacc441fdfc8950abe2145471efd4016f8764da7a8 WatchSource:0}: Error finding container 3bf0eb62e3eb0a4d799708cacc441fdfc8950abe2145471efd4016f8764da7a8: Status 404 returned error can't find the container with id 3bf0eb62e3eb0a4d799708cacc441fdfc8950abe2145471efd4016f8764da7a8
Sep 18 14:11:24.200780 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:24.200741    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs" event={"ID":"76817456-fb85-4dec-8cca-c0e741359e60","Type":"ContainerStarted","Data":"40a61dc2dfa29545db3ad05aa42084c41c7b2dd08a1aeeb4fc52109aab8e8b21"}
Sep 18 14:11:24.200958 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:24.200901    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"
Sep 18 14:11:24.200958 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:24.200923    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs" event={"ID":"76817456-fb85-4dec-8cca-c0e741359e60","Type":"ContainerStarted","Data":"3bf0eb62e3eb0a4d799708cacc441fdfc8950abe2145471efd4016f8764da7a8"}
Sep 18 14:11:24.217316 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:24.217261    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs" podStartSLOduration=1.2172405099999999 podStartE2EDuration="1.21724051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:24.215876358 +0000 UTC m=+1620.150841503" watchObservedRunningTime="2026-09-18 14:11:24.21724051 +0000 UTC m=+1620.152205656"
Sep 18 14:11:35.210421 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:35.210382    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"
Sep 18 14:11:58.123419 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:58.123380    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"]
Sep 18 14:11:58.146121 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:11:58.146088    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"]
Sep 18 14:12:05.605769 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:05.605731    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"]
Sep 18 14:12:05.606256 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:05.606004    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf" podUID="963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f" containerName="maas-api" containerID="cri-o://242b3e065c2fd356b1319a97f69b23ad5a969cffbbc7d335fc78c0ebfbd946ee" gracePeriod=30
Sep 18 14:12:05.674124 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:05.674082    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"]
Sep 18 14:12:05.674410 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:05.674338    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb" podUID="d16b854d-505d-4355-8fd8-5b5d27741794" containerName="payload-processing" containerID="cri-o://85ece69ad951f70f26c6fb30877b6eb201e5c4fe1e0b1ec00e225357ce499d21" gracePeriod=30
Sep 18 14:12:05.944464 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:05.944438    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:12:05.982305 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:05.982280    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"
Sep 18 14:12:06.036826 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.036791    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gbq7\" (UniqueName: \"kubernetes.io/projected/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-kube-api-access-4gbq7\") pod \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\" (UID: \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\") "
Sep 18 14:12:06.037029 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.036877    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-metrics-certs\") pod \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\" (UID: \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\") "
Sep 18 14:12:06.037029 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.036924    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-maas-api-tls\") pod \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\" (UID: \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\") "
Sep 18 14:12:06.037029 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.037000    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-openshift-service-ca\") pod \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\" (UID: \"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f\") "
Sep 18 14:12:06.037552 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.037498    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f" (UID: "963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:12:06.039200 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.039174    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f" (UID: "963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:12:06.039401 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.039383    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f" (UID: "963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:12:06.039475 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.039424    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-kube-api-access-4gbq7" (OuterVolumeSpecName: "kube-api-access-4gbq7") pod "963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f" (UID: "963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f"). InnerVolumeSpecName "kube-api-access-4gbq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:12:06.138499 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.138469    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d16b854d-505d-4355-8fd8-5b5d27741794-plugins-config-volume\") pod \"d16b854d-505d-4355-8fd8-5b5d27741794\" (UID: \"d16b854d-505d-4355-8fd8-5b5d27741794\") "
Sep 18 14:12:06.138674 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.138608    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftdz5\" (UniqueName: \"kubernetes.io/projected/d16b854d-505d-4355-8fd8-5b5d27741794-kube-api-access-ftdz5\") pod \"d16b854d-505d-4355-8fd8-5b5d27741794\" (UID: \"d16b854d-505d-4355-8fd8-5b5d27741794\") "
Sep 18 14:12:06.138821 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.138803    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:06.138867 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.138824    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:06.138867 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.138835    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:06.138867 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.138844    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gbq7\" (UniqueName: \"kubernetes.io/projected/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f-kube-api-access-4gbq7\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:06.138978 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.138868    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16b854d-505d-4355-8fd8-5b5d27741794-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d16b854d-505d-4355-8fd8-5b5d27741794" (UID: "d16b854d-505d-4355-8fd8-5b5d27741794"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:12:06.140831 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.140804    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16b854d-505d-4355-8fd8-5b5d27741794-kube-api-access-ftdz5" (OuterVolumeSpecName: "kube-api-access-ftdz5") pod "d16b854d-505d-4355-8fd8-5b5d27741794" (UID: "d16b854d-505d-4355-8fd8-5b5d27741794"). InnerVolumeSpecName "kube-api-access-ftdz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:12:06.240323 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.240292    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftdz5\" (UniqueName: \"kubernetes.io/projected/d16b854d-505d-4355-8fd8-5b5d27741794-kube-api-access-ftdz5\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:06.240323 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.240320    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d16b854d-505d-4355-8fd8-5b5d27741794-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:06.467955 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.467871    2572 generic.go:356] "Generic (PLEG): container finished" podID="963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f" containerID="242b3e065c2fd356b1319a97f69b23ad5a969cffbbc7d335fc78c0ebfbd946ee" exitCode=0
Sep 18 14:12:06.467955 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.467924    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"
Sep 18 14:12:06.468187 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.467950    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf" event={"ID":"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f","Type":"ContainerDied","Data":"242b3e065c2fd356b1319a97f69b23ad5a969cffbbc7d335fc78c0ebfbd946ee"}
Sep 18 14:12:06.468187 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.468001    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf" event={"ID":"963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f","Type":"ContainerDied","Data":"872c5f41902e833416706dbd44162b31bbd4ee6a068752395a1d6d1310945306"}
Sep 18 14:12:06.468187 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.468024    2572 scope.go:117] "RemoveContainer" containerID="242b3e065c2fd356b1319a97f69b23ad5a969cffbbc7d335fc78c0ebfbd946ee"
Sep 18 14:12:06.469455 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.469430    2572 generic.go:356] "Generic (PLEG): container finished" podID="d16b854d-505d-4355-8fd8-5b5d27741794" containerID="85ece69ad951f70f26c6fb30877b6eb201e5c4fe1e0b1ec00e225357ce499d21" exitCode=0
Sep 18 14:12:06.469596 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.469485    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"
Sep 18 14:12:06.469596 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.469517    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb" event={"ID":"d16b854d-505d-4355-8fd8-5b5d27741794","Type":"ContainerDied","Data":"85ece69ad951f70f26c6fb30877b6eb201e5c4fe1e0b1ec00e225357ce499d21"}
Sep 18 14:12:06.469701 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.469600    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb" event={"ID":"d16b854d-505d-4355-8fd8-5b5d27741794","Type":"ContainerDied","Data":"085965082de84df9ada6120946285e39b1cfc43de8d516fa62729bdf0788f929"}
Sep 18 14:12:06.478263 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.478247    2572 scope.go:117] "RemoveContainer" containerID="242b3e065c2fd356b1319a97f69b23ad5a969cffbbc7d335fc78c0ebfbd946ee"
Sep 18 14:12:06.478599 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:12:06.478578    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242b3e065c2fd356b1319a97f69b23ad5a969cffbbc7d335fc78c0ebfbd946ee\": container with ID starting with 242b3e065c2fd356b1319a97f69b23ad5a969cffbbc7d335fc78c0ebfbd946ee not found: ID does not exist" containerID="242b3e065c2fd356b1319a97f69b23ad5a969cffbbc7d335fc78c0ebfbd946ee"
Sep 18 14:12:06.478675 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.478607    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242b3e065c2fd356b1319a97f69b23ad5a969cffbbc7d335fc78c0ebfbd946ee"} err="failed to get container status \"242b3e065c2fd356b1319a97f69b23ad5a969cffbbc7d335fc78c0ebfbd946ee\": rpc error: code = NotFound desc = could not find container \"242b3e065c2fd356b1319a97f69b23ad5a969cffbbc7d335fc78c0ebfbd946ee\": container with ID starting with 242b3e065c2fd356b1319a97f69b23ad5a969cffbbc7d335fc78c0ebfbd946ee not found: ID does not exist"
Sep 18 14:12:06.478675 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.478622    2572 scope.go:117] "RemoveContainer" containerID="85ece69ad951f70f26c6fb30877b6eb201e5c4fe1e0b1ec00e225357ce499d21"
Sep 18 14:12:06.487187 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.487165    2572 scope.go:117] "RemoveContainer" containerID="85ece69ad951f70f26c6fb30877b6eb201e5c4fe1e0b1ec00e225357ce499d21"
Sep 18 14:12:06.487525 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:12:06.487499    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ece69ad951f70f26c6fb30877b6eb201e5c4fe1e0b1ec00e225357ce499d21\": container with ID starting with 85ece69ad951f70f26c6fb30877b6eb201e5c4fe1e0b1ec00e225357ce499d21 not found: ID does not exist" containerID="85ece69ad951f70f26c6fb30877b6eb201e5c4fe1e0b1ec00e225357ce499d21"
Sep 18 14:12:06.487635 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.487525    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ece69ad951f70f26c6fb30877b6eb201e5c4fe1e0b1ec00e225357ce499d21"} err="failed to get container status \"85ece69ad951f70f26c6fb30877b6eb201e5c4fe1e0b1ec00e225357ce499d21\": rpc error: code = NotFound desc = could not find container \"85ece69ad951f70f26c6fb30877b6eb201e5c4fe1e0b1ec00e225357ce499d21\": container with ID starting with 85ece69ad951f70f26c6fb30877b6eb201e5c4fe1e0b1ec00e225357ce499d21 not found: ID does not exist"
Sep 18 14:12:06.493267 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.493245    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"]
Sep 18 14:12:06.502558 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.497038    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7b5a6c-7557c59f5c-xxktf"]
Sep 18 14:12:06.518866 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.518833    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"]
Sep 18 14:12:06.522642 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:06.522620    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7b5a6c-749c784c95-nf4xb"]
Sep 18 14:12:08.203422 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.202350    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f" path="/var/lib/kubelet/pods/963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f/volumes"
Sep 18 14:12:08.203422 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.202939    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d16b854d-505d-4355-8fd8-5b5d27741794" path="/var/lib/kubelet/pods/d16b854d-505d-4355-8fd8-5b5d27741794/volumes"
Sep 18 14:12:08.205949 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.205922    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"]
Sep 18 14:12:08.206215 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.206170    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9" podUID="c769edea-c5d3-4901-908c-c2c9faf72504" containerName="maas-api" containerID="cri-o://31f1d58b05f606dd4bcc47db22fc67f96e893da5de4d467916581aaa0deb0eda" gracePeriod=30
Sep 18 14:12:08.469115 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.469050    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 14:12:08.485661 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.485629    2572 generic.go:356] "Generic (PLEG): container finished" podID="c769edea-c5d3-4901-908c-c2c9faf72504" containerID="31f1d58b05f606dd4bcc47db22fc67f96e893da5de4d467916581aaa0deb0eda" exitCode=0
Sep 18 14:12:08.485827 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.485674    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"
Sep 18 14:12:08.485827 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.485714    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9" event={"ID":"c769edea-c5d3-4901-908c-c2c9faf72504","Type":"ContainerDied","Data":"31f1d58b05f606dd4bcc47db22fc67f96e893da5de4d467916581aaa0deb0eda"}
Sep 18 14:12:08.485827 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.485765    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9" event={"ID":"c769edea-c5d3-4901-908c-c2c9faf72504","Type":"ContainerDied","Data":"c2817dcf5f1f28424f5f0fec197cff3cede96597fc193832746d6c81b5892501"}
Sep 18 14:12:08.485827 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.485790    2572 scope.go:117] "RemoveContainer" containerID="31f1d58b05f606dd4bcc47db22fc67f96e893da5de4d467916581aaa0deb0eda"
Sep 18 14:12:08.495352 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.495333    2572 scope.go:117] "RemoveContainer" containerID="31f1d58b05f606dd4bcc47db22fc67f96e893da5de4d467916581aaa0deb0eda"
Sep 18 14:12:08.495648 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:12:08.495629    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f1d58b05f606dd4bcc47db22fc67f96e893da5de4d467916581aaa0deb0eda\": container with ID starting with 31f1d58b05f606dd4bcc47db22fc67f96e893da5de4d467916581aaa0deb0eda not found: ID does not exist" containerID="31f1d58b05f606dd4bcc47db22fc67f96e893da5de4d467916581aaa0deb0eda"
Sep 18 14:12:08.495705 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.495657    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f1d58b05f606dd4bcc47db22fc67f96e893da5de4d467916581aaa0deb0eda"} err="failed to get container status \"31f1d58b05f606dd4bcc47db22fc67f96e893da5de4d467916581aaa0deb0eda\": rpc error: code = NotFound desc = could not find container \"31f1d58b05f606dd4bcc47db22fc67f96e893da5de4d467916581aaa0deb0eda\": container with ID starting with 31f1d58b05f606dd4bcc47db22fc67f96e893da5de4d467916581aaa0deb0eda not found: ID does not exist"
Sep 18 14:12:08.564030 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.563989    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c769edea-c5d3-4901-908c-c2c9faf72504-metrics-certs\") pod \"c769edea-c5d3-4901-908c-c2c9faf72504\" (UID: \"c769edea-c5d3-4901-908c-c2c9faf72504\") "
Sep 18 14:12:08.564230 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.564102    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c769edea-c5d3-4901-908c-c2c9faf72504-maas-api-tls\") pod \"c769edea-c5d3-4901-908c-c2c9faf72504\" (UID: \"c769edea-c5d3-4901-908c-c2c9faf72504\") "
Sep 18 14:12:08.564230 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.564146    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8zf5\" (UniqueName: \"kubernetes.io/projected/c769edea-c5d3-4901-908c-c2c9faf72504-kube-api-access-g8zf5\") pod \"c769edea-c5d3-4901-908c-c2c9faf72504\" (UID: \"c769edea-c5d3-4901-908c-c2c9faf72504\") "
Sep 18 14:12:08.564230 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.564209    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c769edea-c5d3-4901-908c-c2c9faf72504-openshift-service-ca\") pod \"c769edea-c5d3-4901-908c-c2c9faf72504\" (UID: \"c769edea-c5d3-4901-908c-c2c9faf72504\") "
Sep 18 14:12:08.564712 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.564679    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c769edea-c5d3-4901-908c-c2c9faf72504-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c769edea-c5d3-4901-908c-c2c9faf72504" (UID: "c769edea-c5d3-4901-908c-c2c9faf72504"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:12:08.566327 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.566299    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c769edea-c5d3-4901-908c-c2c9faf72504-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c769edea-c5d3-4901-908c-c2c9faf72504" (UID: "c769edea-c5d3-4901-908c-c2c9faf72504"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:12:08.566425 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.566406    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c769edea-c5d3-4901-908c-c2c9faf72504-kube-api-access-g8zf5" (OuterVolumeSpecName: "kube-api-access-g8zf5") pod "c769edea-c5d3-4901-908c-c2c9faf72504" (UID: "c769edea-c5d3-4901-908c-c2c9faf72504"). InnerVolumeSpecName "kube-api-access-g8zf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:12:08.566726 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.566706    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c769edea-c5d3-4901-908c-c2c9faf72504-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c769edea-c5d3-4901-908c-c2c9faf72504" (UID: "c769edea-c5d3-4901-908c-c2c9faf72504"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:12:08.665254 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.665213    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c769edea-c5d3-4901-908c-c2c9faf72504-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:08.665254 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.665249    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c769edea-c5d3-4901-908c-c2c9faf72504-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:08.665474 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.665264    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c769edea-c5d3-4901-908c-c2c9faf72504-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:08.665474 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.665276    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8zf5\" (UniqueName: \"kubernetes.io/projected/c769edea-c5d3-4901-908c-c2c9faf72504-kube-api-access-g8zf5\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:08.822544 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.822499    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"]
Sep 18 14:12:08.829256 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:08.829227    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-40970f-596c68794c-9kwb9"]
Sep 18 14:12:10.197840 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:10.197807    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c769edea-c5d3-4901-908c-c2c9faf72504" path="/var/lib/kubelet/pods/c769edea-c5d3-4901-908c-c2c9faf72504/volumes"
Sep 18 14:12:13.143639 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.143583    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96" podUID="049791ec-ffa8-4a2f-a1ea-1f07361a4d1a" containerName="main" containerID="cri-o://663bacb2a92d024dcc2c3e104ea6bf5e56d5e514795c57950ef833d85cb5052e" gracePeriod=45
Sep 18 14:12:13.166195 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.166158    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8" podUID="34192080-ca2c-431e-b7bb-988e5cf7ff8f" containerName="main" containerID="cri-o://664b6522b7970086ad3a4abaca6886075d57d4c50f4a3056e670e2faaf28433a" gracePeriod=45
Sep 18 14:12:13.450604 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.450580    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:12:13.454376 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.454347    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:12:13.528903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.528860    2572 generic.go:356] "Generic (PLEG): container finished" podID="34192080-ca2c-431e-b7bb-988e5cf7ff8f" containerID="664b6522b7970086ad3a4abaca6886075d57d4c50f4a3056e670e2faaf28433a" exitCode=0
Sep 18 14:12:13.528903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.528900    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"
Sep 18 14:12:13.529151 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.528938    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8" event={"ID":"34192080-ca2c-431e-b7bb-988e5cf7ff8f","Type":"ContainerDied","Data":"664b6522b7970086ad3a4abaca6886075d57d4c50f4a3056e670e2faaf28433a"}
Sep 18 14:12:13.529151 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.528999    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8" event={"ID":"34192080-ca2c-431e-b7bb-988e5cf7ff8f","Type":"ContainerDied","Data":"008752a9d7031d0efdcb9e1157d4f209dd5cf3e51efb48a245f1fea7746cd9b1"}
Sep 18 14:12:13.529151 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.529023    2572 scope.go:117] "RemoveContainer" containerID="664b6522b7970086ad3a4abaca6886075d57d4c50f4a3056e670e2faaf28433a"
Sep 18 14:12:13.530559 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.530503    2572 generic.go:356] "Generic (PLEG): container finished" podID="049791ec-ffa8-4a2f-a1ea-1f07361a4d1a" containerID="663bacb2a92d024dcc2c3e104ea6bf5e56d5e514795c57950ef833d85cb5052e" exitCode=0
Sep 18 14:12:13.530697 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.530565    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"
Sep 18 14:12:13.530697 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.530627    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96" event={"ID":"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a","Type":"ContainerDied","Data":"663bacb2a92d024dcc2c3e104ea6bf5e56d5e514795c57950ef833d85cb5052e"}
Sep 18 14:12:13.530697 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.530664    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96" event={"ID":"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a","Type":"ContainerDied","Data":"da5e0a5bfc3e1128ed7d0388754cefcc7eb8f28f9c2d1d1df7f5e800ad94b496"}
Sep 18 14:12:13.541122 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.541100    2572 scope.go:117] "RemoveContainer" containerID="664b6522b7970086ad3a4abaca6886075d57d4c50f4a3056e670e2faaf28433a"
Sep 18 14:12:13.541435 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:12:13.541415    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"664b6522b7970086ad3a4abaca6886075d57d4c50f4a3056e670e2faaf28433a\": container with ID starting with 664b6522b7970086ad3a4abaca6886075d57d4c50f4a3056e670e2faaf28433a not found: ID does not exist" containerID="664b6522b7970086ad3a4abaca6886075d57d4c50f4a3056e670e2faaf28433a"
Sep 18 14:12:13.541548 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.541446    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"664b6522b7970086ad3a4abaca6886075d57d4c50f4a3056e670e2faaf28433a"} err="failed to get container status \"664b6522b7970086ad3a4abaca6886075d57d4c50f4a3056e670e2faaf28433a\": rpc error: code = NotFound desc = could not find container \"664b6522b7970086ad3a4abaca6886075d57d4c50f4a3056e670e2faaf28433a\": container with ID starting with 664b6522b7970086ad3a4abaca6886075d57d4c50f4a3056e670e2faaf28433a not found: ID does not exist"
Sep 18 14:12:13.541548 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.541469    2572 scope.go:117] "RemoveContainer" containerID="663bacb2a92d024dcc2c3e104ea6bf5e56d5e514795c57950ef833d85cb5052e"
Sep 18 14:12:13.550056 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.550021    2572 scope.go:117] "RemoveContainer" containerID="663bacb2a92d024dcc2c3e104ea6bf5e56d5e514795c57950ef833d85cb5052e"
Sep 18 14:12:13.550379 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:12:13.550359    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663bacb2a92d024dcc2c3e104ea6bf5e56d5e514795c57950ef833d85cb5052e\": container with ID starting with 663bacb2a92d024dcc2c3e104ea6bf5e56d5e514795c57950ef833d85cb5052e not found: ID does not exist" containerID="663bacb2a92d024dcc2c3e104ea6bf5e56d5e514795c57950ef833d85cb5052e"
Sep 18 14:12:13.550471 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.550388    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663bacb2a92d024dcc2c3e104ea6bf5e56d5e514795c57950ef833d85cb5052e"} err="failed to get container status \"663bacb2a92d024dcc2c3e104ea6bf5e56d5e514795c57950ef833d85cb5052e\": rpc error: code = NotFound desc = could not find container \"663bacb2a92d024dcc2c3e104ea6bf5e56d5e514795c57950ef833d85cb5052e\": container with ID starting with 663bacb2a92d024dcc2c3e104ea6bf5e56d5e514795c57950ef833d85cb5052e not found: ID does not exist"
Sep 18 14:12:13.618849 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.618046    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-model-cache\") pod \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") "
Sep 18 14:12:13.618849 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.618192    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtxvh\" (UniqueName: \"kubernetes.io/projected/34192080-ca2c-431e-b7bb-988e5cf7ff8f-kube-api-access-xtxvh\") pod \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") "
Sep 18 14:12:13.618849 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.618235    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-tls-certs\") pod \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") "
Sep 18 14:12:13.618849 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.618316    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-tmp-dir\") pod \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") "
Sep 18 14:12:13.618849 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.618350    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34192080-ca2c-431e-b7bb-988e5cf7ff8f-tls-certs\") pod \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") "
Sep 18 14:12:13.618849 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.618376    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-model-cache" (OuterVolumeSpecName: "model-cache") pod "34192080-ca2c-431e-b7bb-988e5cf7ff8f" (UID: "34192080-ca2c-431e-b7bb-988e5cf7ff8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:13.618849 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.618402    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-dshm\") pod \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") "
Sep 18 14:12:13.618849 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.618437    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-home\") pod \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") "
Sep 18 14:12:13.618849 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.618473    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-tmp-dir\") pod \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") "
Sep 18 14:12:13.618849 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.618521    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-dshm\") pod \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\" (UID: \"34192080-ca2c-431e-b7bb-988e5cf7ff8f\") "
Sep 18 14:12:13.618849 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.618560    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r2r5\" (UniqueName: \"kubernetes.io/projected/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-kube-api-access-5r2r5\") pod \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") "
Sep 18 14:12:13.618849 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.618633    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34192080-ca2c-431e-b7bb-988e5cf7ff8f" (UID: "34192080-ca2c-431e-b7bb-988e5cf7ff8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:13.618849 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.618652    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-home\") pod \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") "
Sep 18 14:12:13.618849 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.618721    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-model-cache\") pod \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\" (UID: \"049791ec-ffa8-4a2f-a1ea-1f07361a4d1a\") "
Sep 18 14:12:13.619736 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.618936    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "049791ec-ffa8-4a2f-a1ea-1f07361a4d1a" (UID: "049791ec-ffa8-4a2f-a1ea-1f07361a4d1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:13.619736 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.619169    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-model-cache\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:13.619736 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.619191    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-tmp-dir\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:13.619736 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.619205    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-tmp-dir\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:13.619736 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.619213    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-home" (OuterVolumeSpecName: "home") pod "34192080-ca2c-431e-b7bb-988e5cf7ff8f" (UID: "34192080-ca2c-431e-b7bb-988e5cf7ff8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:13.619736 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.619454    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-home" (OuterVolumeSpecName: "home") pod "049791ec-ffa8-4a2f-a1ea-1f07361a4d1a" (UID: "049791ec-ffa8-4a2f-a1ea-1f07361a4d1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:13.619736 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.619652    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-model-cache" (OuterVolumeSpecName: "model-cache") pod "049791ec-ffa8-4a2f-a1ea-1f07361a4d1a" (UID: "049791ec-ffa8-4a2f-a1ea-1f07361a4d1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:13.621870 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.621836    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "049791ec-ffa8-4a2f-a1ea-1f07361a4d1a" (UID: "049791ec-ffa8-4a2f-a1ea-1f07361a4d1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:12:13.622005 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.621931    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-kube-api-access-5r2r5" (OuterVolumeSpecName: "kube-api-access-5r2r5") pod "049791ec-ffa8-4a2f-a1ea-1f07361a4d1a" (UID: "049791ec-ffa8-4a2f-a1ea-1f07361a4d1a"). InnerVolumeSpecName "kube-api-access-5r2r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:12:13.622176 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.622155    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34192080-ca2c-431e-b7bb-988e5cf7ff8f-kube-api-access-xtxvh" (OuterVolumeSpecName: "kube-api-access-xtxvh") pod "34192080-ca2c-431e-b7bb-988e5cf7ff8f" (UID: "34192080-ca2c-431e-b7bb-988e5cf7ff8f"). InnerVolumeSpecName "kube-api-access-xtxvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:12:13.622341 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.622324    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34192080-ca2c-431e-b7bb-988e5cf7ff8f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34192080-ca2c-431e-b7bb-988e5cf7ff8f" (UID: "34192080-ca2c-431e-b7bb-988e5cf7ff8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:12:13.622427 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.622400    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-dshm" (OuterVolumeSpecName: "dshm") pod "34192080-ca2c-431e-b7bb-988e5cf7ff8f" (UID: "34192080-ca2c-431e-b7bb-988e5cf7ff8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:13.623150 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.623125    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-dshm" (OuterVolumeSpecName: "dshm") pod "049791ec-ffa8-4a2f-a1ea-1f07361a4d1a" (UID: "049791ec-ffa8-4a2f-a1ea-1f07361a4d1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:13.722015 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.721966    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-tls-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:13.722015 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.722002    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34192080-ca2c-431e-b7bb-988e5cf7ff8f-tls-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:13.722015 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.722015    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-dshm\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:13.722365 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.722027    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-home\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:13.722365 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.722041    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34192080-ca2c-431e-b7bb-988e5cf7ff8f-dshm\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:13.722365 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.722055    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r2r5\" (UniqueName: \"kubernetes.io/projected/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-kube-api-access-5r2r5\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:13.722365 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.722067    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-home\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:13.722365 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.722079    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a-model-cache\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:13.722365 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.722093    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtxvh\" (UniqueName: \"kubernetes.io/projected/34192080-ca2c-431e-b7bb-988e5cf7ff8f-kube-api-access-xtxvh\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:13.856928 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.856893    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"]
Sep 18 14:12:13.863355 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.863326    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-40970f/distinct-w0-40970f-kserve-7499dd4449-z4q96"]
Sep 18 14:12:13.881868 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.881837    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"]
Sep 18 14:12:13.887074 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:13.887045    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-40970f/premium-w0-40970f-kserve-5f5f8777bb-txmv8"]
Sep 18 14:12:14.198553 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:14.198509    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="049791ec-ffa8-4a2f-a1ea-1f07361a4d1a" path="/var/lib/kubelet/pods/049791ec-ffa8-4a2f-a1ea-1f07361a4d1a/volumes"
Sep 18 14:12:14.199015 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:14.198937    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34192080-ca2c-431e-b7bb-988e5cf7ff8f" path="/var/lib/kubelet/pods/34192080-ca2c-431e-b7bb-988e5cf7ff8f/volumes"
Sep 18 14:12:30.177168 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.177134    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"]
Sep 18 14:12:30.177665 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.177354    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs" podUID="76817456-fb85-4dec-8cca-c0e741359e60" containerName="payload-pre-processing" containerID="cri-o://40a61dc2dfa29545db3ad05aa42084c41c7b2dd08a1aeeb4fc52109aab8e8b21" gracePeriod=30
Sep 18 14:12:30.460014 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.459983    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"
Sep 18 14:12:30.573117 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.573082    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfgcp\" (UniqueName: \"kubernetes.io/projected/76817456-fb85-4dec-8cca-c0e741359e60-kube-api-access-nfgcp\") pod \"76817456-fb85-4dec-8cca-c0e741359e60\" (UID: \"76817456-fb85-4dec-8cca-c0e741359e60\") "
Sep 18 14:12:30.573301 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.573214    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76817456-fb85-4dec-8cca-c0e741359e60-plugins-config-volume\") pod \"76817456-fb85-4dec-8cca-c0e741359e60\" (UID: \"76817456-fb85-4dec-8cca-c0e741359e60\") "
Sep 18 14:12:30.573573 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.573528    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76817456-fb85-4dec-8cca-c0e741359e60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "76817456-fb85-4dec-8cca-c0e741359e60" (UID: "76817456-fb85-4dec-8cca-c0e741359e60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:12:30.575161 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.575135    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76817456-fb85-4dec-8cca-c0e741359e60-kube-api-access-nfgcp" (OuterVolumeSpecName: "kube-api-access-nfgcp") pod "76817456-fb85-4dec-8cca-c0e741359e60" (UID: "76817456-fb85-4dec-8cca-c0e741359e60"). InnerVolumeSpecName "kube-api-access-nfgcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:12:30.646250 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.646211    2572 generic.go:356] "Generic (PLEG): container finished" podID="76817456-fb85-4dec-8cca-c0e741359e60" containerID="40a61dc2dfa29545db3ad05aa42084c41c7b2dd08a1aeeb4fc52109aab8e8b21" exitCode=0
Sep 18 14:12:30.646425 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.646253    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"
Sep 18 14:12:30.646425 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.646294    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs" event={"ID":"76817456-fb85-4dec-8cca-c0e741359e60","Type":"ContainerDied","Data":"40a61dc2dfa29545db3ad05aa42084c41c7b2dd08a1aeeb4fc52109aab8e8b21"}
Sep 18 14:12:30.646425 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.646351    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs" event={"ID":"76817456-fb85-4dec-8cca-c0e741359e60","Type":"ContainerDied","Data":"3bf0eb62e3eb0a4d799708cacc441fdfc8950abe2145471efd4016f8764da7a8"}
Sep 18 14:12:30.646425 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.646366    2572 scope.go:117] "RemoveContainer" containerID="40a61dc2dfa29545db3ad05aa42084c41c7b2dd08a1aeeb4fc52109aab8e8b21"
Sep 18 14:12:30.655770 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.655749    2572 scope.go:117] "RemoveContainer" containerID="40a61dc2dfa29545db3ad05aa42084c41c7b2dd08a1aeeb4fc52109aab8e8b21"
Sep 18 14:12:30.656039 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:12:30.656020    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40a61dc2dfa29545db3ad05aa42084c41c7b2dd08a1aeeb4fc52109aab8e8b21\": container with ID starting with 40a61dc2dfa29545db3ad05aa42084c41c7b2dd08a1aeeb4fc52109aab8e8b21 not found: ID does not exist" containerID="40a61dc2dfa29545db3ad05aa42084c41c7b2dd08a1aeeb4fc52109aab8e8b21"
Sep 18 14:12:30.656109 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.656051    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a61dc2dfa29545db3ad05aa42084c41c7b2dd08a1aeeb4fc52109aab8e8b21"} err="failed to get container status \"40a61dc2dfa29545db3ad05aa42084c41c7b2dd08a1aeeb4fc52109aab8e8b21\": rpc error: code = NotFound desc = could not find container \"40a61dc2dfa29545db3ad05aa42084c41c7b2dd08a1aeeb4fc52109aab8e8b21\": container with ID starting with 40a61dc2dfa29545db3ad05aa42084c41c7b2dd08a1aeeb4fc52109aab8e8b21 not found: ID does not exist"
Sep 18 14:12:30.670370 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.670342    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"]
Sep 18 14:12:30.672935 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.672915    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c1586-67f84785fd-rwtzs"]
Sep 18 14:12:30.673927 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.673909    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76817456-fb85-4dec-8cca-c0e741359e60-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:30.673979 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:30.673932    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfgcp\" (UniqueName: \"kubernetes.io/projected/76817456-fb85-4dec-8cca-c0e741359e60-kube-api-access-nfgcp\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:32.198264 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:32.198230    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76817456-fb85-4dec-8cca-c0e741359e60" path="/var/lib/kubelet/pods/76817456-fb85-4dec-8cca-c0e741359e60/volumes"
Sep 18 14:12:50.424311 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.424227    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"]
Sep 18 14:12:50.424813 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.424431    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz" podUID="214b90fb-cc46-4298-99f3-8c81dfb90e80" containerName="maas-api" containerID="cri-o://93e797444529a02d0c4baec57a68ae4c64116ef8db3683cae247e20029c7dfd0" gracePeriod=30
Sep 18 14:12:50.490101 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.490066    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"]
Sep 18 14:12:50.490355 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.490313    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb" podUID="1ea32efc-eb86-433a-b6fb-ce0dba10de6d" containerName="payload-processing" containerID="cri-o://8dbb0f2894e0eb8584bb077375b07fb594a7d1ff6f6f8df20ca4950c3160b26d" gracePeriod=30
Sep 18 14:12:50.500393 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.500162    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"]
Sep 18 14:12:50.500659 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.500473    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g" podUID="8e854df5-adb6-414e-a465-43bc706df627" containerName="payload-pre-processing" containerID="cri-o://afb02be9c1ea48a35fdb0476489ae80d08e9e48dcc84d5407fca05649d120d53" gracePeriod=30
Sep 18 14:12:50.774915 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.774881    2572 generic.go:356] "Generic (PLEG): container finished" podID="8e854df5-adb6-414e-a465-43bc706df627" containerID="afb02be9c1ea48a35fdb0476489ae80d08e9e48dcc84d5407fca05649d120d53" exitCode=0
Sep 18 14:12:50.775089 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.774948    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g" event={"ID":"8e854df5-adb6-414e-a465-43bc706df627","Type":"ContainerDied","Data":"afb02be9c1ea48a35fdb0476489ae80d08e9e48dcc84d5407fca05649d120d53"}
Sep 18 14:12:50.777163 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.777133    2572 generic.go:356] "Generic (PLEG): container finished" podID="1ea32efc-eb86-433a-b6fb-ce0dba10de6d" containerID="8dbb0f2894e0eb8584bb077375b07fb594a7d1ff6f6f8df20ca4950c3160b26d" exitCode=0
Sep 18 14:12:50.777295 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.777223    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb" event={"ID":"1ea32efc-eb86-433a-b6fb-ce0dba10de6d","Type":"ContainerDied","Data":"8dbb0f2894e0eb8584bb077375b07fb594a7d1ff6f6f8df20ca4950c3160b26d"}
Sep 18 14:12:50.778897 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.778875    2572 generic.go:356] "Generic (PLEG): container finished" podID="214b90fb-cc46-4298-99f3-8c81dfb90e80" containerID="93e797444529a02d0c4baec57a68ae4c64116ef8db3683cae247e20029c7dfd0" exitCode=0
Sep 18 14:12:50.779022 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.778953    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz" event={"ID":"214b90fb-cc46-4298-99f3-8c81dfb90e80","Type":"ContainerDied","Data":"93e797444529a02d0c4baec57a68ae4c64116ef8db3683cae247e20029c7dfd0"}
Sep 18 14:12:50.779022 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.779007    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz" event={"ID":"214b90fb-cc46-4298-99f3-8c81dfb90e80","Type":"ContainerDied","Data":"6eff8f3eee9085a2b412d9dcb34c6d3a2de8bc075ade012bccb4f2623a4637e6"}
Sep 18 14:12:50.779137 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.779031    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eff8f3eee9085a2b412d9dcb34c6d3a2de8bc075ade012bccb4f2623a4637e6"
Sep 18 14:12:50.779228 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.779212    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:12:50.832998 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.832971    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"
Sep 18 14:12:50.836666 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.836647    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"
Sep 18 14:12:50.870623 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.870593    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hq5rn\" (UniqueName: \"kubernetes.io/projected/214b90fb-cc46-4298-99f3-8c81dfb90e80-kube-api-access-hq5rn\") pod \"214b90fb-cc46-4298-99f3-8c81dfb90e80\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") "
Sep 18 14:12:50.870823 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.870638    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/214b90fb-cc46-4298-99f3-8c81dfb90e80-openshift-service-ca\") pod \"214b90fb-cc46-4298-99f3-8c81dfb90e80\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") "
Sep 18 14:12:50.870823 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.870657    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214b90fb-cc46-4298-99f3-8c81dfb90e80-metrics-certs\") pod \"214b90fb-cc46-4298-99f3-8c81dfb90e80\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") "
Sep 18 14:12:50.870823 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.870682    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/214b90fb-cc46-4298-99f3-8c81dfb90e80-maas-api-tls\") pod \"214b90fb-cc46-4298-99f3-8c81dfb90e80\" (UID: \"214b90fb-cc46-4298-99f3-8c81dfb90e80\") "
Sep 18 14:12:50.871404 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.871372    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/214b90fb-cc46-4298-99f3-8c81dfb90e80-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "214b90fb-cc46-4298-99f3-8c81dfb90e80" (UID: "214b90fb-cc46-4298-99f3-8c81dfb90e80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:12:50.873610 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.873563    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/214b90fb-cc46-4298-99f3-8c81dfb90e80-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "214b90fb-cc46-4298-99f3-8c81dfb90e80" (UID: "214b90fb-cc46-4298-99f3-8c81dfb90e80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:12:50.873935 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.873870    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/214b90fb-cc46-4298-99f3-8c81dfb90e80-kube-api-access-hq5rn" (OuterVolumeSpecName: "kube-api-access-hq5rn") pod "214b90fb-cc46-4298-99f3-8c81dfb90e80" (UID: "214b90fb-cc46-4298-99f3-8c81dfb90e80"). InnerVolumeSpecName "kube-api-access-hq5rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:12:50.874813 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.874770    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/214b90fb-cc46-4298-99f3-8c81dfb90e80-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "214b90fb-cc46-4298-99f3-8c81dfb90e80" (UID: "214b90fb-cc46-4298-99f3-8c81dfb90e80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:12:50.971522 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.971423    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7wrf\" (UniqueName: \"kubernetes.io/projected/8e854df5-adb6-414e-a465-43bc706df627-kube-api-access-w7wrf\") pod \"8e854df5-adb6-414e-a465-43bc706df627\" (UID: \"8e854df5-adb6-414e-a465-43bc706df627\") "
Sep 18 14:12:50.971522 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.971511    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea32efc-eb86-433a-b6fb-ce0dba10de6d-plugins-config-volume\") pod \"1ea32efc-eb86-433a-b6fb-ce0dba10de6d\" (UID: \"1ea32efc-eb86-433a-b6fb-ce0dba10de6d\") "
Sep 18 14:12:50.971762 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.971556    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjsn5\" (UniqueName: \"kubernetes.io/projected/1ea32efc-eb86-433a-b6fb-ce0dba10de6d-kube-api-access-qjsn5\") pod \"1ea32efc-eb86-433a-b6fb-ce0dba10de6d\" (UID: \"1ea32efc-eb86-433a-b6fb-ce0dba10de6d\") "
Sep 18 14:12:50.971762 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.971608    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e854df5-adb6-414e-a465-43bc706df627-plugins-config-volume\") pod \"8e854df5-adb6-414e-a465-43bc706df627\" (UID: \"8e854df5-adb6-414e-a465-43bc706df627\") "
Sep 18 14:12:50.971878 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.971844    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq5rn\" (UniqueName: \"kubernetes.io/projected/214b90fb-cc46-4298-99f3-8c81dfb90e80-kube-api-access-hq5rn\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:50.971878 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.971847    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea32efc-eb86-433a-b6fb-ce0dba10de6d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ea32efc-eb86-433a-b6fb-ce0dba10de6d" (UID: "1ea32efc-eb86-433a-b6fb-ce0dba10de6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:12:50.971878 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.971866    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/214b90fb-cc46-4298-99f3-8c81dfb90e80-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:50.972179 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.971881    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214b90fb-cc46-4298-99f3-8c81dfb90e80-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:50.972179 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.971894    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/214b90fb-cc46-4298-99f3-8c81dfb90e80-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:50.972179 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.972135    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e854df5-adb6-414e-a465-43bc706df627-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e854df5-adb6-414e-a465-43bc706df627" (UID: "8e854df5-adb6-414e-a465-43bc706df627"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:12:50.973610 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.973590    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea32efc-eb86-433a-b6fb-ce0dba10de6d-kube-api-access-qjsn5" (OuterVolumeSpecName: "kube-api-access-qjsn5") pod "1ea32efc-eb86-433a-b6fb-ce0dba10de6d" (UID: "1ea32efc-eb86-433a-b6fb-ce0dba10de6d"). InnerVolumeSpecName "kube-api-access-qjsn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:12:50.973681 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:50.973612    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e854df5-adb6-414e-a465-43bc706df627-kube-api-access-w7wrf" (OuterVolumeSpecName: "kube-api-access-w7wrf") pod "8e854df5-adb6-414e-a465-43bc706df627" (UID: "8e854df5-adb6-414e-a465-43bc706df627"). InnerVolumeSpecName "kube-api-access-w7wrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:12:51.072972 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.072932    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7wrf\" (UniqueName: \"kubernetes.io/projected/8e854df5-adb6-414e-a465-43bc706df627-kube-api-access-w7wrf\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:51.072972 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.072964    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea32efc-eb86-433a-b6fb-ce0dba10de6d-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:51.072972 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.072973    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjsn5\" (UniqueName: \"kubernetes.io/projected/1ea32efc-eb86-433a-b6fb-ce0dba10de6d-kube-api-access-qjsn5\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:51.072972 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.072982    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e854df5-adb6-414e-a465-43bc706df627-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:12:51.787324 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.787291    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"
Sep 18 14:12:51.787840 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.787286    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g" event={"ID":"8e854df5-adb6-414e-a465-43bc706df627","Type":"ContainerDied","Data":"89a5afe73332da4e6a57a89323920295f0b5822a53bec400b2fc5e6f82e21763"}
Sep 18 14:12:51.787840 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.787437    2572 scope.go:117] "RemoveContainer" containerID="afb02be9c1ea48a35fdb0476489ae80d08e9e48dcc84d5407fca05649d120d53"
Sep 18 14:12:51.788939 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.788851    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"
Sep 18 14:12:51.788939 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.788885    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb" event={"ID":"1ea32efc-eb86-433a-b6fb-ce0dba10de6d","Type":"ContainerDied","Data":"72c85f7a4926412262e84e4380c286785d33edcf034759725eb3263fe58f2ec9"}
Sep 18 14:12:51.789074 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.789015    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"
Sep 18 14:12:51.799422 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.799406    2572 scope.go:117] "RemoveContainer" containerID="8dbb0f2894e0eb8584bb077375b07fb594a7d1ff6f6f8df20ca4950c3160b26d"
Sep 18 14:12:51.821092 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.821060    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"]
Sep 18 14:12:51.823882 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.823862    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4e72b-659966b86-9v45g"]
Sep 18 14:12:51.834341 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.834313    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"]
Sep 18 14:12:51.839363 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.839340    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d4e72b-647dc77d7b-g2mgz"]
Sep 18 14:12:51.867443 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.867407    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"]
Sep 18 14:12:51.887798 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:51.887774    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d4e72b-5f7f696b69-cv5zb"]
Sep 18 14:12:52.197379 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:52.197349    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ea32efc-eb86-433a-b6fb-ce0dba10de6d" path="/var/lib/kubelet/pods/1ea32efc-eb86-433a-b6fb-ce0dba10de6d/volumes"
Sep 18 14:12:52.197757 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:52.197741    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="214b90fb-cc46-4298-99f3-8c81dfb90e80" path="/var/lib/kubelet/pods/214b90fb-cc46-4298-99f3-8c81dfb90e80/volumes"
Sep 18 14:12:52.198067 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:12:52.198055    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e854df5-adb6-414e-a465-43bc706df627" path="/var/lib/kubelet/pods/8e854df5-adb6-414e-a465-43bc706df627/volumes"
Sep 18 14:13:08.006877 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.006840    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"]
Sep 18 14:13:08.007717 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.007659    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4" podUID="7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186" containerName="maas-api" containerID="cri-o://58d6f2a241fde94f96724e0297d808ce593daadc55ec5f6448ea9d1214440b60" gracePeriod=30
Sep 18 14:13:08.071823 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.071787    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"]
Sep 18 14:13:08.072080 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.072051    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz" podUID="24e558a8-4d01-44cd-aeb0-9e98d2879e95" containerName="payload-processing" containerID="cri-o://d112cc1bad1ca64156d2110d3b23ec585a1eecb5815daebf8810f332cf53b7bf" gracePeriod=30
Sep 18 14:13:08.343600 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.343572    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:13:08.371561 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.371515    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"
Sep 18 14:13:08.541111 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.541026    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnlpw\" (UniqueName: \"kubernetes.io/projected/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-kube-api-access-pnlpw\") pod \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") "
Sep 18 14:13:08.541111 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.541083    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-maas-api-tls\") pod \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") "
Sep 18 14:13:08.541349 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.541121    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e558a8-4d01-44cd-aeb0-9e98d2879e95-plugins-config-volume\") pod \"24e558a8-4d01-44cd-aeb0-9e98d2879e95\" (UID: \"24e558a8-4d01-44cd-aeb0-9e98d2879e95\") "
Sep 18 14:13:08.541349 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.541175    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-openshift-service-ca\") pod \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") "
Sep 18 14:13:08.541349 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.541217    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-metrics-certs\") pod \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\" (UID: \"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186\") "
Sep 18 14:13:08.541349 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.541303    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w82wr\" (UniqueName: \"kubernetes.io/projected/24e558a8-4d01-44cd-aeb0-9e98d2879e95-kube-api-access-w82wr\") pod \"24e558a8-4d01-44cd-aeb0-9e98d2879e95\" (UID: \"24e558a8-4d01-44cd-aeb0-9e98d2879e95\") "
Sep 18 14:13:08.541669 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.541488    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e558a8-4d01-44cd-aeb0-9e98d2879e95-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24e558a8-4d01-44cd-aeb0-9e98d2879e95" (UID: "24e558a8-4d01-44cd-aeb0-9e98d2879e95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:13:08.541669 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.541599    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186" (UID: "7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:13:08.541669 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.541646    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e558a8-4d01-44cd-aeb0-9e98d2879e95-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:13:08.543471 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.543436    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186" (UID: "7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:13:08.543619 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.543583    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e558a8-4d01-44cd-aeb0-9e98d2879e95-kube-api-access-w82wr" (OuterVolumeSpecName: "kube-api-access-w82wr") pod "24e558a8-4d01-44cd-aeb0-9e98d2879e95" (UID: "24e558a8-4d01-44cd-aeb0-9e98d2879e95"). InnerVolumeSpecName "kube-api-access-w82wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:13:08.543736 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.543719    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-kube-api-access-pnlpw" (OuterVolumeSpecName: "kube-api-access-pnlpw") pod "7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186" (UID: "7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186"). InnerVolumeSpecName "kube-api-access-pnlpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:13:08.543798 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.543760    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186" (UID: "7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:13:08.642249 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.642209    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w82wr\" (UniqueName: \"kubernetes.io/projected/24e558a8-4d01-44cd-aeb0-9e98d2879e95-kube-api-access-w82wr\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:13:08.642249 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.642241    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnlpw\" (UniqueName: \"kubernetes.io/projected/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-kube-api-access-pnlpw\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:13:08.642249 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.642252    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:13:08.642490 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.642263    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:13:08.642490 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.642272    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:13:08.890645 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.890565    2572 generic.go:356] "Generic (PLEG): container finished" podID="24e558a8-4d01-44cd-aeb0-9e98d2879e95" containerID="d112cc1bad1ca64156d2110d3b23ec585a1eecb5815daebf8810f332cf53b7bf" exitCode=0
Sep 18 14:13:08.890810 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.890665    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"
Sep 18 14:13:08.890810 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.890713    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz" event={"ID":"24e558a8-4d01-44cd-aeb0-9e98d2879e95","Type":"ContainerDied","Data":"d112cc1bad1ca64156d2110d3b23ec585a1eecb5815daebf8810f332cf53b7bf"}
Sep 18 14:13:08.890810 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.890768    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz" event={"ID":"24e558a8-4d01-44cd-aeb0-9e98d2879e95","Type":"ContainerDied","Data":"5575b4e7e6ea33f32f6c4a20c7e8422f7e559c66f736f1e11a0ecd9f1c6689d9"}
Sep 18 14:13:08.890810 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.890792    2572 scope.go:117] "RemoveContainer" containerID="d112cc1bad1ca64156d2110d3b23ec585a1eecb5815daebf8810f332cf53b7bf"
Sep 18 14:13:08.892206 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.892174    2572 generic.go:356] "Generic (PLEG): container finished" podID="7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186" containerID="58d6f2a241fde94f96724e0297d808ce593daadc55ec5f6448ea9d1214440b60" exitCode=0
Sep 18 14:13:08.892334 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.892201    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4" event={"ID":"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186","Type":"ContainerDied","Data":"58d6f2a241fde94f96724e0297d808ce593daadc55ec5f6448ea9d1214440b60"}
Sep 18 14:13:08.892334 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.892229    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"
Sep 18 14:13:08.892334 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.892251    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4" event={"ID":"7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186","Type":"ContainerDied","Data":"33077757b49263104fa2b70f6119dff61e8009957a85aada7fe9a53774919534"}
Sep 18 14:13:08.899805 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.899789    2572 scope.go:117] "RemoveContainer" containerID="d112cc1bad1ca64156d2110d3b23ec585a1eecb5815daebf8810f332cf53b7bf"
Sep 18 14:13:08.900120 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:13:08.900092    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d112cc1bad1ca64156d2110d3b23ec585a1eecb5815daebf8810f332cf53b7bf\": container with ID starting with d112cc1bad1ca64156d2110d3b23ec585a1eecb5815daebf8810f332cf53b7bf not found: ID does not exist" containerID="d112cc1bad1ca64156d2110d3b23ec585a1eecb5815daebf8810f332cf53b7bf"
Sep 18 14:13:08.900220 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.900121    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d112cc1bad1ca64156d2110d3b23ec585a1eecb5815daebf8810f332cf53b7bf"} err="failed to get container status \"d112cc1bad1ca64156d2110d3b23ec585a1eecb5815daebf8810f332cf53b7bf\": rpc error: code = NotFound desc = could not find container \"d112cc1bad1ca64156d2110d3b23ec585a1eecb5815daebf8810f332cf53b7bf\": container with ID starting with d112cc1bad1ca64156d2110d3b23ec585a1eecb5815daebf8810f332cf53b7bf not found: ID does not exist"
Sep 18 14:13:08.900220 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.900137    2572 scope.go:117] "RemoveContainer" containerID="58d6f2a241fde94f96724e0297d808ce593daadc55ec5f6448ea9d1214440b60"
Sep 18 14:13:08.908685 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.908669    2572 scope.go:117] "RemoveContainer" containerID="58d6f2a241fde94f96724e0297d808ce593daadc55ec5f6448ea9d1214440b60"
Sep 18 14:13:08.908966 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:13:08.908940    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d6f2a241fde94f96724e0297d808ce593daadc55ec5f6448ea9d1214440b60\": container with ID starting with 58d6f2a241fde94f96724e0297d808ce593daadc55ec5f6448ea9d1214440b60 not found: ID does not exist" containerID="58d6f2a241fde94f96724e0297d808ce593daadc55ec5f6448ea9d1214440b60"
Sep 18 14:13:08.909040 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.908971    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d6f2a241fde94f96724e0297d808ce593daadc55ec5f6448ea9d1214440b60"} err="failed to get container status \"58d6f2a241fde94f96724e0297d808ce593daadc55ec5f6448ea9d1214440b60\": rpc error: code = NotFound desc = could not find container \"58d6f2a241fde94f96724e0297d808ce593daadc55ec5f6448ea9d1214440b60\": container with ID starting with 58d6f2a241fde94f96724e0297d808ce593daadc55ec5f6448ea9d1214440b60 not found: ID does not exist"
Sep 18 14:13:08.918950 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.918919    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"]
Sep 18 14:13:08.925310 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.925280    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-75dcf1-68899cd675-nhnw4"]
Sep 18 14:13:08.933737 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.933714    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"]
Sep 18 14:13:08.941231 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:08.941212    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-75dcf1-6bb79df99f-n8qxz"]
Sep 18 14:13:10.197413 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:10.197370    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24e558a8-4d01-44cd-aeb0-9e98d2879e95" path="/var/lib/kubelet/pods/24e558a8-4d01-44cd-aeb0-9e98d2879e95/volumes"
Sep 18 14:13:10.197991 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:10.197970    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186" path="/var/lib/kubelet/pods/7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186/volumes"
Sep 18 14:13:26.482014 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:13:26.481982    2572 scope.go:117] "RemoveContainer" containerID="93e797444529a02d0c4baec57a68ae4c64116ef8db3683cae247e20029c7dfd0"
Sep 18 14:14:12.497106 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.497022    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"]
Sep 18 14:14:12.498123 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.497277    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln" podUID="9ce59ddc-1186-404f-9e2e-d4d3ce615a83" containerName="maas-api" containerID="cri-o://8c5b156d635d8595ea31c664ac3628abb15e09e54fb8f52ee34f280fb6b46469" gracePeriod=30
Sep 18 14:14:12.548631 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.548596    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"]
Sep 18 14:14:12.548872 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.548847    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2" podUID="0135fa54-68df-4431-abfe-74266565f05e" containerName="payload-processing" containerID="cri-o://bfae8d760d981b5f20276917dd029f998f10780c96032bbb74230e3b48352888" gracePeriod=30
Sep 18 14:14:12.822643 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.822610    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 14:14:12.858815 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.858786    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"
Sep 18 14:14:12.926368 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.926326    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drrkh\" (UniqueName: \"kubernetes.io/projected/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-kube-api-access-drrkh\") pod \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") "
Sep 18 14:14:12.926593 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.926442    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-metrics-certs\") pod \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") "
Sep 18 14:14:12.926593 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.926466    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-maas-api-tls\") pod \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") "
Sep 18 14:14:12.926593 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.926503    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0135fa54-68df-4431-abfe-74266565f05e-plugins-config-volume\") pod \"0135fa54-68df-4431-abfe-74266565f05e\" (UID: \"0135fa54-68df-4431-abfe-74266565f05e\") "
Sep 18 14:14:12.926593 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.926547    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-openshift-service-ca\") pod \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\" (UID: \"9ce59ddc-1186-404f-9e2e-d4d3ce615a83\") "
Sep 18 14:14:12.926824 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.926596    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drbjl\" (UniqueName: \"kubernetes.io/projected/0135fa54-68df-4431-abfe-74266565f05e-kube-api-access-drbjl\") pod \"0135fa54-68df-4431-abfe-74266565f05e\" (UID: \"0135fa54-68df-4431-abfe-74266565f05e\") "
Sep 18 14:14:12.927497 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.927463    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0135fa54-68df-4431-abfe-74266565f05e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0135fa54-68df-4431-abfe-74266565f05e" (UID: "0135fa54-68df-4431-abfe-74266565f05e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:14:12.927852 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.927817    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ce59ddc-1186-404f-9e2e-d4d3ce615a83" (UID: "9ce59ddc-1186-404f-9e2e-d4d3ce615a83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:14:12.928860 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.928838    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ce59ddc-1186-404f-9e2e-d4d3ce615a83" (UID: "9ce59ddc-1186-404f-9e2e-d4d3ce615a83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:14:12.929053 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.929024    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-kube-api-access-drrkh" (OuterVolumeSpecName: "kube-api-access-drrkh") pod "9ce59ddc-1186-404f-9e2e-d4d3ce615a83" (UID: "9ce59ddc-1186-404f-9e2e-d4d3ce615a83"). InnerVolumeSpecName "kube-api-access-drrkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:14:12.929160 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.929090    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0135fa54-68df-4431-abfe-74266565f05e-kube-api-access-drbjl" (OuterVolumeSpecName: "kube-api-access-drbjl") pod "0135fa54-68df-4431-abfe-74266565f05e" (UID: "0135fa54-68df-4431-abfe-74266565f05e"). InnerVolumeSpecName "kube-api-access-drbjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:14:12.929553 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:12.929502    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9ce59ddc-1186-404f-9e2e-d4d3ce615a83" (UID: "9ce59ddc-1186-404f-9e2e-d4d3ce615a83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:14:13.027986 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.027906    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:14:13.027986 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.027937    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:14:13.027986 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.027947    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0135fa54-68df-4431-abfe-74266565f05e-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:14:13.027986 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.027957    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:14:13.027986 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.027966    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drbjl\" (UniqueName: \"kubernetes.io/projected/0135fa54-68df-4431-abfe-74266565f05e-kube-api-access-drbjl\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:14:13.027986 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.027974    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drrkh\" (UniqueName: \"kubernetes.io/projected/9ce59ddc-1186-404f-9e2e-d4d3ce615a83-kube-api-access-drrkh\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:14:13.260202 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.260162    2572 generic.go:356] "Generic (PLEG): container finished" podID="0135fa54-68df-4431-abfe-74266565f05e" containerID="bfae8d760d981b5f20276917dd029f998f10780c96032bbb74230e3b48352888" exitCode=0
Sep 18 14:14:13.260406 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.260213    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"
Sep 18 14:14:13.260406 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.260254    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2" event={"ID":"0135fa54-68df-4431-abfe-74266565f05e","Type":"ContainerDied","Data":"bfae8d760d981b5f20276917dd029f998f10780c96032bbb74230e3b48352888"}
Sep 18 14:14:13.260406 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.260318    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2" event={"ID":"0135fa54-68df-4431-abfe-74266565f05e","Type":"ContainerDied","Data":"b36551d9f08ebb58c03d51ecc9606a8e05cca5ad36b41d18a05f4efa22c7ae24"}
Sep 18 14:14:13.260406 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.260350    2572 scope.go:117] "RemoveContainer" containerID="bfae8d760d981b5f20276917dd029f998f10780c96032bbb74230e3b48352888"
Sep 18 14:14:13.261971 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.261935    2572 generic.go:356] "Generic (PLEG): container finished" podID="9ce59ddc-1186-404f-9e2e-d4d3ce615a83" containerID="8c5b156d635d8595ea31c664ac3628abb15e09e54fb8f52ee34f280fb6b46469" exitCode=0
Sep 18 14:14:13.262109 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.261982    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"
Sep 18 14:14:13.262109 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.261986    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln" event={"ID":"9ce59ddc-1186-404f-9e2e-d4d3ce615a83","Type":"ContainerDied","Data":"8c5b156d635d8595ea31c664ac3628abb15e09e54fb8f52ee34f280fb6b46469"}
Sep 18 14:14:13.262109 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.262047    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln" event={"ID":"9ce59ddc-1186-404f-9e2e-d4d3ce615a83","Type":"ContainerDied","Data":"84276a8a967f8009f4969e8d6ed72111f8becbdd426cf4bd077d9a3792520615"}
Sep 18 14:14:13.272406 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.272386    2572 scope.go:117] "RemoveContainer" containerID="bfae8d760d981b5f20276917dd029f998f10780c96032bbb74230e3b48352888"
Sep 18 14:14:13.272822 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:14:13.272798    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfae8d760d981b5f20276917dd029f998f10780c96032bbb74230e3b48352888\": container with ID starting with bfae8d760d981b5f20276917dd029f998f10780c96032bbb74230e3b48352888 not found: ID does not exist" containerID="bfae8d760d981b5f20276917dd029f998f10780c96032bbb74230e3b48352888"
Sep 18 14:14:13.272981 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.272830    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfae8d760d981b5f20276917dd029f998f10780c96032bbb74230e3b48352888"} err="failed to get container status \"bfae8d760d981b5f20276917dd029f998f10780c96032bbb74230e3b48352888\": rpc error: code = NotFound desc = could not find container \"bfae8d760d981b5f20276917dd029f998f10780c96032bbb74230e3b48352888\": container with ID starting with bfae8d760d981b5f20276917dd029f998f10780c96032bbb74230e3b48352888 not found: ID does not exist"
Sep 18 14:14:13.272981 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.272853    2572 scope.go:117] "RemoveContainer" containerID="8c5b156d635d8595ea31c664ac3628abb15e09e54fb8f52ee34f280fb6b46469"
Sep 18 14:14:13.281715 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.281699    2572 scope.go:117] "RemoveContainer" containerID="8c5b156d635d8595ea31c664ac3628abb15e09e54fb8f52ee34f280fb6b46469"
Sep 18 14:14:13.282048 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:14:13.282027    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5b156d635d8595ea31c664ac3628abb15e09e54fb8f52ee34f280fb6b46469\": container with ID starting with 8c5b156d635d8595ea31c664ac3628abb15e09e54fb8f52ee34f280fb6b46469 not found: ID does not exist" containerID="8c5b156d635d8595ea31c664ac3628abb15e09e54fb8f52ee34f280fb6b46469"
Sep 18 14:14:13.282119 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.282052    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5b156d635d8595ea31c664ac3628abb15e09e54fb8f52ee34f280fb6b46469"} err="failed to get container status \"8c5b156d635d8595ea31c664ac3628abb15e09e54fb8f52ee34f280fb6b46469\": rpc error: code = NotFound desc = could not find container \"8c5b156d635d8595ea31c664ac3628abb15e09e54fb8f52ee34f280fb6b46469\": container with ID starting with 8c5b156d635d8595ea31c664ac3628abb15e09e54fb8f52ee34f280fb6b46469 not found: ID does not exist"
Sep 18 14:14:13.288065 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.287824    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"]
Sep 18 14:14:13.292680 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.292659    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a02b2c-ddd696947-m7qp2"]
Sep 18 14:14:13.303488 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.303467    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"]
Sep 18 14:14:13.306936 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:13.306917    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a02b2c-99bf98f67-8smln"]
Sep 18 14:14:14.197804 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:14.197768    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0135fa54-68df-4431-abfe-74266565f05e" path="/var/lib/kubelet/pods/0135fa54-68df-4431-abfe-74266565f05e/volumes"
Sep 18 14:14:14.198187 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:14.198169    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ce59ddc-1186-404f-9e2e-d4d3ce615a83" path="/var/lib/kubelet/pods/9ce59ddc-1186-404f-9e2e-d4d3ce615a83/volumes"
Sep 18 14:14:26.326251 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:26.326220    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 14:14:26.333037 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:14:26.333001    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 14:15:41.279993 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.279105    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"]
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.280823    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="049791ec-ffa8-4a2f-a1ea-1f07361a4d1a" containerName="main"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.280846    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="049791ec-ffa8-4a2f-a1ea-1f07361a4d1a" containerName="main"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.280865    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186" containerName="maas-api"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.280875    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186" containerName="maas-api"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.280898    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c769edea-c5d3-4901-908c-c2c9faf72504" containerName="maas-api"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.280907    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c769edea-c5d3-4901-908c-c2c9faf72504" containerName="maas-api"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.280927    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0135fa54-68df-4431-abfe-74266565f05e" containerName="payload-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.280946    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0135fa54-68df-4431-abfe-74266565f05e" containerName="payload-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.280969    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ce59ddc-1186-404f-9e2e-d4d3ce615a83" containerName="maas-api"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.280977    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ce59ddc-1186-404f-9e2e-d4d3ce615a83" containerName="maas-api"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.280995    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34192080-ca2c-431e-b7bb-988e5cf7ff8f" containerName="main"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281004    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="34192080-ca2c-431e-b7bb-988e5cf7ff8f" containerName="main"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281014    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e854df5-adb6-414e-a465-43bc706df627" containerName="payload-pre-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281023    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e854df5-adb6-414e-a465-43bc706df627" containerName="payload-pre-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281038    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="214b90fb-cc46-4298-99f3-8c81dfb90e80" containerName="maas-api"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281047    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="214b90fb-cc46-4298-99f3-8c81dfb90e80" containerName="maas-api"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281063    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e558a8-4d01-44cd-aeb0-9e98d2879e95" containerName="payload-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281077    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e558a8-4d01-44cd-aeb0-9e98d2879e95" containerName="payload-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281093    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d16b854d-505d-4355-8fd8-5b5d27741794" containerName="payload-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281101    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d16b854d-505d-4355-8fd8-5b5d27741794" containerName="payload-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281115    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ea32efc-eb86-433a-b6fb-ce0dba10de6d" containerName="payload-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281124    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ea32efc-eb86-433a-b6fb-ce0dba10de6d" containerName="payload-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281139    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f" containerName="maas-api"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281147    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f" containerName="maas-api"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281156    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76817456-fb85-4dec-8cca-c0e741359e60" containerName="payload-pre-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281165    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="76817456-fb85-4dec-8cca-c0e741359e60" containerName="payload-pre-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281380    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="24e558a8-4d01-44cd-aeb0-9e98d2879e95" containerName="payload-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281394    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c769edea-c5d3-4901-908c-c2c9faf72504" containerName="maas-api"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281402    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="214b90fb-cc46-4298-99f3-8c81dfb90e80" containerName="maas-api"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281418    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d16b854d-505d-4355-8fd8-5b5d27741794" containerName="payload-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281444    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e854df5-adb6-414e-a465-43bc706df627" containerName="payload-pre-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281461    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="049791ec-ffa8-4a2f-a1ea-1f07361a4d1a" containerName="main"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281479    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ce59ddc-1186-404f-9e2e-d4d3ce615a83" containerName="maas-api"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281490    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0135fa54-68df-4431-abfe-74266565f05e" containerName="payload-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281505    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="76817456-fb85-4dec-8cca-c0e741359e60" containerName="payload-pre-processing"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281520    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="34192080-ca2c-431e-b7bb-988e5cf7ff8f" containerName="main"
Sep 18 14:15:41.283587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281552    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc27cda-c5ee-4c6c-a154-ea6b8fe5a186" containerName="maas-api"
Sep 18 14:15:41.285324 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281570    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ea32efc-eb86-433a-b6fb-ce0dba10de6d" containerName="payload-processing"
Sep 18 14:15:41.285324 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.281591    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="963f2b89-0b7c-40d5-b96b-d7d1cb72ea6f" containerName="maas-api"
Sep 18 14:15:41.287027 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.287003    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.290192 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.290162    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d634d2\"/\"openshift-service-ca.crt\""
Sep 18 14:15:41.291376 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.291081    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d634d2\"/\"sub-test-model-d634d2-kserve-self-signed-certs\""
Sep 18 14:15:41.291376 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.291121    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d634d2\"/\"default-dockercfg-qwgsc\""
Sep 18 14:15:41.291376 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.291170    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d634d2\"/\"kube-root-ca.crt\""
Sep 18 14:15:41.292415 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.292395    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"]
Sep 18 14:15:41.427662 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.427630    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-home\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.427858 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.427691    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-tmp-dir\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.427858 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.427781    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhrnx\" (UniqueName: \"kubernetes.io/projected/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-kube-api-access-jhrnx\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.427962 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.427871    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-model-cache\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.427962 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.427903    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-dshm\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.427962 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.427939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-tls-certs\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.529322 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.529288    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-model-cache\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.529509 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.529330    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-dshm\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.529509 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.529363    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-tls-certs\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.529509 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.529398    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-home\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.529509 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.529427    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-tmp-dir\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.529509 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.529457    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrnx\" (UniqueName: \"kubernetes.io/projected/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-kube-api-access-jhrnx\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.529822 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.529797    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-model-cache\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.529961 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.529939    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-tmp-dir\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.529961 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.529949    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-home\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.531820 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.531765    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-dshm\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.532155 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.532138    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-tls-certs\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.539846 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.539812    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhrnx\" (UniqueName: \"kubernetes.io/projected/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-kube-api-access-jhrnx\") pod \"sub-test-model-d634d2-kserve-7769b5f549-c2j4p\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") " pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.598518 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.598491    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:41.869375 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:41.869340    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"]
Sep 18 14:15:41.870608 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:15:41.870580    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbc91611_1a4e_44bc_9159_632c3cc9a4a7.slice/crio-96c176f16632103aa343a83b989f3644040fce54997aaf96d67dd38e0e2ec9b8 WatchSource:0}: Error finding container 96c176f16632103aa343a83b989f3644040fce54997aaf96d67dd38e0e2ec9b8: Status 404 returned error can't find the container with id 96c176f16632103aa343a83b989f3644040fce54997aaf96d67dd38e0e2ec9b8
Sep 18 14:15:42.758807 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:42.758767    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p" event={"ID":"fbc91611-1a4e-44bc-9159-632c3cc9a4a7","Type":"ContainerStarted","Data":"1dcbc5c9e51c3f29d928a60daa5b28780a942301daa951e7fbdbd4128065f78a"}
Sep 18 14:15:42.759196 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:42.758816    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p" event={"ID":"fbc91611-1a4e-44bc-9159-632c3cc9a4a7","Type":"ContainerStarted","Data":"96c176f16632103aa343a83b989f3644040fce54997aaf96d67dd38e0e2ec9b8"}
Sep 18 14:15:42.775629 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:42.775586    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p" podStartSLOduration=1.7755713279999998 podStartE2EDuration="1.775571328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:15:42.77426082 +0000 UTC m=+1878.709225965" watchObservedRunningTime="2026-09-18 14:15:42.775571328 +0000 UTC m=+1878.710536471"
Sep 18 14:15:51.599051 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:51.599019    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:51.599508 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:51.599069    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:51.611843 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:51.611815    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:15:51.818989 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:15:51.818961    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:16:17.784225 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:17.784185    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"]
Sep 18 14:16:23.776463 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.776428    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"]
Sep 18 14:16:23.779832 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.779812    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.782670 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.782644    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d25bfb\"/\"rate-test-model-d25bfb-kserve-self-signed-certs\""
Sep 18 14:16:23.782670 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.782665    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d25bfb\"/\"kube-root-ca.crt\""
Sep 18 14:16:23.782836 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.782691    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d25bfb\"/\"default-dockercfg-zxc6l\""
Sep 18 14:16:23.782836 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.782646    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d25bfb\"/\"openshift-service-ca.crt\""
Sep 18 14:16:23.787220 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.787196    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"]
Sep 18 14:16:23.802090 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.802062    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-tls-certs\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.802187 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.802108    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-home\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.802228 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.802207    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfdp9\" (UniqueName: \"kubernetes.io/projected/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-kube-api-access-kfdp9\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.802264 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.802231    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-dshm\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.802298 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.802264    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-tmp-dir\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.802365 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.802345    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-model-cache\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.903402 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.903365    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfdp9\" (UniqueName: \"kubernetes.io/projected/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-kube-api-access-kfdp9\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.903402 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.903403    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-dshm\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.903678 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.903435    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-tmp-dir\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.903678 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.903487    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-model-cache\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.903678 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.903577    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-tls-certs\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.903678 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.903607    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-home\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.903966 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.903943    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-tmp-dir\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.904042 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.903964    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-model-cache\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.904042 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.903981    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-home\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.905734 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.905711    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-dshm\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.905901 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.905883    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-tls-certs\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:23.910814 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:23.910798    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfdp9\" (UniqueName: \"kubernetes.io/projected/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-kube-api-access-kfdp9\") pod \"rate-test-model-d25bfb-kserve-595b4555f-mhhvd\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") " pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:24.091244 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:24.091155    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:24.364563 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:24.364474    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"]
Sep 18 14:16:24.367390 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:16:24.367360    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6bfa616_e5d7_42c7_a8df_94ed9d5794ab.slice/crio-5347597f3f5c3004259645f56acbf96b05568888670bade75f0d17f6af7ccb9c WatchSource:0}: Error finding container 5347597f3f5c3004259645f56acbf96b05568888670bade75f0d17f6af7ccb9c: Status 404 returned error can't find the container with id 5347597f3f5c3004259645f56acbf96b05568888670bade75f0d17f6af7ccb9c
Sep 18 14:16:24.990382 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:24.990348    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd" event={"ID":"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab","Type":"ContainerStarted","Data":"f2ef3b14be4ef51c27483af6f102ad22c1b8fa7da57c3b6cc42cea6ee3249d61"}
Sep 18 14:16:24.990757 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:24.990393    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd" event={"ID":"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab","Type":"ContainerStarted","Data":"5347597f3f5c3004259645f56acbf96b05568888670bade75f0d17f6af7ccb9c"}
Sep 18 14:16:25.007439 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:25.007388    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd" podStartSLOduration=2.007374524 podStartE2EDuration="2.007374524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:16:25.005680114 +0000 UTC m=+1920.940645258" watchObservedRunningTime="2026-09-18 14:16:25.007374524 +0000 UTC m=+1920.942339668"
Sep 18 14:16:32.803434 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:32.803371    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p" podUID="fbc91611-1a4e-44bc-9159-632c3cc9a4a7" containerName="main" containerID="cri-o://1dcbc5c9e51c3f29d928a60daa5b28780a942301daa951e7fbdbd4128065f78a" gracePeriod=45
Sep 18 14:16:33.038194 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.038160    2572 generic.go:356] "Generic (PLEG): container finished" podID="fbc91611-1a4e-44bc-9159-632c3cc9a4a7" containerID="1dcbc5c9e51c3f29d928a60daa5b28780a942301daa951e7fbdbd4128065f78a" exitCode=0
Sep 18 14:16:33.038335 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.038210    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p" event={"ID":"fbc91611-1a4e-44bc-9159-632c3cc9a4a7","Type":"ContainerDied","Data":"1dcbc5c9e51c3f29d928a60daa5b28780a942301daa951e7fbdbd4128065f78a"}
Sep 18 14:16:33.038335 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.038264    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p" event={"ID":"fbc91611-1a4e-44bc-9159-632c3cc9a4a7","Type":"ContainerDied","Data":"96c176f16632103aa343a83b989f3644040fce54997aaf96d67dd38e0e2ec9b8"}
Sep 18 14:16:33.038335 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.038280    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96c176f16632103aa343a83b989f3644040fce54997aaf96d67dd38e0e2ec9b8"
Sep 18 14:16:33.049557 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.049524    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:16:33.186421 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.186334    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-tls-certs\") pod \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") "
Sep 18 14:16:33.186421 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.186373    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-dshm\") pod \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") "
Sep 18 14:16:33.186421 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.186414    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-tmp-dir\") pod \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") "
Sep 18 14:16:33.186757 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.186437    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-home\") pod \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") "
Sep 18 14:16:33.186757 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.186486    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-model-cache\") pod \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") "
Sep 18 14:16:33.186757 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.186581    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhrnx\" (UniqueName: \"kubernetes.io/projected/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-kube-api-access-jhrnx\") pod \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\" (UID: \"fbc91611-1a4e-44bc-9159-632c3cc9a4a7\") "
Sep 18 14:16:33.186757 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.186716    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fbc91611-1a4e-44bc-9159-632c3cc9a4a7" (UID: "fbc91611-1a4e-44bc-9159-632c3cc9a4a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:33.186923 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.186780    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-home" (OuterVolumeSpecName: "home") pod "fbc91611-1a4e-44bc-9159-632c3cc9a4a7" (UID: "fbc91611-1a4e-44bc-9159-632c3cc9a4a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:33.186923 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.186791    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "fbc91611-1a4e-44bc-9159-632c3cc9a4a7" (UID: "fbc91611-1a4e-44bc-9159-632c3cc9a4a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:33.186923 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.186879    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-tmp-dir\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:16:33.186923 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.186892    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-home\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:16:33.186923 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.186901    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-model-cache\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:16:33.188687 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.188666    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-dshm" (OuterVolumeSpecName: "dshm") pod "fbc91611-1a4e-44bc-9159-632c3cc9a4a7" (UID: "fbc91611-1a4e-44bc-9159-632c3cc9a4a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:33.188687 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.188678    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-kube-api-access-jhrnx" (OuterVolumeSpecName: "kube-api-access-jhrnx") pod "fbc91611-1a4e-44bc-9159-632c3cc9a4a7" (UID: "fbc91611-1a4e-44bc-9159-632c3cc9a4a7"). InnerVolumeSpecName "kube-api-access-jhrnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:16:33.188798 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.188679    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fbc91611-1a4e-44bc-9159-632c3cc9a4a7" (UID: "fbc91611-1a4e-44bc-9159-632c3cc9a4a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:16:33.288061 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.288026    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhrnx\" (UniqueName: \"kubernetes.io/projected/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-kube-api-access-jhrnx\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:16:33.288061 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.288057    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-tls-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:16:33.288266 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:33.288071    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc91611-1a4e-44bc-9159-632c3cc9a4a7-dshm\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:16:34.043552 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:34.043499    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"
Sep 18 14:16:34.064141 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:34.064112    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"]
Sep 18 14:16:34.068226 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:34.068202    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d634d2/sub-test-model-d634d2-kserve-7769b5f549-c2j4p"]
Sep 18 14:16:34.091521 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:34.091495    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:34.091662 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:34.091550    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:34.103862 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:34.103837    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:16:34.197090 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:34.197055    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbc91611-1a4e-44bc-9159-632c3cc9a4a7" path="/var/lib/kubelet/pods/fbc91611-1a4e-44bc-9159-632c3cc9a4a7/volumes"
Sep 18 14:16:35.059243 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:16:35.059211    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:17:26.155042 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:26.154949    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"]
Sep 18 14:17:32.174863 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.174825    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"]
Sep 18 14:17:32.175466 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.175411    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc91611-1a4e-44bc-9159-632c3cc9a4a7" containerName="main"
Sep 18 14:17:32.175466 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.175426    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc91611-1a4e-44bc-9159-632c3cc9a4a7" containerName="main"
Sep 18 14:17:32.175653 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.175512    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbc91611-1a4e-44bc-9159-632c3cc9a4a7" containerName="main"
Sep 18 14:17:32.181104 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.181084    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:17:32.183572 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.183545    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e8d773-serving-cert\""
Sep 18 14:17:32.186713 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.186687    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"]
Sep 18 14:17:32.323933 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.323891    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs\" (UID: \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:17:32.323933 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.323935    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs\" (UID: \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:17:32.324157 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.323998    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjdvl\" (UniqueName: \"kubernetes.io/projected/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-kube-api-access-tjdvl\") pod \"maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs\" (UID: \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:17:32.324157 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.324072    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-metrics-certs\") pod \"maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs\" (UID: \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:17:32.425498 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.425405    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjdvl\" (UniqueName: \"kubernetes.io/projected/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-kube-api-access-tjdvl\") pod \"maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs\" (UID: \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:17:32.425498 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.425456    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-metrics-certs\") pod \"maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs\" (UID: \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:17:32.425770 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.425588    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs\" (UID: \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:17:32.425770 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.425611    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs\" (UID: \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:17:32.426396 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.426358    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs\" (UID: \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:17:32.427935 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.427914    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs\" (UID: \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:17:32.428096 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.428073    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-metrics-certs\") pod \"maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs\" (UID: \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:17:32.434238 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.434210    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjdvl\" (UniqueName: \"kubernetes.io/projected/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-kube-api-access-tjdvl\") pod \"maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs\" (UID: \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:17:32.494234 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.494203    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:17:32.776483 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.776453    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"]
Sep 18 14:17:32.777055 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:17:32.777016    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580cab1d_6aeb_4635_ae03_7cbdf5b4a64d.slice/crio-bcbf198ac4ac83e76cd4c71b83f175a3cfec18f6f51b076bce188e3da96e75fa WatchSource:0}: Error finding container bcbf198ac4ac83e76cd4c71b83f175a3cfec18f6f51b076bce188e3da96e75fa: Status 404 returned error can't find the container with id bcbf198ac4ac83e76cd4c71b83f175a3cfec18f6f51b076bce188e3da96e75fa
Sep 18 14:17:32.778578 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.778558    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:17:32.823469 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.823435    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"]
Sep 18 14:17:32.827673 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.827655    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"
Sep 18 14:17:32.829808 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.829785    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e8d773\""
Sep 18 14:17:32.829914 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.829815    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e8d773-dockercfg-z4ln7\""
Sep 18 14:17:32.838522 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.838496    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"]
Sep 18 14:17:32.846434 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.846409    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"]
Sep 18 14:17:32.849759 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.849740    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"
Sep 18 14:17:32.861904 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.861882    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"]
Sep 18 14:17:32.930741 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.930694    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn9lx\" (UniqueName: \"kubernetes.io/projected/be075da4-9e12-4bcd-a5e9-38479e02a609-kube-api-access-pn9lx\") pod \"payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl\" (UID: \"be075da4-9e12-4bcd-a5e9-38479e02a609\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"
Sep 18 14:17:32.930902 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.930820    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be075da4-9e12-4bcd-a5e9-38479e02a609-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl\" (UID: \"be075da4-9e12-4bcd-a5e9-38479e02a609\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"
Sep 18 14:17:32.930902 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.930876    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s55h\" (UniqueName: \"kubernetes.io/projected/a4b0ad44-88aa-4768-a051-82973acaf233-kube-api-access-6s55h\") pod \"payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h\" (UID: \"a4b0ad44-88aa-4768-a051-82973acaf233\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"
Sep 18 14:17:32.930983 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:32.930970    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b0ad44-88aa-4768-a051-82973acaf233-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h\" (UID: \"a4b0ad44-88aa-4768-a051-82973acaf233\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"
Sep 18 14:17:33.032417 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:33.032321    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn9lx\" (UniqueName: \"kubernetes.io/projected/be075da4-9e12-4bcd-a5e9-38479e02a609-kube-api-access-pn9lx\") pod \"payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl\" (UID: \"be075da4-9e12-4bcd-a5e9-38479e02a609\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"
Sep 18 14:17:33.032596 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:33.032433    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be075da4-9e12-4bcd-a5e9-38479e02a609-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl\" (UID: \"be075da4-9e12-4bcd-a5e9-38479e02a609\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"
Sep 18 14:17:33.032596 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:33.032469    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s55h\" (UniqueName: \"kubernetes.io/projected/a4b0ad44-88aa-4768-a051-82973acaf233-kube-api-access-6s55h\") pod \"payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h\" (UID: \"a4b0ad44-88aa-4768-a051-82973acaf233\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"
Sep 18 14:17:33.032596 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:33.032587    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b0ad44-88aa-4768-a051-82973acaf233-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h\" (UID: \"a4b0ad44-88aa-4768-a051-82973acaf233\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"
Sep 18 14:17:33.033199 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:33.033179    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be075da4-9e12-4bcd-a5e9-38479e02a609-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl\" (UID: \"be075da4-9e12-4bcd-a5e9-38479e02a609\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"
Sep 18 14:17:33.033390 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:33.033374    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b0ad44-88aa-4768-a051-82973acaf233-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h\" (UID: \"a4b0ad44-88aa-4768-a051-82973acaf233\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"
Sep 18 14:17:33.040006 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:33.039984    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s55h\" (UniqueName: \"kubernetes.io/projected/a4b0ad44-88aa-4768-a051-82973acaf233-kube-api-access-6s55h\") pod \"payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h\" (UID: \"a4b0ad44-88aa-4768-a051-82973acaf233\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"
Sep 18 14:17:33.040115 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:33.040030    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn9lx\" (UniqueName: \"kubernetes.io/projected/be075da4-9e12-4bcd-a5e9-38479e02a609-kube-api-access-pn9lx\") pod \"payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl\" (UID: \"be075da4-9e12-4bcd-a5e9-38479e02a609\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"
Sep 18 14:17:33.139799 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:33.139766    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"
Sep 18 14:17:33.159889 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:33.159865    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"
Sep 18 14:17:33.367652 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:33.367562    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs" event={"ID":"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d","Type":"ContainerStarted","Data":"bcbf198ac4ac83e76cd4c71b83f175a3cfec18f6f51b076bce188e3da96e75fa"}
Sep 18 14:17:33.411434 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:33.411408    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"]
Sep 18 14:17:33.412659 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:17:33.412626    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe075da4_9e12_4bcd_a5e9_38479e02a609.slice/crio-9d8c711b317d6eb3f870666abdfaf2d48c382e0471561ca7ba7118a1a7dd5586 WatchSource:0}: Error finding container 9d8c711b317d6eb3f870666abdfaf2d48c382e0471561ca7ba7118a1a7dd5586: Status 404 returned error can't find the container with id 9d8c711b317d6eb3f870666abdfaf2d48c382e0471561ca7ba7118a1a7dd5586
Sep 18 14:17:33.440194 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:33.440165    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"]
Sep 18 14:17:33.441556 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:17:33.441501    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b0ad44_88aa_4768_a051_82973acaf233.slice/crio-a391bceb6a48a6186b8c2cc84db93f1f0ade13e0be2f85a65a0b477b4c86cba9 WatchSource:0}: Error finding container a391bceb6a48a6186b8c2cc84db93f1f0ade13e0be2f85a65a0b477b4c86cba9: Status 404 returned error can't find the container with id a391bceb6a48a6186b8c2cc84db93f1f0ade13e0be2f85a65a0b477b4c86cba9
Sep 18 14:17:34.375145 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:34.375104    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs" event={"ID":"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d","Type":"ContainerStarted","Data":"d8e7342d781fa6e1bc5b0d7b0fb0fbde5460ff10f8bf11259915253e0647aa05"}
Sep 18 14:17:34.375627 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:34.375286    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:17:34.376734 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:34.376711    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl" event={"ID":"be075da4-9e12-4bcd-a5e9-38479e02a609","Type":"ContainerStarted","Data":"6b65942b6a5fa8f01cd163dfe8ae94f2f91b3b4d738674e17835e830b597e4c0"}
Sep 18 14:17:34.376845 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:34.376764    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"
Sep 18 14:17:34.376845 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:34.376784    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl" event={"ID":"be075da4-9e12-4bcd-a5e9-38479e02a609","Type":"ContainerStarted","Data":"9d8c711b317d6eb3f870666abdfaf2d48c382e0471561ca7ba7118a1a7dd5586"}
Sep 18 14:17:34.378378 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:34.378357    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h" event={"ID":"a4b0ad44-88aa-4768-a051-82973acaf233","Type":"ContainerStarted","Data":"a01e05f075703ecc77bcecb7f0fa3382a6fd57e253b116b8c73eec27ad0125f1"}
Sep 18 14:17:34.378479 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:34.378390    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h" event={"ID":"a4b0ad44-88aa-4768-a051-82973acaf233","Type":"ContainerStarted","Data":"a391bceb6a48a6186b8c2cc84db93f1f0ade13e0be2f85a65a0b477b4c86cba9"}
Sep 18 14:17:34.378580 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:34.378504    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"
Sep 18 14:17:34.393387 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:34.393334    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs" podStartSLOduration=1.593376066 podStartE2EDuration="2.393318318s" totalImagesPullingTime="799.942252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:17:34.392433228 +0000 UTC m=+1990.327398375" watchObservedRunningTime="2026-09-18 14:17:34.393318318 +0000 UTC m=+1990.328283466"
Sep 18 14:17:34.409983 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:34.409941    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h" podStartSLOduration=2.409928843 podStartE2EDuration="2.409928843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:17:34.408399571 +0000 UTC m=+1990.343364720" watchObservedRunningTime="2026-09-18 14:17:34.409928843 +0000 UTC m=+1990.344894018"
Sep 18 14:17:34.426228 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:34.426186    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl" podStartSLOduration=2.426172268 podStartE2EDuration="2.426172268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:17:34.423843578 +0000 UTC m=+1990.358808725" watchObservedRunningTime="2026-09-18 14:17:34.426172268 +0000 UTC m=+1990.361137413"
Sep 18 14:17:40.389192 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:40.389162    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:17:41.174374 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.174327    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd" podUID="d6bfa616-e5d7-42c7-a8df-94ed9d5794ab" containerName="main" containerID="cri-o://f2ef3b14be4ef51c27483af6f102ad22c1b8fa7da57c3b6cc42cea6ee3249d61" gracePeriod=45
Sep 18 14:17:41.418644 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.418609    2572 generic.go:356] "Generic (PLEG): container finished" podID="d6bfa616-e5d7-42c7-a8df-94ed9d5794ab" containerID="f2ef3b14be4ef51c27483af6f102ad22c1b8fa7da57c3b6cc42cea6ee3249d61" exitCode=0
Sep 18 14:17:41.418948 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.418658    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd" event={"ID":"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab","Type":"ContainerDied","Data":"f2ef3b14be4ef51c27483af6f102ad22c1b8fa7da57c3b6cc42cea6ee3249d61"}
Sep 18 14:17:41.418948 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.418691    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd" event={"ID":"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab","Type":"ContainerDied","Data":"5347597f3f5c3004259645f56acbf96b05568888670bade75f0d17f6af7ccb9c"}
Sep 18 14:17:41.418948 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.418706    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5347597f3f5c3004259645f56acbf96b05568888670bade75f0d17f6af7ccb9c"
Sep 18 14:17:41.419081 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.419065    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:17:41.513362 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.513336    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-model-cache\") pod \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") "
Sep 18 14:17:41.513612 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.513371    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-home\") pod \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") "
Sep 18 14:17:41.513612 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.513390    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-tmp-dir\") pod \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") "
Sep 18 14:17:41.513612 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.513422    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfdp9\" (UniqueName: \"kubernetes.io/projected/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-kube-api-access-kfdp9\") pod \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") "
Sep 18 14:17:41.513612 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.513469    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-dshm\") pod \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") "
Sep 18 14:17:41.513612 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.513492    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-tls-certs\") pod \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\" (UID: \"d6bfa616-e5d7-42c7-a8df-94ed9d5794ab\") "
Sep 18 14:17:41.513612 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.513597    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6bfa616-e5d7-42c7-a8df-94ed9d5794ab" (UID: "d6bfa616-e5d7-42c7-a8df-94ed9d5794ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:41.513612 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.513609    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-home" (OuterVolumeSpecName: "home") pod "d6bfa616-e5d7-42c7-a8df-94ed9d5794ab" (UID: "d6bfa616-e5d7-42c7-a8df-94ed9d5794ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:41.513962 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.513719    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6bfa616-e5d7-42c7-a8df-94ed9d5794ab" (UID: "d6bfa616-e5d7-42c7-a8df-94ed9d5794ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:41.513962 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.513813    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-model-cache\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:17:41.513962 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.513828    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-home\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:17:41.513962 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.513842    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-tmp-dir\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:17:41.515673 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.515648    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6bfa616-e5d7-42c7-a8df-94ed9d5794ab" (UID: "d6bfa616-e5d7-42c7-a8df-94ed9d5794ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:17:41.515947 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.515928    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-dshm" (OuterVolumeSpecName: "dshm") pod "d6bfa616-e5d7-42c7-a8df-94ed9d5794ab" (UID: "d6bfa616-e5d7-42c7-a8df-94ed9d5794ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:41.516014 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.515956    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-kube-api-access-kfdp9" (OuterVolumeSpecName: "kube-api-access-kfdp9") pod "d6bfa616-e5d7-42c7-a8df-94ed9d5794ab" (UID: "d6bfa616-e5d7-42c7-a8df-94ed9d5794ab"). InnerVolumeSpecName "kube-api-access-kfdp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:41.615076 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.615038    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-dshm\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:17:41.615076 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.615074    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-tls-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:17:41.615249 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:41.615090    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfdp9\" (UniqueName: \"kubernetes.io/projected/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab-kube-api-access-kfdp9\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:17:42.424361 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:42.424328    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"
Sep 18 14:17:42.439876 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:42.439844    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"]
Sep 18 14:17:42.443487 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:42.443463    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d25bfb/rate-test-model-d25bfb-kserve-595b4555f-mhhvd"]
Sep 18 14:17:44.197288 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:44.197251    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6bfa616-e5d7-42c7-a8df-94ed9d5794ab" path="/var/lib/kubelet/pods/d6bfa616-e5d7-42c7-a8df-94ed9d5794ab/volumes"
Sep 18 14:17:45.385899 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:45.385867    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"
Sep 18 14:17:45.386344 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:17:45.385934    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"
Sep 18 14:18:02.474143 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.474106    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"]
Sep 18 14:18:02.474626 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.474489    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6bfa616-e5d7-42c7-a8df-94ed9d5794ab" containerName="main"
Sep 18 14:18:02.474626 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.474500    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6bfa616-e5d7-42c7-a8df-94ed9d5794ab" containerName="main"
Sep 18 14:18:02.474626 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.474588    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6bfa616-e5d7-42c7-a8df-94ed9d5794ab" containerName="main"
Sep 18 14:18:02.477703 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.477681    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:18:02.480027 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.480007    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a035eb-serving-cert\""
Sep 18 14:18:02.486980 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.486952    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"]
Sep 18 14:18:02.601074 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.601044    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5jl7\" (UniqueName: \"kubernetes.io/projected/d5a51719-3fee-461f-b00b-e5d44d2b4c19-kube-api-access-g5jl7\") pod \"maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq\" (UID: \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:18:02.601243 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.601096    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5a51719-3fee-461f-b00b-e5d44d2b4c19-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq\" (UID: \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:18:02.601243 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.601129    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a51719-3fee-461f-b00b-e5d44d2b4c19-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq\" (UID: \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:18:02.601243 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.601154    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5a51719-3fee-461f-b00b-e5d44d2b4c19-metrics-certs\") pod \"maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq\" (UID: \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:18:02.702270 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.702233    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5a51719-3fee-461f-b00b-e5d44d2b4c19-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq\" (UID: \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:18:02.702455 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.702276    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a51719-3fee-461f-b00b-e5d44d2b4c19-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq\" (UID: \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:18:02.702455 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.702307    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5a51719-3fee-461f-b00b-e5d44d2b4c19-metrics-certs\") pod \"maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq\" (UID: \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:18:02.702455 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.702363    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5jl7\" (UniqueName: \"kubernetes.io/projected/d5a51719-3fee-461f-b00b-e5d44d2b4c19-kube-api-access-g5jl7\") pod \"maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq\" (UID: \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:18:02.702969 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.702947    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a51719-3fee-461f-b00b-e5d44d2b4c19-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq\" (UID: \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:18:02.704721 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.704699    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5a51719-3fee-461f-b00b-e5d44d2b4c19-metrics-certs\") pod \"maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq\" (UID: \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:18:02.704814 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.704730    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5a51719-3fee-461f-b00b-e5d44d2b4c19-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq\" (UID: \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:18:02.711325 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.711305    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5jl7\" (UniqueName: \"kubernetes.io/projected/d5a51719-3fee-461f-b00b-e5d44d2b4c19-kube-api-access-g5jl7\") pod \"maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq\" (UID: \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:18:02.790199 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:02.790120    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:18:03.086607 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.086563    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"]
Sep 18 14:18:03.093648 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.093625    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"]
Sep 18 14:18:03.098028 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.098013    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"
Sep 18 14:18:03.100722 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.100701    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a035eb\""
Sep 18 14:18:03.102500 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.102482    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a035eb-dockercfg-p4r5b\""
Sep 18 14:18:03.110980 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.110959    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"]
Sep 18 14:18:03.125247 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.125222    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"]
Sep 18 14:18:03.128944 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.128911    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"
Sep 18 14:18:03.146176 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.146142    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"]
Sep 18 14:18:03.206293 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.206205    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3bcf09e-7a12-479d-9369-07ee07d681a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7\" (UID: \"e3bcf09e-7a12-479d-9369-07ee07d681a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"
Sep 18 14:18:03.206479 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.206347    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmh94\" (UniqueName: \"kubernetes.io/projected/e3bcf09e-7a12-479d-9369-07ee07d681a2-kube-api-access-kmh94\") pod \"payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7\" (UID: \"e3bcf09e-7a12-479d-9369-07ee07d681a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"
Sep 18 14:18:03.206479 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.206448    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clllf\" (UniqueName: \"kubernetes.io/projected/29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00-kube-api-access-clllf\") pod \"payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4\" (UID: \"29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"
Sep 18 14:18:03.206622 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.206496    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4\" (UID: \"29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"
Sep 18 14:18:03.308076 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.308031    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3bcf09e-7a12-479d-9369-07ee07d681a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7\" (UID: \"e3bcf09e-7a12-479d-9369-07ee07d681a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"
Sep 18 14:18:03.308337 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.308209    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmh94\" (UniqueName: \"kubernetes.io/projected/e3bcf09e-7a12-479d-9369-07ee07d681a2-kube-api-access-kmh94\") pod \"payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7\" (UID: \"e3bcf09e-7a12-479d-9369-07ee07d681a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"
Sep 18 14:18:03.308337 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.308243    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clllf\" (UniqueName: \"kubernetes.io/projected/29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00-kube-api-access-clllf\") pod \"payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4\" (UID: \"29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"
Sep 18 14:18:03.308337 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.308260    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4\" (UID: \"29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"
Sep 18 14:18:03.308798 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.308779    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3bcf09e-7a12-479d-9369-07ee07d681a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7\" (UID: \"e3bcf09e-7a12-479d-9369-07ee07d681a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"
Sep 18 14:18:03.309020 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.309002    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4\" (UID: \"29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"
Sep 18 14:18:03.316840 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.316807    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmh94\" (UniqueName: \"kubernetes.io/projected/e3bcf09e-7a12-479d-9369-07ee07d681a2-kube-api-access-kmh94\") pod \"payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7\" (UID: \"e3bcf09e-7a12-479d-9369-07ee07d681a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"
Sep 18 14:18:03.316951 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.316902    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clllf\" (UniqueName: \"kubernetes.io/projected/29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00-kube-api-access-clllf\") pod \"payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4\" (UID: \"29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"
Sep 18 14:18:03.408285 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.408213    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"
Sep 18 14:18:03.440587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.440551    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"
Sep 18 14:18:03.548112 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.548077    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq" event={"ID":"d5a51719-3fee-461f-b00b-e5d44d2b4c19","Type":"ContainerStarted","Data":"ee4a5d0e8f24e8f8d1673995db3cb94a6d7235f4037a4710a1fb1d946bcad96e"}
Sep 18 14:18:03.685162 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.685137    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"]
Sep 18 14:18:03.686405 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:18:03.686366    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3bcf09e_7a12_479d_9369_07ee07d681a2.slice/crio-8b089a12f5739827e71e9e4ba688e1eaba8dfae6e725973ab8db4b4c5ed246ee WatchSource:0}: Error finding container 8b089a12f5739827e71e9e4ba688e1eaba8dfae6e725973ab8db4b4c5ed246ee: Status 404 returned error can't find the container with id 8b089a12f5739827e71e9e4ba688e1eaba8dfae6e725973ab8db4b4c5ed246ee
Sep 18 14:18:03.723471 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:03.723446    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"]
Sep 18 14:18:03.734858 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:18:03.734827    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b372d0_3f2e_43bd_a4b2_f3d4a0e22b00.slice/crio-cab64b422ace6f6f880cd5c764f4fab00f482a802271a7af0d56ec324c37b4c0 WatchSource:0}: Error finding container cab64b422ace6f6f880cd5c764f4fab00f482a802271a7af0d56ec324c37b4c0: Status 404 returned error can't find the container with id cab64b422ace6f6f880cd5c764f4fab00f482a802271a7af0d56ec324c37b4c0
Sep 18 14:18:04.556229 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:04.556192    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7" event={"ID":"e3bcf09e-7a12-479d-9369-07ee07d681a2","Type":"ContainerStarted","Data":"a48fa95309a81859284f89df84a5867304c1288b64633e6c8cebd08e39cd82aa"}
Sep 18 14:18:04.556733 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:04.556250    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7" event={"ID":"e3bcf09e-7a12-479d-9369-07ee07d681a2","Type":"ContainerStarted","Data":"8b089a12f5739827e71e9e4ba688e1eaba8dfae6e725973ab8db4b4c5ed246ee"}
Sep 18 14:18:04.556733 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:04.556374    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"
Sep 18 14:18:04.557978 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:04.557948    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq" event={"ID":"d5a51719-3fee-461f-b00b-e5d44d2b4c19","Type":"ContainerStarted","Data":"34b9031ab85329339063764470497623c9057345bdfe99a71c2981903c1099a0"}
Sep 18 14:18:04.558102 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:04.558015    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:18:04.559647 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:04.559624    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4" event={"ID":"29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00","Type":"ContainerStarted","Data":"7c2704c459416a9dca5b4fbc3fd9b3bce087ad19b5b5f26c5e7055defab135b0"}
Sep 18 14:18:04.559820 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:04.559666    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4" event={"ID":"29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00","Type":"ContainerStarted","Data":"cab64b422ace6f6f880cd5c764f4fab00f482a802271a7af0d56ec324c37b4c0"}
Sep 18 14:18:04.559820 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:04.559698    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"
Sep 18 14:18:04.572234 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:04.572182    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7" podStartSLOduration=1.5721689840000002 podStartE2EDuration="1.572168984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:04.571121622 +0000 UTC m=+2020.506086767" watchObservedRunningTime="2026-09-18 14:18:04.572168984 +0000 UTC m=+2020.507134119"
Sep 18 14:18:04.589695 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:04.589650    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq" podStartSLOduration=2.102706235 podStartE2EDuration="2.589638148s" totalImagesPullingTime="486.931913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:04.587028666 +0000 UTC m=+2020.521993810" watchObservedRunningTime="2026-09-18 14:18:04.589638148 +0000 UTC m=+2020.524603292"
Sep 18 14:18:04.604296 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:04.604257    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4" podStartSLOduration=1.604243997 podStartE2EDuration="1.604243997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:04.602651349 +0000 UTC m=+2020.537616507" watchObservedRunningTime="2026-09-18 14:18:04.604243997 +0000 UTC m=+2020.539209202"
Sep 18 14:18:10.571188 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:10.571156    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:18:15.568434 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:15.568399    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"
Sep 18 14:18:15.568890 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:15.568473    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"
Sep 18 14:18:29.732095 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:18:29.732056    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4_29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00/payload-processing/0.log"
Sep 18 14:19:05.658201 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:05.658165    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h_a4b0ad44-88aa-4768-a051-82973acaf233/payload-processing/0.log"
Sep 18 14:19:05.892904 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:05.892873    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4_29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00/payload-processing/0.log"
Sep 18 14:19:06.013471 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:06.013444    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4_29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00/payload-processing/0.log"
Sep 18 14:19:14.017621 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.017587    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"]
Sep 18 14:19:14.021265 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.021248    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"
Sep 18 14:19:14.023679 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.023658    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-4ce51f\""
Sep 18 14:19:14.023800 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.023703    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-4ce51f-dockercfg-nck7n\""
Sep 18 14:19:14.034769 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.034744    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"]
Sep 18 14:19:14.066289 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.066262    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc6093d-2532-47a6-ac15-4311a3ada407-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g\" (UID: \"4fc6093d-2532-47a6-ac15-4311a3ada407\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"
Sep 18 14:19:14.066391 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.066303    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfk9g\" (UniqueName: \"kubernetes.io/projected/4fc6093d-2532-47a6-ac15-4311a3ada407-kube-api-access-nfk9g\") pod \"payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g\" (UID: \"4fc6093d-2532-47a6-ac15-4311a3ada407\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"
Sep 18 14:19:14.167210 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.167173    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfk9g\" (UniqueName: \"kubernetes.io/projected/4fc6093d-2532-47a6-ac15-4311a3ada407-kube-api-access-nfk9g\") pod \"payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g\" (UID: \"4fc6093d-2532-47a6-ac15-4311a3ada407\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"
Sep 18 14:19:14.167384 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.167270    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc6093d-2532-47a6-ac15-4311a3ada407-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g\" (UID: \"4fc6093d-2532-47a6-ac15-4311a3ada407\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"
Sep 18 14:19:14.167935 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.167916    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc6093d-2532-47a6-ac15-4311a3ada407-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g\" (UID: \"4fc6093d-2532-47a6-ac15-4311a3ada407\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"
Sep 18 14:19:14.175109 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.175085    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfk9g\" (UniqueName: \"kubernetes.io/projected/4fc6093d-2532-47a6-ac15-4311a3ada407-kube-api-access-nfk9g\") pod \"payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g\" (UID: \"4fc6093d-2532-47a6-ac15-4311a3ada407\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"
Sep 18 14:19:14.331290 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.331209    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"
Sep 18 14:19:14.602135 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.602107    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"]
Sep 18 14:19:14.603003 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:19:14.602971    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc6093d_2532_47a6_ac15_4311a3ada407.slice/crio-9fac48330a825afb301739f060cd893e3b00f55c04ec857084b301e02bbe17e8 WatchSource:0}: Error finding container 9fac48330a825afb301739f060cd893e3b00f55c04ec857084b301e02bbe17e8: Status 404 returned error can't find the container with id 9fac48330a825afb301739f060cd893e3b00f55c04ec857084b301e02bbe17e8
Sep 18 14:19:14.965716 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.965675    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g" event={"ID":"4fc6093d-2532-47a6-ac15-4311a3ada407","Type":"ContainerStarted","Data":"d4d3427ef6778fa451eb86512e5c087771d87a430621844a69d99c212aae1488"}
Sep 18 14:19:14.965875 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.965738    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g" event={"ID":"4fc6093d-2532-47a6-ac15-4311a3ada407","Type":"ContainerStarted","Data":"9fac48330a825afb301739f060cd893e3b00f55c04ec857084b301e02bbe17e8"}
Sep 18 14:19:14.965875 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.965765    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"
Sep 18 14:19:14.984183 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:14.984137    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g" podStartSLOduration=1.984121837 podStartE2EDuration="1.984121837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:19:14.982328495 +0000 UTC m=+2090.917293640" watchObservedRunningTime="2026-09-18 14:19:14.984121837 +0000 UTC m=+2090.919086980"
Sep 18 14:19:25.973641 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:25.973610    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"
Sep 18 14:19:26.368949 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:26.368880    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 14:19:26.376963 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:26.376935    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 14:19:51.800441 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:51.800407    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"]
Sep 18 14:19:51.800981 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:51.800669    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g" podUID="4fc6093d-2532-47a6-ac15-4311a3ada407" containerName="payload-pre-processing" containerID="cri-o://d4d3427ef6778fa451eb86512e5c087771d87a430621844a69d99c212aae1488" gracePeriod=30
Sep 18 14:19:52.051292 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:52.051233    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"
Sep 18 14:19:52.122514 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:52.122468    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc6093d-2532-47a6-ac15-4311a3ada407-plugins-config-volume\") pod \"4fc6093d-2532-47a6-ac15-4311a3ada407\" (UID: \"4fc6093d-2532-47a6-ac15-4311a3ada407\") "
Sep 18 14:19:52.122736 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:52.122557    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfk9g\" (UniqueName: \"kubernetes.io/projected/4fc6093d-2532-47a6-ac15-4311a3ada407-kube-api-access-nfk9g\") pod \"4fc6093d-2532-47a6-ac15-4311a3ada407\" (UID: \"4fc6093d-2532-47a6-ac15-4311a3ada407\") "
Sep 18 14:19:52.123649 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:52.123612    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc6093d-2532-47a6-ac15-4311a3ada407-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4fc6093d-2532-47a6-ac15-4311a3ada407" (UID: "4fc6093d-2532-47a6-ac15-4311a3ada407"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:19:52.125383 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:52.125359    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc6093d-2532-47a6-ac15-4311a3ada407-kube-api-access-nfk9g" (OuterVolumeSpecName: "kube-api-access-nfk9g") pod "4fc6093d-2532-47a6-ac15-4311a3ada407" (UID: "4fc6093d-2532-47a6-ac15-4311a3ada407"). InnerVolumeSpecName "kube-api-access-nfk9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:19:52.186807 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:52.184994    2572 generic.go:356] "Generic (PLEG): container finished" podID="4fc6093d-2532-47a6-ac15-4311a3ada407" containerID="d4d3427ef6778fa451eb86512e5c087771d87a430621844a69d99c212aae1488" exitCode=0
Sep 18 14:19:52.186807 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:52.185157    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g" event={"ID":"4fc6093d-2532-47a6-ac15-4311a3ada407","Type":"ContainerDied","Data":"d4d3427ef6778fa451eb86512e5c087771d87a430621844a69d99c212aae1488"}
Sep 18 14:19:52.186807 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:52.185207    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g" event={"ID":"4fc6093d-2532-47a6-ac15-4311a3ada407","Type":"ContainerDied","Data":"9fac48330a825afb301739f060cd893e3b00f55c04ec857084b301e02bbe17e8"}
Sep 18 14:19:52.186807 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:52.185234    2572 scope.go:117] "RemoveContainer" containerID="d4d3427ef6778fa451eb86512e5c087771d87a430621844a69d99c212aae1488"
Sep 18 14:19:52.186807 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:52.185403    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"
Sep 18 14:19:52.196585 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:52.196564    2572 scope.go:117] "RemoveContainer" containerID="d4d3427ef6778fa451eb86512e5c087771d87a430621844a69d99c212aae1488"
Sep 18 14:19:52.196862 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:19:52.196840    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d3427ef6778fa451eb86512e5c087771d87a430621844a69d99c212aae1488\": container with ID starting with d4d3427ef6778fa451eb86512e5c087771d87a430621844a69d99c212aae1488 not found: ID does not exist" containerID="d4d3427ef6778fa451eb86512e5c087771d87a430621844a69d99c212aae1488"
Sep 18 14:19:52.196922 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:52.196872    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d3427ef6778fa451eb86512e5c087771d87a430621844a69d99c212aae1488"} err="failed to get container status \"d4d3427ef6778fa451eb86512e5c087771d87a430621844a69d99c212aae1488\": rpc error: code = NotFound desc = could not find container \"d4d3427ef6778fa451eb86512e5c087771d87a430621844a69d99c212aae1488\": container with ID starting with d4d3427ef6778fa451eb86512e5c087771d87a430621844a69d99c212aae1488 not found: ID does not exist"
Sep 18 14:19:52.208231 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:52.208208    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"]
Sep 18 14:19:52.217770 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:52.217743    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4ce51f-865b8b57b6-s7x2g"]
Sep 18 14:19:52.224025 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:52.223994    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc6093d-2532-47a6-ac15-4311a3ada407-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:19:52.224127 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:52.224032    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfk9g\" (UniqueName: \"kubernetes.io/projected/4fc6093d-2532-47a6-ac15-4311a3ada407-kube-api-access-nfk9g\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:19:54.197543 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:19:54.197489    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fc6093d-2532-47a6-ac15-4311a3ada407" path="/var/lib/kubelet/pods/4fc6093d-2532-47a6-ac15-4311a3ada407/volumes"
Sep 18 14:20:17.777256 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:17.777145    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"]
Sep 18 14:20:17.777844 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:17.777474    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs" podUID="580cab1d-6aeb-4635-ae03-7cbdf5b4a64d" containerName="maas-api" containerID="cri-o://d8e7342d781fa6e1bc5b0d7b0fb0fbde5460ff10f8bf11259915253e0647aa05" gracePeriod=30
Sep 18 14:20:17.831617 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:17.831585    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"]
Sep 18 14:20:17.831879 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:17.831830    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h" podUID="a4b0ad44-88aa-4768-a051-82973acaf233" containerName="payload-processing" containerID="cri-o://a01e05f075703ecc77bcecb7f0fa3382a6fd57e253b116b8c73eec27ad0125f1" gracePeriod=30
Sep 18 14:20:17.840811 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:17.840786    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"]
Sep 18 14:20:17.840962 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:17.840944    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl" podUID="be075da4-9e12-4bcd-a5e9-38479e02a609" containerName="payload-pre-processing" containerID="cri-o://6b65942b6a5fa8f01cd163dfe8ae94f2f91b3b4d738674e17835e830b597e4c0" gracePeriod=30
Sep 18 14:20:18.092656 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.092629    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:20:18.145967 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.145942    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"
Sep 18 14:20:18.166760 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.166727    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"
Sep 18 14:20:18.288734 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.288703    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-metrics-certs\") pod \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\" (UID: \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\") "
Sep 18 14:20:18.288902 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.288740    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-maas-api-tls\") pod \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\" (UID: \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\") "
Sep 18 14:20:18.288902 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.288769    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6s55h\" (UniqueName: \"kubernetes.io/projected/a4b0ad44-88aa-4768-a051-82973acaf233-kube-api-access-6s55h\") pod \"a4b0ad44-88aa-4768-a051-82973acaf233\" (UID: \"a4b0ad44-88aa-4768-a051-82973acaf233\") "
Sep 18 14:20:18.288902 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.288799    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjdvl\" (UniqueName: \"kubernetes.io/projected/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-kube-api-access-tjdvl\") pod \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\" (UID: \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\") "
Sep 18 14:20:18.289006 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.288963    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn9lx\" (UniqueName: \"kubernetes.io/projected/be075da4-9e12-4bcd-a5e9-38479e02a609-kube-api-access-pn9lx\") pod \"be075da4-9e12-4bcd-a5e9-38479e02a609\" (UID: \"be075da4-9e12-4bcd-a5e9-38479e02a609\") "
Sep 18 14:20:18.289056 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.289025    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be075da4-9e12-4bcd-a5e9-38479e02a609-plugins-config-volume\") pod \"be075da4-9e12-4bcd-a5e9-38479e02a609\" (UID: \"be075da4-9e12-4bcd-a5e9-38479e02a609\") "
Sep 18 14:20:18.289107 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.289055    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b0ad44-88aa-4768-a051-82973acaf233-plugins-config-volume\") pod \"a4b0ad44-88aa-4768-a051-82973acaf233\" (UID: \"a4b0ad44-88aa-4768-a051-82973acaf233\") "
Sep 18 14:20:18.289107 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.289082    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-openshift-service-ca\") pod \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\" (UID: \"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d\") "
Sep 18 14:20:18.289916 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.289457    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be075da4-9e12-4bcd-a5e9-38479e02a609-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be075da4-9e12-4bcd-a5e9-38479e02a609" (UID: "be075da4-9e12-4bcd-a5e9-38479e02a609"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:20:18.289916 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.289865    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "580cab1d-6aeb-4635-ae03-7cbdf5b4a64d" (UID: "580cab1d-6aeb-4635-ae03-7cbdf5b4a64d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:20:18.289916 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.289864    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b0ad44-88aa-4768-a051-82973acaf233-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a4b0ad44-88aa-4768-a051-82973acaf233" (UID: "a4b0ad44-88aa-4768-a051-82973acaf233"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:20:18.291190 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.291164    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be075da4-9e12-4bcd-a5e9-38479e02a609-kube-api-access-pn9lx" (OuterVolumeSpecName: "kube-api-access-pn9lx") pod "be075da4-9e12-4bcd-a5e9-38479e02a609" (UID: "be075da4-9e12-4bcd-a5e9-38479e02a609"). InnerVolumeSpecName "kube-api-access-pn9lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:20:18.291517 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.291496    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "580cab1d-6aeb-4635-ae03-7cbdf5b4a64d" (UID: "580cab1d-6aeb-4635-ae03-7cbdf5b4a64d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:20:18.291517 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.291506    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-kube-api-access-tjdvl" (OuterVolumeSpecName: "kube-api-access-tjdvl") pod "580cab1d-6aeb-4635-ae03-7cbdf5b4a64d" (UID: "580cab1d-6aeb-4635-ae03-7cbdf5b4a64d"). InnerVolumeSpecName "kube-api-access-tjdvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:20:18.291637 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.291514    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b0ad44-88aa-4768-a051-82973acaf233-kube-api-access-6s55h" (OuterVolumeSpecName: "kube-api-access-6s55h") pod "a4b0ad44-88aa-4768-a051-82973acaf233" (UID: "a4b0ad44-88aa-4768-a051-82973acaf233"). InnerVolumeSpecName "kube-api-access-6s55h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:20:18.291637 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.291514    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "580cab1d-6aeb-4635-ae03-7cbdf5b4a64d" (UID: "580cab1d-6aeb-4635-ae03-7cbdf5b4a64d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:20:18.337677 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.337573    2572 generic.go:356] "Generic (PLEG): container finished" podID="a4b0ad44-88aa-4768-a051-82973acaf233" containerID="a01e05f075703ecc77bcecb7f0fa3382a6fd57e253b116b8c73eec27ad0125f1" exitCode=0
Sep 18 14:20:18.337677 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.337624    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"
Sep 18 14:20:18.337677 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.337657    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h" event={"ID":"a4b0ad44-88aa-4768-a051-82973acaf233","Type":"ContainerDied","Data":"a01e05f075703ecc77bcecb7f0fa3382a6fd57e253b116b8c73eec27ad0125f1"}
Sep 18 14:20:18.337958 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.337717    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h" event={"ID":"a4b0ad44-88aa-4768-a051-82973acaf233","Type":"ContainerDied","Data":"a391bceb6a48a6186b8c2cc84db93f1f0ade13e0be2f85a65a0b477b4c86cba9"}
Sep 18 14:20:18.337958 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.337752    2572 scope.go:117] "RemoveContainer" containerID="a01e05f075703ecc77bcecb7f0fa3382a6fd57e253b116b8c73eec27ad0125f1"
Sep 18 14:20:18.339236 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.339210    2572 generic.go:356] "Generic (PLEG): container finished" podID="580cab1d-6aeb-4635-ae03-7cbdf5b4a64d" containerID="d8e7342d781fa6e1bc5b0d7b0fb0fbde5460ff10f8bf11259915253e0647aa05" exitCode=0
Sep 18 14:20:18.339370 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.339239    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs" event={"ID":"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d","Type":"ContainerDied","Data":"d8e7342d781fa6e1bc5b0d7b0fb0fbde5460ff10f8bf11259915253e0647aa05"}
Sep 18 14:20:18.339370 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.339268    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"
Sep 18 14:20:18.339370 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.339305    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs" event={"ID":"580cab1d-6aeb-4635-ae03-7cbdf5b4a64d","Type":"ContainerDied","Data":"bcbf198ac4ac83e76cd4c71b83f175a3cfec18f6f51b076bce188e3da96e75fa"}
Sep 18 14:20:18.340952 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.340915    2572 generic.go:356] "Generic (PLEG): container finished" podID="be075da4-9e12-4bcd-a5e9-38479e02a609" containerID="6b65942b6a5fa8f01cd163dfe8ae94f2f91b3b4d738674e17835e830b597e4c0" exitCode=0
Sep 18 14:20:18.341091 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.341006    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl" event={"ID":"be075da4-9e12-4bcd-a5e9-38479e02a609","Type":"ContainerDied","Data":"6b65942b6a5fa8f01cd163dfe8ae94f2f91b3b4d738674e17835e830b597e4c0"}
Sep 18 14:20:18.341091 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.341038    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl" event={"ID":"be075da4-9e12-4bcd-a5e9-38479e02a609","Type":"ContainerDied","Data":"9d8c711b317d6eb3f870666abdfaf2d48c382e0471561ca7ba7118a1a7dd5586"}
Sep 18 14:20:18.341091 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.341082    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"
Sep 18 14:20:18.353503 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.353475    2572 scope.go:117] "RemoveContainer" containerID="a01e05f075703ecc77bcecb7f0fa3382a6fd57e253b116b8c73eec27ad0125f1"
Sep 18 14:20:18.353917 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:20:18.353891    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01e05f075703ecc77bcecb7f0fa3382a6fd57e253b116b8c73eec27ad0125f1\": container with ID starting with a01e05f075703ecc77bcecb7f0fa3382a6fd57e253b116b8c73eec27ad0125f1 not found: ID does not exist" containerID="a01e05f075703ecc77bcecb7f0fa3382a6fd57e253b116b8c73eec27ad0125f1"
Sep 18 14:20:18.354007 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.353930    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01e05f075703ecc77bcecb7f0fa3382a6fd57e253b116b8c73eec27ad0125f1"} err="failed to get container status \"a01e05f075703ecc77bcecb7f0fa3382a6fd57e253b116b8c73eec27ad0125f1\": rpc error: code = NotFound desc = could not find container \"a01e05f075703ecc77bcecb7f0fa3382a6fd57e253b116b8c73eec27ad0125f1\": container with ID starting with a01e05f075703ecc77bcecb7f0fa3382a6fd57e253b116b8c73eec27ad0125f1 not found: ID does not exist"
Sep 18 14:20:18.354007 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.353958    2572 scope.go:117] "RemoveContainer" containerID="d8e7342d781fa6e1bc5b0d7b0fb0fbde5460ff10f8bf11259915253e0647aa05"
Sep 18 14:20:18.363233 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.363216    2572 scope.go:117] "RemoveContainer" containerID="d8e7342d781fa6e1bc5b0d7b0fb0fbde5460ff10f8bf11259915253e0647aa05"
Sep 18 14:20:18.363507 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:20:18.363488    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e7342d781fa6e1bc5b0d7b0fb0fbde5460ff10f8bf11259915253e0647aa05\": container with ID starting with d8e7342d781fa6e1bc5b0d7b0fb0fbde5460ff10f8bf11259915253e0647aa05 not found: ID does not exist" containerID="d8e7342d781fa6e1bc5b0d7b0fb0fbde5460ff10f8bf11259915253e0647aa05"
Sep 18 14:20:18.363621 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.363514    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e7342d781fa6e1bc5b0d7b0fb0fbde5460ff10f8bf11259915253e0647aa05"} err="failed to get container status \"d8e7342d781fa6e1bc5b0d7b0fb0fbde5460ff10f8bf11259915253e0647aa05\": rpc error: code = NotFound desc = could not find container \"d8e7342d781fa6e1bc5b0d7b0fb0fbde5460ff10f8bf11259915253e0647aa05\": container with ID starting with d8e7342d781fa6e1bc5b0d7b0fb0fbde5460ff10f8bf11259915253e0647aa05 not found: ID does not exist"
Sep 18 14:20:18.363621 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.363545    2572 scope.go:117] "RemoveContainer" containerID="6b65942b6a5fa8f01cd163dfe8ae94f2f91b3b4d738674e17835e830b597e4c0"
Sep 18 14:20:18.371441 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.371415    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"]
Sep 18 14:20:18.373307 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.373283    2572 scope.go:117] "RemoveContainer" containerID="6b65942b6a5fa8f01cd163dfe8ae94f2f91b3b4d738674e17835e830b597e4c0"
Sep 18 14:20:18.373631 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:20:18.373601    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b65942b6a5fa8f01cd163dfe8ae94f2f91b3b4d738674e17835e830b597e4c0\": container with ID starting with 6b65942b6a5fa8f01cd163dfe8ae94f2f91b3b4d738674e17835e830b597e4c0 not found: ID does not exist" containerID="6b65942b6a5fa8f01cd163dfe8ae94f2f91b3b4d738674e17835e830b597e4c0"
Sep 18 14:20:18.373697 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.373637    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b65942b6a5fa8f01cd163dfe8ae94f2f91b3b4d738674e17835e830b597e4c0"} err="failed to get container status \"6b65942b6a5fa8f01cd163dfe8ae94f2f91b3b4d738674e17835e830b597e4c0\": rpc error: code = NotFound desc = could not find container \"6b65942b6a5fa8f01cd163dfe8ae94f2f91b3b4d738674e17835e830b597e4c0\": container with ID starting with 6b65942b6a5fa8f01cd163dfe8ae94f2f91b3b4d738674e17835e830b597e4c0 not found: ID does not exist"
Sep 18 14:20:18.374705 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.374681    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e8d773-5c68b457cc-rx76h"]
Sep 18 14:20:18.386799 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.386778    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"]
Sep 18 14:20:18.390499 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.390475    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e8d773-7cbc777764-jzjbs"]
Sep 18 14:20:18.390980 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.390959    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn9lx\" (UniqueName: \"kubernetes.io/projected/be075da4-9e12-4bcd-a5e9-38479e02a609-kube-api-access-pn9lx\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:20:18.391078 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.390986    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be075da4-9e12-4bcd-a5e9-38479e02a609-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:20:18.391078 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.391002    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b0ad44-88aa-4768-a051-82973acaf233-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:20:18.391078 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.391016    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:20:18.391078 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.391027    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:20:18.391078 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.391035    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:20:18.391078 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.391043    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s55h\" (UniqueName: \"kubernetes.io/projected/a4b0ad44-88aa-4768-a051-82973acaf233-kube-api-access-6s55h\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:20:18.391078 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.391052    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjdvl\" (UniqueName: \"kubernetes.io/projected/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d-kube-api-access-tjdvl\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:20:18.401819 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.401797    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"]
Sep 18 14:20:18.405039 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:18.405020    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e8d773-7bcbc55766-5gxzl"]
Sep 18 14:20:20.197136 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:20.197103    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="580cab1d-6aeb-4635-ae03-7cbdf5b4a64d" path="/var/lib/kubelet/pods/580cab1d-6aeb-4635-ae03-7cbdf5b4a64d/volumes"
Sep 18 14:20:20.197496 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:20.197466    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4b0ad44-88aa-4768-a051-82973acaf233" path="/var/lib/kubelet/pods/a4b0ad44-88aa-4768-a051-82973acaf233/volumes"
Sep 18 14:20:20.197817 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:20:20.197804    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be075da4-9e12-4bcd-a5e9-38479e02a609" path="/var/lib/kubelet/pods/be075da4-9e12-4bcd-a5e9-38479e02a609/volumes"
Sep 18 14:21:07.767138 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:07.767095    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"]
Sep 18 14:21:07.767645 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:07.767368    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq" podUID="d5a51719-3fee-461f-b00b-e5d44d2b4c19" containerName="maas-api" containerID="cri-o://34b9031ab85329339063764470497623c9057345bdfe99a71c2981903c1099a0" gracePeriod=30
Sep 18 14:21:07.824150 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:07.824110    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"]
Sep 18 14:21:07.824415 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:07.824365    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4" podUID="29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00" containerName="payload-processing" containerID="cri-o://7c2704c459416a9dca5b4fbc3fd9b3bce087ad19b5b5f26c5e7055defab135b0" gracePeriod=30
Sep 18 14:21:07.839614 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:07.839583    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"]
Sep 18 14:21:07.839865 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:07.839817    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7" podUID="e3bcf09e-7a12-479d-9369-07ee07d681a2" containerName="payload-pre-processing" containerID="cri-o://a48fa95309a81859284f89df84a5867304c1288b64633e6c8cebd08e39cd82aa" gracePeriod=30
Sep 18 14:21:08.122143 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.122119    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"
Sep 18 14:21:08.125587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.125559    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:21:08.134187 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.134165    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"
Sep 18 14:21:08.160091 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.160061    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clllf\" (UniqueName: \"kubernetes.io/projected/29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00-kube-api-access-clllf\") pod \"29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00\" (UID: \"29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00\") "
Sep 18 14:21:08.160239 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.160138    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5a51719-3fee-461f-b00b-e5d44d2b4c19-maas-api-tls\") pod \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\" (UID: \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\") "
Sep 18 14:21:08.160239 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.160197    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5jl7\" (UniqueName: \"kubernetes.io/projected/d5a51719-3fee-461f-b00b-e5d44d2b4c19-kube-api-access-g5jl7\") pod \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\" (UID: \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\") "
Sep 18 14:21:08.160239 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.160226    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00-plugins-config-volume\") pod \"29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00\" (UID: \"29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00\") "
Sep 18 14:21:08.160395 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.160291    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5a51719-3fee-461f-b00b-e5d44d2b4c19-metrics-certs\") pod \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\" (UID: \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\") "
Sep 18 14:21:08.160395 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.160319    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a51719-3fee-461f-b00b-e5d44d2b4c19-openshift-service-ca\") pod \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\" (UID: \"d5a51719-3fee-461f-b00b-e5d44d2b4c19\") "
Sep 18 14:21:08.160805 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.160751    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00" (UID: "29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:21:08.160805 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.160758    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a51719-3fee-461f-b00b-e5d44d2b4c19-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5a51719-3fee-461f-b00b-e5d44d2b4c19" (UID: "d5a51719-3fee-461f-b00b-e5d44d2b4c19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:21:08.162686 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.162591    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00-kube-api-access-clllf" (OuterVolumeSpecName: "kube-api-access-clllf") pod "29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00" (UID: "29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00"). InnerVolumeSpecName "kube-api-access-clllf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:21:08.162804 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.162774    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a51719-3fee-461f-b00b-e5d44d2b4c19-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d5a51719-3fee-461f-b00b-e5d44d2b4c19" (UID: "d5a51719-3fee-461f-b00b-e5d44d2b4c19"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:21:08.162893 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.162871    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a51719-3fee-461f-b00b-e5d44d2b4c19-kube-api-access-g5jl7" (OuterVolumeSpecName: "kube-api-access-g5jl7") pod "d5a51719-3fee-461f-b00b-e5d44d2b4c19" (UID: "d5a51719-3fee-461f-b00b-e5d44d2b4c19"). InnerVolumeSpecName "kube-api-access-g5jl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:21:08.163433 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.163417    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a51719-3fee-461f-b00b-e5d44d2b4c19-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5a51719-3fee-461f-b00b-e5d44d2b4c19" (UID: "d5a51719-3fee-461f-b00b-e5d44d2b4c19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:21:08.261178 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.261151    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3bcf09e-7a12-479d-9369-07ee07d681a2-plugins-config-volume\") pod \"e3bcf09e-7a12-479d-9369-07ee07d681a2\" (UID: \"e3bcf09e-7a12-479d-9369-07ee07d681a2\") "
Sep 18 14:21:08.261334 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.261213    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmh94\" (UniqueName: \"kubernetes.io/projected/e3bcf09e-7a12-479d-9369-07ee07d681a2-kube-api-access-kmh94\") pod \"e3bcf09e-7a12-479d-9369-07ee07d681a2\" (UID: \"e3bcf09e-7a12-479d-9369-07ee07d681a2\") "
Sep 18 14:21:08.261442 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.261429    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5a51719-3fee-461f-b00b-e5d44d2b4c19-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:21:08.261480 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.261445    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5jl7\" (UniqueName: \"kubernetes.io/projected/d5a51719-3fee-461f-b00b-e5d44d2b4c19-kube-api-access-g5jl7\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:21:08.261480 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.261461    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:21:08.261480 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.261471    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5a51719-3fee-461f-b00b-e5d44d2b4c19-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:21:08.261480 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.261480    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a51719-3fee-461f-b00b-e5d44d2b4c19-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:21:08.261635 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.261475    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3bcf09e-7a12-479d-9369-07ee07d681a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e3bcf09e-7a12-479d-9369-07ee07d681a2" (UID: "e3bcf09e-7a12-479d-9369-07ee07d681a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:21:08.261635 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.261488    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clllf\" (UniqueName: \"kubernetes.io/projected/29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00-kube-api-access-clllf\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:21:08.263374 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.263353    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bcf09e-7a12-479d-9369-07ee07d681a2-kube-api-access-kmh94" (OuterVolumeSpecName: "kube-api-access-kmh94") pod "e3bcf09e-7a12-479d-9369-07ee07d681a2" (UID: "e3bcf09e-7a12-479d-9369-07ee07d681a2"). InnerVolumeSpecName "kube-api-access-kmh94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:21:08.362514 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.362430    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmh94\" (UniqueName: \"kubernetes.io/projected/e3bcf09e-7a12-479d-9369-07ee07d681a2-kube-api-access-kmh94\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:21:08.362514 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.362460    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3bcf09e-7a12-479d-9369-07ee07d681a2-plugins-config-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:21:08.624167 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.624088    2572 generic.go:356] "Generic (PLEG): container finished" podID="29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00" containerID="7c2704c459416a9dca5b4fbc3fd9b3bce087ad19b5b5f26c5e7055defab135b0" exitCode=0
Sep 18 14:21:08.624167 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.624127    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4" event={"ID":"29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00","Type":"ContainerDied","Data":"7c2704c459416a9dca5b4fbc3fd9b3bce087ad19b5b5f26c5e7055defab135b0"}
Sep 18 14:21:08.624464 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.624181    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4" event={"ID":"29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00","Type":"ContainerDied","Data":"cab64b422ace6f6f880cd5c764f4fab00f482a802271a7af0d56ec324c37b4c0"}
Sep 18 14:21:08.624464 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.624104    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"
Sep 18 14:21:08.624464 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.624209    2572 scope.go:117] "RemoveContainer" containerID="7c2704c459416a9dca5b4fbc3fd9b3bce087ad19b5b5f26c5e7055defab135b0"
Sep 18 14:21:08.625644 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.625622    2572 generic.go:356] "Generic (PLEG): container finished" podID="e3bcf09e-7a12-479d-9369-07ee07d681a2" containerID="a48fa95309a81859284f89df84a5867304c1288b64633e6c8cebd08e39cd82aa" exitCode=0
Sep 18 14:21:08.625775 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.625655    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7" event={"ID":"e3bcf09e-7a12-479d-9369-07ee07d681a2","Type":"ContainerDied","Data":"a48fa95309a81859284f89df84a5867304c1288b64633e6c8cebd08e39cd82aa"}
Sep 18 14:21:08.625775 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.625676    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"
Sep 18 14:21:08.625775 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.625702    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7" event={"ID":"e3bcf09e-7a12-479d-9369-07ee07d681a2","Type":"ContainerDied","Data":"8b089a12f5739827e71e9e4ba688e1eaba8dfae6e725973ab8db4b4c5ed246ee"}
Sep 18 14:21:08.627371 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.627346    2572 generic.go:356] "Generic (PLEG): container finished" podID="d5a51719-3fee-461f-b00b-e5d44d2b4c19" containerID="34b9031ab85329339063764470497623c9057345bdfe99a71c2981903c1099a0" exitCode=0
Sep 18 14:21:08.627505 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.627385    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"
Sep 18 14:21:08.627505 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.627403    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq" event={"ID":"d5a51719-3fee-461f-b00b-e5d44d2b4c19","Type":"ContainerDied","Data":"34b9031ab85329339063764470497623c9057345bdfe99a71c2981903c1099a0"}
Sep 18 14:21:08.627505 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.627440    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq" event={"ID":"d5a51719-3fee-461f-b00b-e5d44d2b4c19","Type":"ContainerDied","Data":"ee4a5d0e8f24e8f8d1673995db3cb94a6d7235f4037a4710a1fb1d946bcad96e"}
Sep 18 14:21:08.634154 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.633992    2572 scope.go:117] "RemoveContainer" containerID="7c2704c459416a9dca5b4fbc3fd9b3bce087ad19b5b5f26c5e7055defab135b0"
Sep 18 14:21:08.634404 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:21:08.634366    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2704c459416a9dca5b4fbc3fd9b3bce087ad19b5b5f26c5e7055defab135b0\": container with ID starting with 7c2704c459416a9dca5b4fbc3fd9b3bce087ad19b5b5f26c5e7055defab135b0 not found: ID does not exist" containerID="7c2704c459416a9dca5b4fbc3fd9b3bce087ad19b5b5f26c5e7055defab135b0"
Sep 18 14:21:08.634506 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.634401    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2704c459416a9dca5b4fbc3fd9b3bce087ad19b5b5f26c5e7055defab135b0"} err="failed to get container status \"7c2704c459416a9dca5b4fbc3fd9b3bce087ad19b5b5f26c5e7055defab135b0\": rpc error: code = NotFound desc = could not find container \"7c2704c459416a9dca5b4fbc3fd9b3bce087ad19b5b5f26c5e7055defab135b0\": container with ID starting with 7c2704c459416a9dca5b4fbc3fd9b3bce087ad19b5b5f26c5e7055defab135b0 not found: ID does not exist"
Sep 18 14:21:08.634506 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.634420    2572 scope.go:117] "RemoveContainer" containerID="a48fa95309a81859284f89df84a5867304c1288b64633e6c8cebd08e39cd82aa"
Sep 18 14:21:08.642287 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.642270    2572 scope.go:117] "RemoveContainer" containerID="a48fa95309a81859284f89df84a5867304c1288b64633e6c8cebd08e39cd82aa"
Sep 18 14:21:08.642565 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:21:08.642545    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48fa95309a81859284f89df84a5867304c1288b64633e6c8cebd08e39cd82aa\": container with ID starting with a48fa95309a81859284f89df84a5867304c1288b64633e6c8cebd08e39cd82aa not found: ID does not exist" containerID="a48fa95309a81859284f89df84a5867304c1288b64633e6c8cebd08e39cd82aa"
Sep 18 14:21:08.642642 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.642570    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48fa95309a81859284f89df84a5867304c1288b64633e6c8cebd08e39cd82aa"} err="failed to get container status \"a48fa95309a81859284f89df84a5867304c1288b64633e6c8cebd08e39cd82aa\": rpc error: code = NotFound desc = could not find container \"a48fa95309a81859284f89df84a5867304c1288b64633e6c8cebd08e39cd82aa\": container with ID starting with a48fa95309a81859284f89df84a5867304c1288b64633e6c8cebd08e39cd82aa not found: ID does not exist"
Sep 18 14:21:08.642642 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.642586    2572 scope.go:117] "RemoveContainer" containerID="34b9031ab85329339063764470497623c9057345bdfe99a71c2981903c1099a0"
Sep 18 14:21:08.649603 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.649557    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"]
Sep 18 14:21:08.650498 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.650489    2572 scope.go:117] "RemoveContainer" containerID="34b9031ab85329339063764470497623c9057345bdfe99a71c2981903c1099a0"
Sep 18 14:21:08.650922 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:21:08.650820    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b9031ab85329339063764470497623c9057345bdfe99a71c2981903c1099a0\": container with ID starting with 34b9031ab85329339063764470497623c9057345bdfe99a71c2981903c1099a0 not found: ID does not exist" containerID="34b9031ab85329339063764470497623c9057345bdfe99a71c2981903c1099a0"
Sep 18 14:21:08.650922 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.650853    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b9031ab85329339063764470497623c9057345bdfe99a71c2981903c1099a0"} err="failed to get container status \"34b9031ab85329339063764470497623c9057345bdfe99a71c2981903c1099a0\": rpc error: code = NotFound desc = could not find container \"34b9031ab85329339063764470497623c9057345bdfe99a71c2981903c1099a0\": container with ID starting with 34b9031ab85329339063764470497623c9057345bdfe99a71c2981903c1099a0 not found: ID does not exist"
Sep 18 14:21:08.653684 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.653664    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a035eb-7997b5d5dc-g4mm4"]
Sep 18 14:21:08.662698 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.662679    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"]
Sep 18 14:21:08.666959 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.666941    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a035eb-76f8c8765d-wqkt7"]
Sep 18 14:21:08.676620 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.676601    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"]
Sep 18 14:21:08.682508 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:08.682487    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a035eb-68789dcc8f-sk4jq"]
Sep 18 14:21:10.197367 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:10.197335    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00" path="/var/lib/kubelet/pods/29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00/volumes"
Sep 18 14:21:10.197801 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:10.197781    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5a51719-3fee-461f-b00b-e5d44d2b4c19" path="/var/lib/kubelet/pods/d5a51719-3fee-461f-b00b-e5d44d2b4c19/volumes"
Sep 18 14:21:10.198138 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:10.198126    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3bcf09e-7a12-479d-9369-07ee07d681a2" path="/var/lib/kubelet/pods/e3bcf09e-7a12-479d-9369-07ee07d681a2/volumes"
Sep 18 14:21:44.044456 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.044370    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"]
Sep 18 14:21:44.045055 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.044627    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl" podUID="976fdf61-e598-4e60-b736-fc42cea248e6" containerName="maas-api" containerID="cri-o://e24f110b57f5465fd97f1db2d2bbecdf7011641d4a7ab36733f1c19bfd01e2ce" gracePeriod=30
Sep 18 14:21:44.302637 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.302577    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:21:44.386330 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.386299    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffzpf\" (UniqueName: \"kubernetes.io/projected/976fdf61-e598-4e60-b736-fc42cea248e6-kube-api-access-ffzpf\") pod \"976fdf61-e598-4e60-b736-fc42cea248e6\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") "
Sep 18 14:21:44.386491 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.386342    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976fdf61-e598-4e60-b736-fc42cea248e6-maas-api-tls\") pod \"976fdf61-e598-4e60-b736-fc42cea248e6\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") "
Sep 18 14:21:44.386491 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.386367    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/976fdf61-e598-4e60-b736-fc42cea248e6-metrics-certs\") pod \"976fdf61-e598-4e60-b736-fc42cea248e6\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") "
Sep 18 14:21:44.386491 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.386413    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/976fdf61-e598-4e60-b736-fc42cea248e6-openshift-service-ca\") pod \"976fdf61-e598-4e60-b736-fc42cea248e6\" (UID: \"976fdf61-e598-4e60-b736-fc42cea248e6\") "
Sep 18 14:21:44.386846 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.386817    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976fdf61-e598-4e60-b736-fc42cea248e6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "976fdf61-e598-4e60-b736-fc42cea248e6" (UID: "976fdf61-e598-4e60-b736-fc42cea248e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:21:44.388569 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.388517    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976fdf61-e598-4e60-b736-fc42cea248e6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "976fdf61-e598-4e60-b736-fc42cea248e6" (UID: "976fdf61-e598-4e60-b736-fc42cea248e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:21:44.388666 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.388571    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976fdf61-e598-4e60-b736-fc42cea248e6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "976fdf61-e598-4e60-b736-fc42cea248e6" (UID: "976fdf61-e598-4e60-b736-fc42cea248e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:21:44.388666 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.388612    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976fdf61-e598-4e60-b736-fc42cea248e6-kube-api-access-ffzpf" (OuterVolumeSpecName: "kube-api-access-ffzpf") pod "976fdf61-e598-4e60-b736-fc42cea248e6" (UID: "976fdf61-e598-4e60-b736-fc42cea248e6"). InnerVolumeSpecName "kube-api-access-ffzpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:21:44.487595 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.487566    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/976fdf61-e598-4e60-b736-fc42cea248e6-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:21:44.487595 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.487593    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/976fdf61-e598-4e60-b736-fc42cea248e6-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:21:44.487782 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.487608    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffzpf\" (UniqueName: \"kubernetes.io/projected/976fdf61-e598-4e60-b736-fc42cea248e6-kube-api-access-ffzpf\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:21:44.487782 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.487621    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976fdf61-e598-4e60-b736-fc42cea248e6-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:21:44.828401 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.828367    2572 generic.go:356] "Generic (PLEG): container finished" podID="976fdf61-e598-4e60-b736-fc42cea248e6" containerID="e24f110b57f5465fd97f1db2d2bbecdf7011641d4a7ab36733f1c19bfd01e2ce" exitCode=0
Sep 18 14:21:44.828618 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.828415    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"
Sep 18 14:21:44.828618 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.828456    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl" event={"ID":"976fdf61-e598-4e60-b736-fc42cea248e6","Type":"ContainerDied","Data":"e24f110b57f5465fd97f1db2d2bbecdf7011641d4a7ab36733f1c19bfd01e2ce"}
Sep 18 14:21:44.828618 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.828510    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl" event={"ID":"976fdf61-e598-4e60-b736-fc42cea248e6","Type":"ContainerDied","Data":"fdf18ea9f480fb787b44c953f069023cfecdb974118b54c6ccccf0b7102bdd14"}
Sep 18 14:21:44.828618 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.828556    2572 scope.go:117] "RemoveContainer" containerID="e24f110b57f5465fd97f1db2d2bbecdf7011641d4a7ab36733f1c19bfd01e2ce"
Sep 18 14:21:44.839052 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.839027    2572 scope.go:117] "RemoveContainer" containerID="e24f110b57f5465fd97f1db2d2bbecdf7011641d4a7ab36733f1c19bfd01e2ce"
Sep 18 14:21:44.839320 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:21:44.839303    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24f110b57f5465fd97f1db2d2bbecdf7011641d4a7ab36733f1c19bfd01e2ce\": container with ID starting with e24f110b57f5465fd97f1db2d2bbecdf7011641d4a7ab36733f1c19bfd01e2ce not found: ID does not exist" containerID="e24f110b57f5465fd97f1db2d2bbecdf7011641d4a7ab36733f1c19bfd01e2ce"
Sep 18 14:21:44.839368 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.839327    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24f110b57f5465fd97f1db2d2bbecdf7011641d4a7ab36733f1c19bfd01e2ce"} err="failed to get container status \"e24f110b57f5465fd97f1db2d2bbecdf7011641d4a7ab36733f1c19bfd01e2ce\": rpc error: code = NotFound desc = could not find container \"e24f110b57f5465fd97f1db2d2bbecdf7011641d4a7ab36733f1c19bfd01e2ce\": container with ID starting with e24f110b57f5465fd97f1db2d2bbecdf7011641d4a7ab36733f1c19bfd01e2ce not found: ID does not exist"
Sep 18 14:21:44.852109 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.852079    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"]
Sep 18 14:21:44.857127 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:44.857104    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d634d2-5bdbfd4c65-ztvrl"]
Sep 18 14:21:46.198229 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:46.198195    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="976fdf61-e598-4e60-b736-fc42cea248e6" path="/var/lib/kubelet/pods/976fdf61-e598-4e60-b736-fc42cea248e6/volumes"
Sep 18 14:21:55.541251 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.541211    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58db965b5-wscbg"]
Sep 18 14:21:55.541773 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.541465    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58db965b5-wscbg" podUID="6fe90d56-762e-42f2-be8e-eaa5444bed8d" containerName="manager" containerID="cri-o://ce106636041b21ca5c70615c2ea886ae609461a7df1005d102f39e2a6427ece0" gracePeriod=30
Sep 18 14:21:55.785640 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.785616    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58db965b5-wscbg"
Sep 18 14:21:55.888338 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.888255    2572 generic.go:356] "Generic (PLEG): container finished" podID="6fe90d56-762e-42f2-be8e-eaa5444bed8d" containerID="ce106636041b21ca5c70615c2ea886ae609461a7df1005d102f39e2a6427ece0" exitCode=0
Sep 18 14:21:55.888338 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.888295    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58db965b5-wscbg"
Sep 18 14:21:55.888338 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.888317    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58db965b5-wscbg" event={"ID":"6fe90d56-762e-42f2-be8e-eaa5444bed8d","Type":"ContainerDied","Data":"ce106636041b21ca5c70615c2ea886ae609461a7df1005d102f39e2a6427ece0"}
Sep 18 14:21:55.888566 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.888356    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58db965b5-wscbg" event={"ID":"6fe90d56-762e-42f2-be8e-eaa5444bed8d","Type":"ContainerDied","Data":"c6424f30e8fb95e5ea89f0f45f890c94c23f5a6e7615b140f266e8328791643f"}
Sep 18 14:21:55.888566 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.888381    2572 scope.go:117] "RemoveContainer" containerID="ce106636041b21ca5c70615c2ea886ae609461a7df1005d102f39e2a6427ece0"
Sep 18 14:21:55.892504 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.892478    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fe90d56-762e-42f2-be8e-eaa5444bed8d-webhook-cert\") pod \"6fe90d56-762e-42f2-be8e-eaa5444bed8d\" (UID: \"6fe90d56-762e-42f2-be8e-eaa5444bed8d\") "
Sep 18 14:21:55.892618 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.892542    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvn52\" (UniqueName: \"kubernetes.io/projected/6fe90d56-762e-42f2-be8e-eaa5444bed8d-kube-api-access-cvn52\") pod \"6fe90d56-762e-42f2-be8e-eaa5444bed8d\" (UID: \"6fe90d56-762e-42f2-be8e-eaa5444bed8d\") "
Sep 18 14:21:55.892618 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.892584    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe90d56-762e-42f2-be8e-eaa5444bed8d-metrics-certs\") pod \"6fe90d56-762e-42f2-be8e-eaa5444bed8d\" (UID: \"6fe90d56-762e-42f2-be8e-eaa5444bed8d\") "
Sep 18 14:21:55.894806 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.894782    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fe90d56-762e-42f2-be8e-eaa5444bed8d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6fe90d56-762e-42f2-be8e-eaa5444bed8d" (UID: "6fe90d56-762e-42f2-be8e-eaa5444bed8d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:21:55.894914 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.894857    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fe90d56-762e-42f2-be8e-eaa5444bed8d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6fe90d56-762e-42f2-be8e-eaa5444bed8d" (UID: "6fe90d56-762e-42f2-be8e-eaa5444bed8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:21:55.894956 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.894924    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe90d56-762e-42f2-be8e-eaa5444bed8d-kube-api-access-cvn52" (OuterVolumeSpecName: "kube-api-access-cvn52") pod "6fe90d56-762e-42f2-be8e-eaa5444bed8d" (UID: "6fe90d56-762e-42f2-be8e-eaa5444bed8d"). InnerVolumeSpecName "kube-api-access-cvn52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:21:55.898303 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.898284    2572 scope.go:117] "RemoveContainer" containerID="ce106636041b21ca5c70615c2ea886ae609461a7df1005d102f39e2a6427ece0"
Sep 18 14:21:55.898590 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:21:55.898571    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce106636041b21ca5c70615c2ea886ae609461a7df1005d102f39e2a6427ece0\": container with ID starting with ce106636041b21ca5c70615c2ea886ae609461a7df1005d102f39e2a6427ece0 not found: ID does not exist" containerID="ce106636041b21ca5c70615c2ea886ae609461a7df1005d102f39e2a6427ece0"
Sep 18 14:21:55.898657 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.898603    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce106636041b21ca5c70615c2ea886ae609461a7df1005d102f39e2a6427ece0"} err="failed to get container status \"ce106636041b21ca5c70615c2ea886ae609461a7df1005d102f39e2a6427ece0\": rpc error: code = NotFound desc = could not find container \"ce106636041b21ca5c70615c2ea886ae609461a7df1005d102f39e2a6427ece0\": container with ID starting with ce106636041b21ca5c70615c2ea886ae609461a7df1005d102f39e2a6427ece0 not found: ID does not exist"
Sep 18 14:21:55.993420 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.993395    2572 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fe90d56-762e-42f2-be8e-eaa5444bed8d-webhook-cert\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:21:55.993420 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.993417    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvn52\" (UniqueName: \"kubernetes.io/projected/6fe90d56-762e-42f2-be8e-eaa5444bed8d-kube-api-access-cvn52\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:21:55.993628 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:55.993427    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe90d56-762e-42f2-be8e-eaa5444bed8d-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:21:56.214448 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:56.214409    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58db965b5-wscbg"]
Sep 18 14:21:56.218635 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:56.218609    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58db965b5-wscbg"]
Sep 18 14:21:58.197835 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:21:58.197800    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fe90d56-762e-42f2-be8e-eaa5444bed8d" path="/var/lib/kubelet/pods/6fe90d56-762e-42f2-be8e-eaa5444bed8d/volumes"
Sep 18 14:22:26.625103 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:26.625071    2572 scope.go:117] "RemoveContainer" containerID="f2ef3b14be4ef51c27483af6f102ad22c1b8fa7da57c3b6cc42cea6ee3249d61"
Sep 18 14:22:26.633620 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:26.633596    2572 scope.go:117] "RemoveContainer" containerID="1dcbc5c9e51c3f29d928a60daa5b28780a942301daa951e7fbdbd4128065f78a"
Sep 18 14:22:57.009917 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.009881    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58db965b5-5nkw6"]
Sep 18 14:22:57.010308 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010230    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00" containerName="payload-processing"
Sep 18 14:22:57.010308 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010242    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00" containerName="payload-processing"
Sep 18 14:22:57.010308 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010254    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a51719-3fee-461f-b00b-e5d44d2b4c19" containerName="maas-api"
Sep 18 14:22:57.010308 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010260    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a51719-3fee-461f-b00b-e5d44d2b4c19" containerName="maas-api"
Sep 18 14:22:57.010308 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010267    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4b0ad44-88aa-4768-a051-82973acaf233" containerName="payload-processing"
Sep 18 14:22:57.010308 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010273    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4b0ad44-88aa-4768-a051-82973acaf233" containerName="payload-processing"
Sep 18 14:22:57.010308 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010280    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="580cab1d-6aeb-4635-ae03-7cbdf5b4a64d" containerName="maas-api"
Sep 18 14:22:57.010308 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010285    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="580cab1d-6aeb-4635-ae03-7cbdf5b4a64d" containerName="maas-api"
Sep 18 14:22:57.010308 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010290    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc6093d-2532-47a6-ac15-4311a3ada407" containerName="payload-pre-processing"
Sep 18 14:22:57.010308 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010295    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc6093d-2532-47a6-ac15-4311a3ada407" containerName="payload-pre-processing"
Sep 18 14:22:57.010308 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010303    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976fdf61-e598-4e60-b736-fc42cea248e6" containerName="maas-api"
Sep 18 14:22:57.010308 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010308    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="976fdf61-e598-4e60-b736-fc42cea248e6" containerName="maas-api"
Sep 18 14:22:57.010308 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010315    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe90d56-762e-42f2-be8e-eaa5444bed8d" containerName="manager"
Sep 18 14:22:57.010903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010320    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe90d56-762e-42f2-be8e-eaa5444bed8d" containerName="manager"
Sep 18 14:22:57.010903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010330    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be075da4-9e12-4bcd-a5e9-38479e02a609" containerName="payload-pre-processing"
Sep 18 14:22:57.010903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010335    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="be075da4-9e12-4bcd-a5e9-38479e02a609" containerName="payload-pre-processing"
Sep 18 14:22:57.010903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010341    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3bcf09e-7a12-479d-9369-07ee07d681a2" containerName="payload-pre-processing"
Sep 18 14:22:57.010903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010346    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3bcf09e-7a12-479d-9369-07ee07d681a2" containerName="payload-pre-processing"
Sep 18 14:22:57.010903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010402    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="580cab1d-6aeb-4635-ae03-7cbdf5b4a64d" containerName="maas-api"
Sep 18 14:22:57.010903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010409    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a51719-3fee-461f-b00b-e5d44d2b4c19" containerName="maas-api"
Sep 18 14:22:57.010903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010414    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3bcf09e-7a12-479d-9369-07ee07d681a2" containerName="payload-pre-processing"
Sep 18 14:22:57.010903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010422    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe90d56-762e-42f2-be8e-eaa5444bed8d" containerName="manager"
Sep 18 14:22:57.010903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010426    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="be075da4-9e12-4bcd-a5e9-38479e02a609" containerName="payload-pre-processing"
Sep 18 14:22:57.010903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010432    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="976fdf61-e598-4e60-b736-fc42cea248e6" containerName="maas-api"
Sep 18 14:22:57.010903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010437    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc6093d-2532-47a6-ac15-4311a3ada407" containerName="payload-pre-processing"
Sep 18 14:22:57.010903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010442    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="29b372d0-3f2e-43bd-a4b2-f3d4a0e22b00" containerName="payload-processing"
Sep 18 14:22:57.010903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.010448    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4b0ad44-88aa-4768-a051-82973acaf233" containerName="payload-processing"
Sep 18 14:22:57.013447 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.013432    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58db965b5-5nkw6"
Sep 18 14:22:57.016581 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.016560    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nhzv4\""
Sep 18 14:22:57.016581 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.016566    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 14:22:57.016767 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.016591    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 14:22:57.016767 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.016591    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 14:22:57.024961 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.024942    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58db965b5-5nkw6"]
Sep 18 14:22:57.131141 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.131113    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efac31fe-e202-432d-a2df-3cb61297a88c-metrics-certs\") pod \"maas-controller-58db965b5-5nkw6\" (UID: \"efac31fe-e202-432d-a2df-3cb61297a88c\") " pod="opendatahub/maas-controller-58db965b5-5nkw6"
Sep 18 14:22:57.131280 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.131206    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tgts\" (UniqueName: \"kubernetes.io/projected/efac31fe-e202-432d-a2df-3cb61297a88c-kube-api-access-4tgts\") pod \"maas-controller-58db965b5-5nkw6\" (UID: \"efac31fe-e202-432d-a2df-3cb61297a88c\") " pod="opendatahub/maas-controller-58db965b5-5nkw6"
Sep 18 14:22:57.131280 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.131227    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efac31fe-e202-432d-a2df-3cb61297a88c-webhook-cert\") pod \"maas-controller-58db965b5-5nkw6\" (UID: \"efac31fe-e202-432d-a2df-3cb61297a88c\") " pod="opendatahub/maas-controller-58db965b5-5nkw6"
Sep 18 14:22:57.231911 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.231882    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tgts\" (UniqueName: \"kubernetes.io/projected/efac31fe-e202-432d-a2df-3cb61297a88c-kube-api-access-4tgts\") pod \"maas-controller-58db965b5-5nkw6\" (UID: \"efac31fe-e202-432d-a2df-3cb61297a88c\") " pod="opendatahub/maas-controller-58db965b5-5nkw6"
Sep 18 14:22:57.232086 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.231919    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efac31fe-e202-432d-a2df-3cb61297a88c-webhook-cert\") pod \"maas-controller-58db965b5-5nkw6\" (UID: \"efac31fe-e202-432d-a2df-3cb61297a88c\") " pod="opendatahub/maas-controller-58db965b5-5nkw6"
Sep 18 14:22:57.232086 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.231937    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efac31fe-e202-432d-a2df-3cb61297a88c-metrics-certs\") pod \"maas-controller-58db965b5-5nkw6\" (UID: \"efac31fe-e202-432d-a2df-3cb61297a88c\") " pod="opendatahub/maas-controller-58db965b5-5nkw6"
Sep 18 14:22:57.234451 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.234429    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efac31fe-e202-432d-a2df-3cb61297a88c-metrics-certs\") pod \"maas-controller-58db965b5-5nkw6\" (UID: \"efac31fe-e202-432d-a2df-3cb61297a88c\") " pod="opendatahub/maas-controller-58db965b5-5nkw6"
Sep 18 14:22:57.234526 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.234429    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efac31fe-e202-432d-a2df-3cb61297a88c-webhook-cert\") pod \"maas-controller-58db965b5-5nkw6\" (UID: \"efac31fe-e202-432d-a2df-3cb61297a88c\") " pod="opendatahub/maas-controller-58db965b5-5nkw6"
Sep 18 14:22:57.240470 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.240445    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tgts\" (UniqueName: \"kubernetes.io/projected/efac31fe-e202-432d-a2df-3cb61297a88c-kube-api-access-4tgts\") pod \"maas-controller-58db965b5-5nkw6\" (UID: \"efac31fe-e202-432d-a2df-3cb61297a88c\") " pod="opendatahub/maas-controller-58db965b5-5nkw6"
Sep 18 14:22:57.323797 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.323713    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58db965b5-5nkw6"
Sep 18 14:22:57.593002 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.592971    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58db965b5-5nkw6"]
Sep 18 14:22:57.594023 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:22:57.593994    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefac31fe_e202_432d_a2df_3cb61297a88c.slice/crio-ecf9f42bf2fcc6caa7eda41c32558c5a329caef68db8649d99db806fe637b10d WatchSource:0}: Error finding container ecf9f42bf2fcc6caa7eda41c32558c5a329caef68db8649d99db806fe637b10d: Status 404 returned error can't find the container with id ecf9f42bf2fcc6caa7eda41c32558c5a329caef68db8649d99db806fe637b10d
Sep 18 14:22:57.595568 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:57.595550    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:22:58.225591 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:58.225528    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58db965b5-5nkw6" event={"ID":"efac31fe-e202-432d-a2df-3cb61297a88c","Type":"ContainerStarted","Data":"ecf9f42bf2fcc6caa7eda41c32558c5a329caef68db8649d99db806fe637b10d"}
Sep 18 14:22:59.232917 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:59.232880    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58db965b5-5nkw6" event={"ID":"efac31fe-e202-432d-a2df-3cb61297a88c","Type":"ContainerStarted","Data":"3c9cf9a0987c49e7fc77c898a37429e07c8ff57ae4f640350e69f4580f9e8996"}
Sep 18 14:22:59.233297 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:59.232939    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58db965b5-5nkw6"
Sep 18 14:22:59.249846 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:22:59.249800    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58db965b5-5nkw6" podStartSLOduration=2.341363838 podStartE2EDuration="3.249774076s" totalImagesPullingTime="908.410238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:22:59.249244937 +0000 UTC m=+2315.184210082" watchObservedRunningTime="2026-09-18 14:22:59.249774076 +0000 UTC m=+2315.184739219"
Sep 18 14:23:10.239834 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:10.239745    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58db965b5-5nkw6"
Sep 18 14:23:17.482932 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:17.482890    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58db965b5-5nkw6"]
Sep 18 14:23:17.483409 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:17.483187    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58db965b5-5nkw6" podUID="efac31fe-e202-432d-a2df-3cb61297a88c" containerName="manager" containerID="cri-o://3c9cf9a0987c49e7fc77c898a37429e07c8ff57ae4f640350e69f4580f9e8996" gracePeriod=30
Sep 18 14:23:17.732590 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:17.732566    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58db965b5-5nkw6"
Sep 18 14:23:17.800980 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:17.800905    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tgts\" (UniqueName: \"kubernetes.io/projected/efac31fe-e202-432d-a2df-3cb61297a88c-kube-api-access-4tgts\") pod \"efac31fe-e202-432d-a2df-3cb61297a88c\" (UID: \"efac31fe-e202-432d-a2df-3cb61297a88c\") "
Sep 18 14:23:17.801116 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:17.800997    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efac31fe-e202-432d-a2df-3cb61297a88c-webhook-cert\") pod \"efac31fe-e202-432d-a2df-3cb61297a88c\" (UID: \"efac31fe-e202-432d-a2df-3cb61297a88c\") "
Sep 18 14:23:17.801116 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:17.801064    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efac31fe-e202-432d-a2df-3cb61297a88c-metrics-certs\") pod \"efac31fe-e202-432d-a2df-3cb61297a88c\" (UID: \"efac31fe-e202-432d-a2df-3cb61297a88c\") "
Sep 18 14:23:17.802930 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:17.802895    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efac31fe-e202-432d-a2df-3cb61297a88c-kube-api-access-4tgts" (OuterVolumeSpecName: "kube-api-access-4tgts") pod "efac31fe-e202-432d-a2df-3cb61297a88c" (UID: "efac31fe-e202-432d-a2df-3cb61297a88c"). InnerVolumeSpecName "kube-api-access-4tgts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:23:17.803070 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:17.803051    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efac31fe-e202-432d-a2df-3cb61297a88c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "efac31fe-e202-432d-a2df-3cb61297a88c" (UID: "efac31fe-e202-432d-a2df-3cb61297a88c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:23:17.803128 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:17.803077    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efac31fe-e202-432d-a2df-3cb61297a88c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "efac31fe-e202-432d-a2df-3cb61297a88c" (UID: "efac31fe-e202-432d-a2df-3cb61297a88c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:23:17.902474 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:17.902444    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efac31fe-e202-432d-a2df-3cb61297a88c-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:23:17.902474 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:17.902468    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tgts\" (UniqueName: \"kubernetes.io/projected/efac31fe-e202-432d-a2df-3cb61297a88c-kube-api-access-4tgts\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:23:17.902474 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:17.902478    2572 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efac31fe-e202-432d-a2df-3cb61297a88c-webhook-cert\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:23:18.329519 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:18.329488    2572 generic.go:356] "Generic (PLEG): container finished" podID="efac31fe-e202-432d-a2df-3cb61297a88c" containerID="3c9cf9a0987c49e7fc77c898a37429e07c8ff57ae4f640350e69f4580f9e8996" exitCode=0
Sep 18 14:23:18.329700 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:18.329545    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58db965b5-5nkw6"
Sep 18 14:23:18.329700 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:18.329565    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58db965b5-5nkw6" event={"ID":"efac31fe-e202-432d-a2df-3cb61297a88c","Type":"ContainerDied","Data":"3c9cf9a0987c49e7fc77c898a37429e07c8ff57ae4f640350e69f4580f9e8996"}
Sep 18 14:23:18.329700 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:18.329689    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58db965b5-5nkw6" event={"ID":"efac31fe-e202-432d-a2df-3cb61297a88c","Type":"ContainerDied","Data":"ecf9f42bf2fcc6caa7eda41c32558c5a329caef68db8649d99db806fe637b10d"}
Sep 18 14:23:18.329820 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:18.329711    2572 scope.go:117] "RemoveContainer" containerID="3c9cf9a0987c49e7fc77c898a37429e07c8ff57ae4f640350e69f4580f9e8996"
Sep 18 14:23:18.338604 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:18.338574    2572 scope.go:117] "RemoveContainer" containerID="3c9cf9a0987c49e7fc77c898a37429e07c8ff57ae4f640350e69f4580f9e8996"
Sep 18 14:23:18.338924 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:23:18.338893    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c9cf9a0987c49e7fc77c898a37429e07c8ff57ae4f640350e69f4580f9e8996\": container with ID starting with 3c9cf9a0987c49e7fc77c898a37429e07c8ff57ae4f640350e69f4580f9e8996 not found: ID does not exist" containerID="3c9cf9a0987c49e7fc77c898a37429e07c8ff57ae4f640350e69f4580f9e8996"
Sep 18 14:23:18.339030 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:18.338933    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c9cf9a0987c49e7fc77c898a37429e07c8ff57ae4f640350e69f4580f9e8996"} err="failed to get container status \"3c9cf9a0987c49e7fc77c898a37429e07c8ff57ae4f640350e69f4580f9e8996\": rpc error: code = NotFound desc = could not find container \"3c9cf9a0987c49e7fc77c898a37429e07c8ff57ae4f640350e69f4580f9e8996\": container with ID starting with 3c9cf9a0987c49e7fc77c898a37429e07c8ff57ae4f640350e69f4580f9e8996 not found: ID does not exist"
Sep 18 14:23:18.350006 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:18.349975    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58db965b5-5nkw6"]
Sep 18 14:23:18.356065 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:18.356042    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58db965b5-5nkw6"]
Sep 18 14:23:19.299869 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.299837    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58db965b5-nxvlz"]
Sep 18 14:23:19.300257 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.300244    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efac31fe-e202-432d-a2df-3cb61297a88c" containerName="manager"
Sep 18 14:23:19.300305 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.300258    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="efac31fe-e202-432d-a2df-3cb61297a88c" containerName="manager"
Sep 18 14:23:19.300336 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.300326    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="efac31fe-e202-432d-a2df-3cb61297a88c" containerName="manager"
Sep 18 14:23:19.304917 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.304886    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58db965b5-nxvlz"
Sep 18 14:23:19.307265 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.307239    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 14:23:19.307557 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.307542    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 14:23:19.308325 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.308306    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nhzv4\""
Sep 18 14:23:19.308418 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.308323    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 14:23:19.316607 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.315493    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58db965b5-nxvlz"]
Sep 18 14:23:19.415914 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.415881    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f2ed63-c33e-4ae6-a475-ce090e5215e5-metrics-certs\") pod \"maas-controller-58db965b5-nxvlz\" (UID: \"63f2ed63-c33e-4ae6-a475-ce090e5215e5\") " pod="opendatahub/maas-controller-58db965b5-nxvlz"
Sep 18 14:23:19.416090 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.415949    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmtpv\" (UniqueName: \"kubernetes.io/projected/63f2ed63-c33e-4ae6-a475-ce090e5215e5-kube-api-access-jmtpv\") pod \"maas-controller-58db965b5-nxvlz\" (UID: \"63f2ed63-c33e-4ae6-a475-ce090e5215e5\") " pod="opendatahub/maas-controller-58db965b5-nxvlz"
Sep 18 14:23:19.416090 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.415965    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63f2ed63-c33e-4ae6-a475-ce090e5215e5-webhook-cert\") pod \"maas-controller-58db965b5-nxvlz\" (UID: \"63f2ed63-c33e-4ae6-a475-ce090e5215e5\") " pod="opendatahub/maas-controller-58db965b5-nxvlz"
Sep 18 14:23:19.516775 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.516746    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63f2ed63-c33e-4ae6-a475-ce090e5215e5-webhook-cert\") pod \"maas-controller-58db965b5-nxvlz\" (UID: \"63f2ed63-c33e-4ae6-a475-ce090e5215e5\") " pod="opendatahub/maas-controller-58db965b5-nxvlz"
Sep 18 14:23:19.516895 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.516785    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f2ed63-c33e-4ae6-a475-ce090e5215e5-metrics-certs\") pod \"maas-controller-58db965b5-nxvlz\" (UID: \"63f2ed63-c33e-4ae6-a475-ce090e5215e5\") " pod="opendatahub/maas-controller-58db965b5-nxvlz"
Sep 18 14:23:19.516895 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.516852    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmtpv\" (UniqueName: \"kubernetes.io/projected/63f2ed63-c33e-4ae6-a475-ce090e5215e5-kube-api-access-jmtpv\") pod \"maas-controller-58db965b5-nxvlz\" (UID: \"63f2ed63-c33e-4ae6-a475-ce090e5215e5\") " pod="opendatahub/maas-controller-58db965b5-nxvlz"
Sep 18 14:23:19.519391 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.519365    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63f2ed63-c33e-4ae6-a475-ce090e5215e5-webhook-cert\") pod \"maas-controller-58db965b5-nxvlz\" (UID: \"63f2ed63-c33e-4ae6-a475-ce090e5215e5\") " pod="opendatahub/maas-controller-58db965b5-nxvlz"
Sep 18 14:23:19.519502 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.519412    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f2ed63-c33e-4ae6-a475-ce090e5215e5-metrics-certs\") pod \"maas-controller-58db965b5-nxvlz\" (UID: \"63f2ed63-c33e-4ae6-a475-ce090e5215e5\") " pod="opendatahub/maas-controller-58db965b5-nxvlz"
Sep 18 14:23:19.525461 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.525437    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmtpv\" (UniqueName: \"kubernetes.io/projected/63f2ed63-c33e-4ae6-a475-ce090e5215e5-kube-api-access-jmtpv\") pod \"maas-controller-58db965b5-nxvlz\" (UID: \"63f2ed63-c33e-4ae6-a475-ce090e5215e5\") " pod="opendatahub/maas-controller-58db965b5-nxvlz"
Sep 18 14:23:19.622067 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.622007    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58db965b5-nxvlz"
Sep 18 14:23:19.892830 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:19.892801    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58db965b5-nxvlz"]
Sep 18 14:23:19.894260 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:23:19.894226    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f2ed63_c33e_4ae6_a475_ce090e5215e5.slice/crio-9cb6f89597af1344b110552e11e0de83641ca0c275a800456b340f9ebc70cdea WatchSource:0}: Error finding container 9cb6f89597af1344b110552e11e0de83641ca0c275a800456b340f9ebc70cdea: Status 404 returned error can't find the container with id 9cb6f89597af1344b110552e11e0de83641ca0c275a800456b340f9ebc70cdea
Sep 18 14:23:20.197478 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:20.197443    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efac31fe-e202-432d-a2df-3cb61297a88c" path="/var/lib/kubelet/pods/efac31fe-e202-432d-a2df-3cb61297a88c/volumes"
Sep 18 14:23:20.342827 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:20.342790    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58db965b5-nxvlz" event={"ID":"63f2ed63-c33e-4ae6-a475-ce090e5215e5","Type":"ContainerStarted","Data":"fc0b24a34f1d1f308613da307a54172a580c1e91681d54c616c38de5b9db1440"}
Sep 18 14:23:20.343226 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:20.342845    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58db965b5-nxvlz"
Sep 18 14:23:20.343226 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:20.342864    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58db965b5-nxvlz" event={"ID":"63f2ed63-c33e-4ae6-a475-ce090e5215e5","Type":"ContainerStarted","Data":"9cb6f89597af1344b110552e11e0de83641ca0c275a800456b340f9ebc70cdea"}
Sep 18 14:23:20.359799 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:20.359755    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58db965b5-nxvlz" podStartSLOduration=1.010584068 podStartE2EDuration="1.359741181s" totalImagesPullingTime="349.157113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:23:20.359381059 +0000 UTC m=+2336.294346201" watchObservedRunningTime="2026-09-18 14:23:20.359741181 +0000 UTC m=+2336.294706327"
Sep 18 14:23:31.351521 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:23:31.351485    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58db965b5-nxvlz"
Sep 18 14:24:26.404166 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:26.404135    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 14:24:26.412110 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:26.412086    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 14:24:59.462069 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.461987    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"]
Sep 18 14:24:59.462581 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.462205    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5" podUID="7b8d50ee-46b2-4558-8de2-63491740f19e" containerName="manager" containerID="cri-o://63f31dca88b59dca2bcb24520693f339649b58cf0f3e7ededdaf27b6a262992c" gracePeriod=10
Sep 18 14:24:59.703348 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.703324    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"
Sep 18 14:24:59.799187 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.799103    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b8d50ee-46b2-4558-8de2-63491740f19e-extensions-socket-volume\") pod \"7b8d50ee-46b2-4558-8de2-63491740f19e\" (UID: \"7b8d50ee-46b2-4558-8de2-63491740f19e\") "
Sep 18 14:24:59.799187 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.799177    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2jmb\" (UniqueName: \"kubernetes.io/projected/7b8d50ee-46b2-4558-8de2-63491740f19e-kube-api-access-c2jmb\") pod \"7b8d50ee-46b2-4558-8de2-63491740f19e\" (UID: \"7b8d50ee-46b2-4558-8de2-63491740f19e\") "
Sep 18 14:24:59.799465 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.799442    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8d50ee-46b2-4558-8de2-63491740f19e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7b8d50ee-46b2-4558-8de2-63491740f19e" (UID: "7b8d50ee-46b2-4558-8de2-63491740f19e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:24:59.801220 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.801197    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8d50ee-46b2-4558-8de2-63491740f19e-kube-api-access-c2jmb" (OuterVolumeSpecName: "kube-api-access-c2jmb") pod "7b8d50ee-46b2-4558-8de2-63491740f19e" (UID: "7b8d50ee-46b2-4558-8de2-63491740f19e"). InnerVolumeSpecName "kube-api-access-c2jmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:24:59.867213 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.867175    2572 generic.go:356] "Generic (PLEG): container finished" podID="7b8d50ee-46b2-4558-8de2-63491740f19e" containerID="63f31dca88b59dca2bcb24520693f339649b58cf0f3e7ededdaf27b6a262992c" exitCode=0
Sep 18 14:24:59.867398 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.867218    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"
Sep 18 14:24:59.867398 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.867261    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5" event={"ID":"7b8d50ee-46b2-4558-8de2-63491740f19e","Type":"ContainerDied","Data":"63f31dca88b59dca2bcb24520693f339649b58cf0f3e7ededdaf27b6a262992c"}
Sep 18 14:24:59.867398 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.867313    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5" event={"ID":"7b8d50ee-46b2-4558-8de2-63491740f19e","Type":"ContainerDied","Data":"801f7356e575813153b84026aba887527639d81c659c45fff77ce336f9f729c6"}
Sep 18 14:24:59.867398 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.867337    2572 scope.go:117] "RemoveContainer" containerID="63f31dca88b59dca2bcb24520693f339649b58cf0f3e7ededdaf27b6a262992c"
Sep 18 14:24:59.876947 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.876927    2572 scope.go:117] "RemoveContainer" containerID="63f31dca88b59dca2bcb24520693f339649b58cf0f3e7ededdaf27b6a262992c"
Sep 18 14:24:59.877223 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:24:59.877204    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f31dca88b59dca2bcb24520693f339649b58cf0f3e7ededdaf27b6a262992c\": container with ID starting with 63f31dca88b59dca2bcb24520693f339649b58cf0f3e7ededdaf27b6a262992c not found: ID does not exist" containerID="63f31dca88b59dca2bcb24520693f339649b58cf0f3e7ededdaf27b6a262992c"
Sep 18 14:24:59.877289 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.877236    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f31dca88b59dca2bcb24520693f339649b58cf0f3e7ededdaf27b6a262992c"} err="failed to get container status \"63f31dca88b59dca2bcb24520693f339649b58cf0f3e7ededdaf27b6a262992c\": rpc error: code = NotFound desc = could not find container \"63f31dca88b59dca2bcb24520693f339649b58cf0f3e7ededdaf27b6a262992c\": container with ID starting with 63f31dca88b59dca2bcb24520693f339649b58cf0f3e7ededdaf27b6a262992c not found: ID does not exist"
Sep 18 14:24:59.890912 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.890886    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"]
Sep 18 14:24:59.896321 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.896298    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-n9pw5"]
Sep 18 14:24:59.900350 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.900331    2572 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b8d50ee-46b2-4558-8de2-63491740f19e-extensions-socket-volume\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:24:59.900350 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:24:59.900350    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2jmb\" (UniqueName: \"kubernetes.io/projected/7b8d50ee-46b2-4558-8de2-63491740f19e-kube-api-access-c2jmb\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:25:00.197180 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:25:00.197148    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b8d50ee-46b2-4558-8de2-63491740f19e" path="/var/lib/kubelet/pods/7b8d50ee-46b2-4558-8de2-63491740f19e/volumes"
Sep 18 14:26:00.124069 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:00.124031    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w6md7"]
Sep 18 14:26:00.124573 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:00.124371    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b8d50ee-46b2-4558-8de2-63491740f19e" containerName="manager"
Sep 18 14:26:00.124573 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:00.124382    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b8d50ee-46b2-4558-8de2-63491740f19e" containerName="manager"
Sep 18 14:26:00.124573 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:00.124446    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b8d50ee-46b2-4558-8de2-63491740f19e" containerName="manager"
Sep 18 14:26:00.127318 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:00.127303    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w6md7"
Sep 18 14:26:00.130050 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:00.130033    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r7g84\""
Sep 18 14:26:00.143220 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:00.143196    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w6md7"]
Sep 18 14:26:00.215145 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:00.215118    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51c10299-14fe-4b62-9390-22f85fc356e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w6md7\" (UID: \"51c10299-14fe-4b62-9390-22f85fc356e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w6md7"
Sep 18 14:26:00.215278 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:00.215176    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb4tz\" (UniqueName: \"kubernetes.io/projected/51c10299-14fe-4b62-9390-22f85fc356e7-kube-api-access-gb4tz\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w6md7\" (UID: \"51c10299-14fe-4b62-9390-22f85fc356e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w6md7"
Sep 18 14:26:00.316200 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:00.316172    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb4tz\" (UniqueName: \"kubernetes.io/projected/51c10299-14fe-4b62-9390-22f85fc356e7-kube-api-access-gb4tz\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w6md7\" (UID: \"51c10299-14fe-4b62-9390-22f85fc356e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w6md7"
Sep 18 14:26:00.316312 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:00.316231    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51c10299-14fe-4b62-9390-22f85fc356e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w6md7\" (UID: \"51c10299-14fe-4b62-9390-22f85fc356e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w6md7"
Sep 18 14:26:00.316584 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:00.316567    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51c10299-14fe-4b62-9390-22f85fc356e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w6md7\" (UID: \"51c10299-14fe-4b62-9390-22f85fc356e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w6md7"
Sep 18 14:26:00.326208 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:00.326183    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb4tz\" (UniqueName: \"kubernetes.io/projected/51c10299-14fe-4b62-9390-22f85fc356e7-kube-api-access-gb4tz\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w6md7\" (UID: \"51c10299-14fe-4b62-9390-22f85fc356e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w6md7"
Sep 18 14:26:00.437187 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:00.437092    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w6md7"
Sep 18 14:26:00.725507 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:00.725479    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w6md7"]
Sep 18 14:26:00.727572 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:26:00.727520    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c10299_14fe_4b62_9390_22f85fc356e7.slice/crio-14c8b0c69c4916ff35d7cb141dc4a62abc356b7f5ad5345ed5638369865cc10a WatchSource:0}: Error finding container 14c8b0c69c4916ff35d7cb141dc4a62abc356b7f5ad5345ed5638369865cc10a: Status 404 returned error can't find the container with id 14c8b0c69c4916ff35d7cb141dc4a62abc356b7f5ad5345ed5638369865cc10a
Sep 18 14:26:01.185590 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:01.185483    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w6md7" event={"ID":"51c10299-14fe-4b62-9390-22f85fc356e7","Type":"ContainerStarted","Data":"e098f740cd5a838cbc1f56e51bd382d876c55051d695853fc7ccdc790e714715"}
Sep 18 14:26:01.185590 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:01.185556    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w6md7" event={"ID":"51c10299-14fe-4b62-9390-22f85fc356e7","Type":"ContainerStarted","Data":"14c8b0c69c4916ff35d7cb141dc4a62abc356b7f5ad5345ed5638369865cc10a"}
Sep 18 14:26:01.186020 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:01.185595    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w6md7"
Sep 18 14:26:01.226573 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:01.226500    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w6md7" podStartSLOduration=1.226485123 podStartE2EDuration="1.226485123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:26:01.22549379 +0000 UTC m=+2497.160458933" watchObservedRunningTime="2026-09-18 14:26:01.226485123 +0000 UTC m=+2497.161450267"
Sep 18 14:26:12.198467 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:26:12.198440    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w6md7"
Sep 18 14:27:59.998891 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:27:59.998665    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"]
Sep 18 14:28:00.001842 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.001820    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"
Sep 18 14:28:00.014589 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.014558    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"]
Sep 18 14:28:00.084361 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.084322    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c753073d-4ab8-44e9-bf72-6bb34ef7feca-maas-api-tls\") pod \"maas-api-74bfccbffb-n5jvx\" (UID: \"c753073d-4ab8-44e9-bf72-6bb34ef7feca\") " pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"
Sep 18 14:28:00.084562 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.084427    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c753073d-4ab8-44e9-bf72-6bb34ef7feca-openshift-service-ca\") pod \"maas-api-74bfccbffb-n5jvx\" (UID: \"c753073d-4ab8-44e9-bf72-6bb34ef7feca\") " pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"
Sep 18 14:28:00.084562 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.084454    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c753073d-4ab8-44e9-bf72-6bb34ef7feca-metrics-certs\") pod \"maas-api-74bfccbffb-n5jvx\" (UID: \"c753073d-4ab8-44e9-bf72-6bb34ef7feca\") " pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"
Sep 18 14:28:00.084562 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.084479    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hcrh\" (UniqueName: \"kubernetes.io/projected/c753073d-4ab8-44e9-bf72-6bb34ef7feca-kube-api-access-5hcrh\") pod \"maas-api-74bfccbffb-n5jvx\" (UID: \"c753073d-4ab8-44e9-bf72-6bb34ef7feca\") " pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"
Sep 18 14:28:00.185908 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.185856    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c753073d-4ab8-44e9-bf72-6bb34ef7feca-maas-api-tls\") pod \"maas-api-74bfccbffb-n5jvx\" (UID: \"c753073d-4ab8-44e9-bf72-6bb34ef7feca\") " pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"
Sep 18 14:28:00.186072 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.185986    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c753073d-4ab8-44e9-bf72-6bb34ef7feca-openshift-service-ca\") pod \"maas-api-74bfccbffb-n5jvx\" (UID: \"c753073d-4ab8-44e9-bf72-6bb34ef7feca\") " pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"
Sep 18 14:28:00.186072 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.186010    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c753073d-4ab8-44e9-bf72-6bb34ef7feca-metrics-certs\") pod \"maas-api-74bfccbffb-n5jvx\" (UID: \"c753073d-4ab8-44e9-bf72-6bb34ef7feca\") " pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"
Sep 18 14:28:00.186072 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.186040    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hcrh\" (UniqueName: \"kubernetes.io/projected/c753073d-4ab8-44e9-bf72-6bb34ef7feca-kube-api-access-5hcrh\") pod \"maas-api-74bfccbffb-n5jvx\" (UID: \"c753073d-4ab8-44e9-bf72-6bb34ef7feca\") " pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"
Sep 18 14:28:00.186771 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.186745    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c753073d-4ab8-44e9-bf72-6bb34ef7feca-openshift-service-ca\") pod \"maas-api-74bfccbffb-n5jvx\" (UID: \"c753073d-4ab8-44e9-bf72-6bb34ef7feca\") " pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"
Sep 18 14:28:00.188467 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.188445    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c753073d-4ab8-44e9-bf72-6bb34ef7feca-metrics-certs\") pod \"maas-api-74bfccbffb-n5jvx\" (UID: \"c753073d-4ab8-44e9-bf72-6bb34ef7feca\") " pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"
Sep 18 14:28:00.188601 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.188450    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c753073d-4ab8-44e9-bf72-6bb34ef7feca-maas-api-tls\") pod \"maas-api-74bfccbffb-n5jvx\" (UID: \"c753073d-4ab8-44e9-bf72-6bb34ef7feca\") " pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"
Sep 18 14:28:00.194328 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.194300    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hcrh\" (UniqueName: \"kubernetes.io/projected/c753073d-4ab8-44e9-bf72-6bb34ef7feca-kube-api-access-5hcrh\") pod \"maas-api-74bfccbffb-n5jvx\" (UID: \"c753073d-4ab8-44e9-bf72-6bb34ef7feca\") " pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"
Sep 18 14:28:00.313609 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.313516    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"
Sep 18 14:28:00.592195 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.592166    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"]
Sep 18 14:28:00.593601 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:28:00.593568    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc753073d_4ab8_44e9_bf72_6bb34ef7feca.slice/crio-c052e7843809487ef5df7902c519372f285d6e19156a20717a58fb6696b1d878 WatchSource:0}: Error finding container c052e7843809487ef5df7902c519372f285d6e19156a20717a58fb6696b1d878: Status 404 returned error can't find the container with id c052e7843809487ef5df7902c519372f285d6e19156a20717a58fb6696b1d878
Sep 18 14:28:00.595108 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.595086    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:28:00.829165 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:00.829121    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx" event={"ID":"c753073d-4ab8-44e9-bf72-6bb34ef7feca","Type":"ContainerStarted","Data":"c052e7843809487ef5df7902c519372f285d6e19156a20717a58fb6696b1d878"}
Sep 18 14:28:01.835836 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:01.835799    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx" event={"ID":"c753073d-4ab8-44e9-bf72-6bb34ef7feca","Type":"ContainerStarted","Data":"098adc9581e15abece549eb60136e354714a30765f131942e13605420971a86a"}
Sep 18 14:28:01.836323 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:01.835880    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"
Sep 18 14:28:01.855343 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:01.855290    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx" podStartSLOduration=2.305129444 podStartE2EDuration="2.855276675s" totalImagesPullingTime="550.147231ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:28:01.852411759 +0000 UTC m=+2617.787376903" watchObservedRunningTime="2026-09-18 14:28:01.855276675 +0000 UTC m=+2617.790241841"
Sep 18 14:28:07.847034 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:07.847003    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74bfccbffb-n5jvx"
Sep 18 14:28:07.894464 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:07.894420    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"]
Sep 18 14:28:07.894903 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:07.894815    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw" podUID="9d506203-21f4-4793-80dd-14e00744995a" containerName="maas-api" containerID="cri-o://c7d3d8e86d6b74e538b232489a26cfbe2ab7ddf9e33a4d64f8783978b41caee6" gracePeriod=30
Sep 18 14:28:09.148107 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.148084    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 14:28:09.269386 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.269309    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d506203-21f4-4793-80dd-14e00744995a-maas-api-tls\") pod \"9d506203-21f4-4793-80dd-14e00744995a\" (UID: \"9d506203-21f4-4793-80dd-14e00744995a\") "
Sep 18 14:28:09.269386 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.269372    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d506203-21f4-4793-80dd-14e00744995a-openshift-service-ca\") pod \"9d506203-21f4-4793-80dd-14e00744995a\" (UID: \"9d506203-21f4-4793-80dd-14e00744995a\") "
Sep 18 14:28:09.269386 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.269388    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d506203-21f4-4793-80dd-14e00744995a-metrics-certs\") pod \"9d506203-21f4-4793-80dd-14e00744995a\" (UID: \"9d506203-21f4-4793-80dd-14e00744995a\") "
Sep 18 14:28:09.269712 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.269417    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t9dk\" (UniqueName: \"kubernetes.io/projected/9d506203-21f4-4793-80dd-14e00744995a-kube-api-access-2t9dk\") pod \"9d506203-21f4-4793-80dd-14e00744995a\" (UID: \"9d506203-21f4-4793-80dd-14e00744995a\") "
Sep 18 14:28:09.269794 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.269766    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d506203-21f4-4793-80dd-14e00744995a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d506203-21f4-4793-80dd-14e00744995a" (UID: "9d506203-21f4-4793-80dd-14e00744995a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:28:09.271519 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.271494    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d506203-21f4-4793-80dd-14e00744995a-kube-api-access-2t9dk" (OuterVolumeSpecName: "kube-api-access-2t9dk") pod "9d506203-21f4-4793-80dd-14e00744995a" (UID: "9d506203-21f4-4793-80dd-14e00744995a"). InnerVolumeSpecName "kube-api-access-2t9dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:28:09.271737 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.271520    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d506203-21f4-4793-80dd-14e00744995a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9d506203-21f4-4793-80dd-14e00744995a" (UID: "9d506203-21f4-4793-80dd-14e00744995a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:28:09.271737 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.271561    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d506203-21f4-4793-80dd-14e00744995a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d506203-21f4-4793-80dd-14e00744995a" (UID: "9d506203-21f4-4793-80dd-14e00744995a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:28:09.370650 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.370619    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d506203-21f4-4793-80dd-14e00744995a-openshift-service-ca\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:28:09.370650 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.370646    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d506203-21f4-4793-80dd-14e00744995a-metrics-certs\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:28:09.370801 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.370660    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t9dk\" (UniqueName: \"kubernetes.io/projected/9d506203-21f4-4793-80dd-14e00744995a-kube-api-access-2t9dk\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:28:09.370801 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.370673    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d506203-21f4-4793-80dd-14e00744995a-maas-api-tls\") on node \"ip-10-0-128-141.ec2.internal\" DevicePath \"\""
Sep 18 14:28:09.879677 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.879639    2572 generic.go:356] "Generic (PLEG): container finished" podID="9d506203-21f4-4793-80dd-14e00744995a" containerID="c7d3d8e86d6b74e538b232489a26cfbe2ab7ddf9e33a4d64f8783978b41caee6" exitCode=0
Sep 18 14:28:09.879886 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.879712    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"
Sep 18 14:28:09.879886 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.879716    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw" event={"ID":"9d506203-21f4-4793-80dd-14e00744995a","Type":"ContainerDied","Data":"c7d3d8e86d6b74e538b232489a26cfbe2ab7ddf9e33a4d64f8783978b41caee6"}
Sep 18 14:28:09.879886 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.879769    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw" event={"ID":"9d506203-21f4-4793-80dd-14e00744995a","Type":"ContainerDied","Data":"352356ab197d979e985d76617a3fb0d03b0889f28d94669c7d411fc87c60eb3c"}
Sep 18 14:28:09.879886 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.879791    2572 scope.go:117] "RemoveContainer" containerID="c7d3d8e86d6b74e538b232489a26cfbe2ab7ddf9e33a4d64f8783978b41caee6"
Sep 18 14:28:09.889879 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.889864    2572 scope.go:117] "RemoveContainer" containerID="c7d3d8e86d6b74e538b232489a26cfbe2ab7ddf9e33a4d64f8783978b41caee6"
Sep 18 14:28:09.890110 ip-10-0-128-141 kubenswrapper[2572]: E0918 14:28:09.890090    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d3d8e86d6b74e538b232489a26cfbe2ab7ddf9e33a4d64f8783978b41caee6\": container with ID starting with c7d3d8e86d6b74e538b232489a26cfbe2ab7ddf9e33a4d64f8783978b41caee6 not found: ID does not exist" containerID="c7d3d8e86d6b74e538b232489a26cfbe2ab7ddf9e33a4d64f8783978b41caee6"
Sep 18 14:28:09.890170 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.890122    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d3d8e86d6b74e538b232489a26cfbe2ab7ddf9e33a4d64f8783978b41caee6"} err="failed to get container status \"c7d3d8e86d6b74e538b232489a26cfbe2ab7ddf9e33a4d64f8783978b41caee6\": rpc error: code = NotFound desc = could not find container \"c7d3d8e86d6b74e538b232489a26cfbe2ab7ddf9e33a4d64f8783978b41caee6\": container with ID starting with c7d3d8e86d6b74e538b232489a26cfbe2ab7ddf9e33a4d64f8783978b41caee6 not found: ID does not exist"
Sep 18 14:28:09.901210 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.901181    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"]
Sep 18 14:28:09.905435 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:09.905407    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc7cc57dd-7pfjw"]
Sep 18 14:28:10.198733 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:10.198701    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d506203-21f4-4793-80dd-14e00744995a" path="/var/lib/kubelet/pods/9d506203-21f4-4793-80dd-14e00744995a/volumes"
Sep 18 14:28:34.845677 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:34.845643    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-qktrl_1d645000-9b99-4314-bcb1-bad1a334ec2b/manager/0.log"
Sep 18 14:28:34.966714 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:34.966686    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58db965b5-nxvlz_63f2ed63-c33e-4ae6-a475-ce090e5215e5/manager/0.log"
Sep 18 14:28:35.410096 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:35.410063    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz_66150446-024c-4fd4-994e-fa7eeb9c9f5a/copy-manifests/0.log"
Sep 18 14:28:35.416539 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:35.416518    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz_66150446-024c-4fd4-994e-fa7eeb9c9f5a/manager/0.log"
Sep 18 14:28:35.868156 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:35.868129    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-74bfccbffb-n5jvx_c753073d-4ab8-44e9-bf72-6bb34ef7feca/maas-api/0.log"
Sep 18 14:28:37.153518 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:37.153484    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-c5zpq_2e44db4b-0b3c-46af-9d8b-2657f8abad67/manager/0.log"
Sep 18 14:28:37.262758 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:37.262720    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-984kr_057ec4ca-bd80-4f90-a33d-6ab3dc34f6f1/manager/0.log"
Sep 18 14:28:37.371975 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:37.371948    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xnd24_3f9ff756-b015-4591-aa32-dbcc4a797eca/kuadrant-console-plugin/0.log"
Sep 18 14:28:37.495413 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:37.495383    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-w6md7_51c10299-14fe-4b62-9390-22f85fc356e7/manager/0.log"
Sep 18 14:28:38.249284 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:38.249248    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mrcfl_c908b76c-afff-4af9-81bb-be1c7f8df54e/discovery/0.log"
Sep 18 14:28:38.463273 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:38.463229    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849fd8447d-n2lrt_4144f489-44f8-4933-8d38-6c0fea51a78f/kube-auth-proxy/0.log"
Sep 18 14:28:38.975300 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:38.975247    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77578d5fdb-w9ttn_5b1b97a5-adb1-4a5f-89a9-785ea463af4e/router/0.log"
Sep 18 14:28:39.771194 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:39.771161    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-lmm9x_9751eb95-f681-4e01-829f-462338c0a314/main/0.log"
Sep 18 14:28:40.146158 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:40.146065    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-vjvf6_8dd86f94-9b6d-4606-845b-4a6ffdee9e48/main/0.log"
Sep 18 14:28:49.333924 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:49.333880    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-stclq_cbb05c7c-6add-4a50-82c5-ca67901571f7/global-pull-secret-syncer/0.log"
Sep 18 14:28:49.422834 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:49.422803    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kpjj7_2a57abd9-a0ec-4d26-afe6-b74b8cc663a6/konnectivity-agent/0.log"
Sep 18 14:28:49.483877 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:49.483846    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-141.ec2.internal_f61923dd52bddbca0af64d8f40dd2b70/haproxy/0.log"
Sep 18 14:28:53.907862 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:53.907830    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-c5zpq_2e44db4b-0b3c-46af-9d8b-2657f8abad67/manager/0.log"
Sep 18 14:28:53.957084 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:53.957053    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-984kr_057ec4ca-bd80-4f90-a33d-6ab3dc34f6f1/manager/0.log"
Sep 18 14:28:53.986202 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:53.986183    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xnd24_3f9ff756-b015-4591-aa32-dbcc4a797eca/kuadrant-console-plugin/0.log"
Sep 18 14:28:54.034028 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:54.033996    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-w6md7_51c10299-14fe-4b62-9390-22f85fc356e7/manager/0.log"
Sep 18 14:28:55.923666 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:55.923633    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-58q2l_0722ceea-9544-4e73-84f2-aae4fbe4ff98/cluster-monitoring-operator/0.log"
Sep 18 14:28:56.289489 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:56.289455    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n67xl_9843c12f-6b4e-4cf1-b7c4-c042f83b05b6/node-exporter/0.log"
Sep 18 14:28:56.313665 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:56.313638    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n67xl_9843c12f-6b4e-4cf1-b7c4-c042f83b05b6/kube-rbac-proxy/0.log"
Sep 18 14:28:56.336010 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:56.335986    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n67xl_9843c12f-6b4e-4cf1-b7c4-c042f83b05b6/init-textfile/0.log"
Sep 18 14:28:56.852839 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:56.852805    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7dbcfb5c5-np254_0bf1f531-64a6-4adc-92e0-dca725db2aa8/telemeter-client/0.log"
Sep 18 14:28:56.921155 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:56.921124    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7dbcfb5c5-np254_0bf1f531-64a6-4adc-92e0-dca725db2aa8/reload/0.log"
Sep 18 14:28:56.981204 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:56.981165    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7dbcfb5c5-np254_0bf1f531-64a6-4adc-92e0-dca725db2aa8/kube-rbac-proxy/0.log"
Sep 18 14:28:59.233814 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.233781    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/1.log"
Sep 18 14:28:59.241618 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.241583    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-lj48l_24a56b4d-b52e-4af5-b13e-5e9d26a998e8/console-operator/2.log"
Sep 18 14:28:59.740130 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.740088    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"]
Sep 18 14:28:59.740496 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.740484    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d506203-21f4-4793-80dd-14e00744995a" containerName="maas-api"
Sep 18 14:28:59.740629 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.740498    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d506203-21f4-4793-80dd-14e00744995a" containerName="maas-api"
Sep 18 14:28:59.740629 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.740588    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d506203-21f4-4793-80dd-14e00744995a" containerName="maas-api"
Sep 18 14:28:59.743760 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.743742    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:28:59.746007 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.745977    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pkp9v\"/\"kube-root-ca.crt\""
Sep 18 14:28:59.746126 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.746029    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pkp9v\"/\"default-dockercfg-5xwr9\""
Sep 18 14:28:59.746655 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.746638    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pkp9v\"/\"openshift-service-ca.crt\""
Sep 18 14:28:59.751138 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.750716    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"]
Sep 18 14:28:59.807723 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.807679    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a509825-85eb-4d81-999c-94dc220be99d-lib-modules\") pod \"perf-node-gather-daemonset-x4x2q\" (UID: \"8a509825-85eb-4d81-999c-94dc220be99d\") " pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:28:59.807723 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.807723    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a509825-85eb-4d81-999c-94dc220be99d-podres\") pod \"perf-node-gather-daemonset-x4x2q\" (UID: \"8a509825-85eb-4d81-999c-94dc220be99d\") " pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:28:59.807942 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.807742    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jk7t\" (UniqueName: \"kubernetes.io/projected/8a509825-85eb-4d81-999c-94dc220be99d-kube-api-access-9jk7t\") pod \"perf-node-gather-daemonset-x4x2q\" (UID: \"8a509825-85eb-4d81-999c-94dc220be99d\") " pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:28:59.807942 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.807832    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a509825-85eb-4d81-999c-94dc220be99d-proc\") pod \"perf-node-gather-daemonset-x4x2q\" (UID: \"8a509825-85eb-4d81-999c-94dc220be99d\") " pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:28:59.807942 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.807871    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a509825-85eb-4d81-999c-94dc220be99d-sys\") pod \"perf-node-gather-daemonset-x4x2q\" (UID: \"8a509825-85eb-4d81-999c-94dc220be99d\") " pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:28:59.894120 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.894087    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-vtlp8_be82ea7b-c939-4ee3-af33-ec978fe19ba4/download-server/0.log"
Sep 18 14:28:59.909078 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.909047    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a509825-85eb-4d81-999c-94dc220be99d-sys\") pod \"perf-node-gather-daemonset-x4x2q\" (UID: \"8a509825-85eb-4d81-999c-94dc220be99d\") " pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:28:59.909288 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.909105    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a509825-85eb-4d81-999c-94dc220be99d-lib-modules\") pod \"perf-node-gather-daemonset-x4x2q\" (UID: \"8a509825-85eb-4d81-999c-94dc220be99d\") " pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:28:59.909288 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.909127    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a509825-85eb-4d81-999c-94dc220be99d-podres\") pod \"perf-node-gather-daemonset-x4x2q\" (UID: \"8a509825-85eb-4d81-999c-94dc220be99d\") " pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:28:59.909288 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.909147    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jk7t\" (UniqueName: \"kubernetes.io/projected/8a509825-85eb-4d81-999c-94dc220be99d-kube-api-access-9jk7t\") pod \"perf-node-gather-daemonset-x4x2q\" (UID: \"8a509825-85eb-4d81-999c-94dc220be99d\") " pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:28:59.909288 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.909180    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a509825-85eb-4d81-999c-94dc220be99d-sys\") pod \"perf-node-gather-daemonset-x4x2q\" (UID: \"8a509825-85eb-4d81-999c-94dc220be99d\") " pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:28:59.909288 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.909188    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a509825-85eb-4d81-999c-94dc220be99d-proc\") pod \"perf-node-gather-daemonset-x4x2q\" (UID: \"8a509825-85eb-4d81-999c-94dc220be99d\") " pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:28:59.909288 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.909252    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a509825-85eb-4d81-999c-94dc220be99d-proc\") pod \"perf-node-gather-daemonset-x4x2q\" (UID: \"8a509825-85eb-4d81-999c-94dc220be99d\") " pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:28:59.909513 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.909319    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a509825-85eb-4d81-999c-94dc220be99d-lib-modules\") pod \"perf-node-gather-daemonset-x4x2q\" (UID: \"8a509825-85eb-4d81-999c-94dc220be99d\") " pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:28:59.909513 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.909319    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a509825-85eb-4d81-999c-94dc220be99d-podres\") pod \"perf-node-gather-daemonset-x4x2q\" (UID: \"8a509825-85eb-4d81-999c-94dc220be99d\") " pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:28:59.917968 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:28:59.917935    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jk7t\" (UniqueName: \"kubernetes.io/projected/8a509825-85eb-4d81-999c-94dc220be99d-kube-api-access-9jk7t\") pod \"perf-node-gather-daemonset-x4x2q\" (UID: \"8a509825-85eb-4d81-999c-94dc220be99d\") " pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:29:00.054396 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:00.054306    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:29:00.328572 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:00.328549    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"]
Sep 18 14:29:00.330593 ip-10-0-128-141 kubenswrapper[2572]: W0918 14:29:00.330562    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8a509825_85eb_4d81_999c_94dc220be99d.slice/crio-01d736ca30bcf9bd1999a5f29140deaaa41a5b82ccdd693f7f4772858b024768 WatchSource:0}: Error finding container 01d736ca30bcf9bd1999a5f29140deaaa41a5b82ccdd693f7f4772858b024768: Status 404 returned error can't find the container with id 01d736ca30bcf9bd1999a5f29140deaaa41a5b82ccdd693f7f4772858b024768
Sep 18 14:29:00.596985 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:00.596883    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-lb7tl_1b1982f4-552c-47bb-8679-cfaf7091f1be/volume-data-source-validator/0.log"
Sep 18 14:29:01.151506 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:01.151460    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q" event={"ID":"8a509825-85eb-4d81-999c-94dc220be99d","Type":"ContainerStarted","Data":"b13f58a8b44a739f975389ca32f4fe5547c7467e5c839338f2fe1838e06340eb"}
Sep 18 14:29:01.151726 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:01.151524    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q" event={"ID":"8a509825-85eb-4d81-999c-94dc220be99d","Type":"ContainerStarted","Data":"01d736ca30bcf9bd1999a5f29140deaaa41a5b82ccdd693f7f4772858b024768"}
Sep 18 14:29:01.151726 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:01.151591    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:29:01.168100 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:01.168042    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q" podStartSLOduration=2.168026095 podStartE2EDuration="2.168026095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 14:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:29:01.166391429 +0000 UTC m=+2677.101356574" watchObservedRunningTime="2026-09-18 14:29:01.168026095 +0000 UTC m=+2677.102991240"
Sep 18 14:29:01.865614 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:01.865573    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qmrw7_233eb0fd-25e7-4116-88c8-c2aad00a3fac/dns/0.log"
Sep 18 14:29:01.886522 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:01.886492    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qmrw7_233eb0fd-25e7-4116-88c8-c2aad00a3fac/kube-rbac-proxy/0.log"
Sep 18 14:29:01.934267 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:01.934240    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-txrvf_d34b196c-8eaa-40cf-a2e4-6b9c5a07652f/dns-node-resolver/0.log"
Sep 18 14:29:02.531131 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:02.531097    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-ddf994ddf-lbkh4_4357fd99-f1f1-4070-8b74-59f84c4a483a/registry/0.log"
Sep 18 14:29:02.583373 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:02.583338    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n2zrq_3b7d8ad0-cdab-41f1-81d7-de60c5b2d692/node-ca/0.log"
Sep 18 14:29:04.388265 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:04.388223    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mrcfl_c908b76c-afff-4af9-81bb-be1c7f8df54e/discovery/0.log"
Sep 18 14:29:04.434547 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:04.434507    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849fd8447d-n2lrt_4144f489-44f8-4933-8d38-6c0fea51a78f/kube-auth-proxy/0.log"
Sep 18 14:29:04.567897 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:04.567850    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77578d5fdb-w9ttn_5b1b97a5-adb1-4a5f-89a9-785ea463af4e/router/0.log"
Sep 18 14:29:05.502931 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:05.502898    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nm5v8_7f3f5b5d-0e90-4535-b2e9-d994388cbc51/serve-healthcheck-canary/0.log"
Sep 18 14:29:06.011052 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:06.011019    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bzrnp_1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e/kube-rbac-proxy/0.log"
Sep 18 14:29:06.034046 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:06.034021    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bzrnp_1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e/exporter/0.log"
Sep 18 14:29:06.056626 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:06.056597    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bzrnp_1cc1f02a-2e4b-43e7-8a7b-a3721a9be63e/extractor/0.log"
Sep 18 14:29:07.166601 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:07.166574    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pkp9v/perf-node-gather-daemonset-x4x2q"
Sep 18 14:29:08.958552 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:08.958507    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-qktrl_1d645000-9b99-4314-bcb1-bad1a334ec2b/manager/0.log"
Sep 18 14:29:09.030360 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:09.030328    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58db965b5-nxvlz_63f2ed63-c33e-4ae6-a475-ce090e5215e5/manager/0.log"
Sep 18 14:29:09.256770 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:09.256737    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz_66150446-024c-4fd4-994e-fa7eeb9c9f5a/manager/0.log"
Sep 18 14:29:09.279369 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:09.279345    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64b7fcb4fd-hb5wz_66150446-024c-4fd4-994e-fa7eeb9c9f5a/copy-manifests/0.log"
Sep 18 14:29:11.250587 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:11.250512    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mrcfl_c908b76c-afff-4af9-81bb-be1c7f8df54e/discovery/0.log"
Sep 18 14:29:11.276492 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:11.276445    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849fd8447d-n2lrt_4144f489-44f8-4933-8d38-6c0fea51a78f/kube-auth-proxy/0.log"
Sep 18 14:29:11.384045 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:11.384008    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77578d5fdb-w9ttn_5b1b97a5-adb1-4a5f-89a9-785ea463af4e/router/0.log"
Sep 18 14:29:13.847810 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:13.847775    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mrcfl_c908b76c-afff-4af9-81bb-be1c7f8df54e/discovery/0.log"
Sep 18 14:29:13.882518 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:13.882488    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849fd8447d-n2lrt_4144f489-44f8-4933-8d38-6c0fea51a78f/kube-auth-proxy/0.log"
Sep 18 14:29:13.984107 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:13.984074    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77578d5fdb-w9ttn_5b1b97a5-adb1-4a5f-89a9-785ea463af4e/router/0.log"
Sep 18 14:29:16.425979 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:16.425891    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-4wcgz_2cb3e59e-5c9c-486d-89dd-99474a7edade/kube-storage-version-migrator-operator/1.log"
Sep 18 14:29:16.427741 ip-10-0-128-141 kubenswrapper[2572]: I0918 14:29:16.427712    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-4wcgz_2cb3e59e-5c9c-486d-89dd-99474a7edade/kube-storage-version-migrator-operator/0.log"