Sep 07 10:11:37.396078 ip-10-0-141-62 systemd[1]: Starting Kubernetes Kubelet...
Sep 07 10:11:37.750075 ip-10-0-141-62 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 10:11:37.750075 ip-10-0-141-62 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 07 10:11:37.750075 ip-10-0-141-62 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 10:11:37.750075 ip-10-0-141-62 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 07 10:11:37.750075 ip-10-0-141-62 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 10:11:37.752152 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.752065    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 07 10:11:37.758395 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758368    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:11:37.758395 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758390    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:11:37.758395 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758394    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:11:37.758395 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758398    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:11:37.758395 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758401    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758404    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758407    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758410    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758413    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758416    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758419    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758422    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758425    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758427    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758430    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758432    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758435    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758438    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758440    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758443    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758445    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758450    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758453    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758457    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:11:37.758598 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758459    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758462    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758465    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758468    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758471    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758475    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758479    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758481    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758484    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758487    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758490    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758493    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758496    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758499    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758503    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758507    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758510    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758513    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758515    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:11:37.759096 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758518    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758522    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758525    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758528    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758530    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758533    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758535    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758539    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758542    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758544    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758547    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758550    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758552    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758555    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758558    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758560    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758563    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758565    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758568    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:11:37.759588 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758571    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758573    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758576    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758579    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758581    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758584    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758587    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758591    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758594    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758597    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758600    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758602    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758605    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758608    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758611    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758614    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758617    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758619    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758622    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758624    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:11:37.760072 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758629    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758633    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758636    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758639    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758641    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758644    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758646    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758649    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758651    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758654    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758657    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758660    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758662    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758665    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758667    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758670    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.758672    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759400    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759407    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759411    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:11:37.760540 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759415    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759417    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759420    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759423    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759427    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759429    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759432    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759435    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759438    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759441    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759443    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759447    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759449    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759452    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759455    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759458    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759461    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759465    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759468    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:11:37.761034 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759472    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759475    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759477    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759480    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759483    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759486    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759489    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759491    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759495    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759499    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759502    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759505    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759508    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759511    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759514    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759517    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759520    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759522    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759525    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759528    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:11:37.761496 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759530    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759533    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759535    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759538    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759540    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759543    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759546    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759548    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759552    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759554    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759557    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759560    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759562    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759565    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759567    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759570    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759573    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759576    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759579    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759581    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:11:37.762004 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759584    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759586    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759589    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759592    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759595    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759598    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759601    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759604    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759606    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759610    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759613    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759616    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759619    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759622    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759625    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759627    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759630    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759632    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759635    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:11:37.762484 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759638    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759641    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759644    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759648    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759651    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759653    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759656    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759658    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759661    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759665    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759668    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759671    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759673    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759676    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759678    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759681    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759684    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.759687    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760080    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760085    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:11:37.762950 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760088    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760092    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760094    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760098    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760100    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760103    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760106    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760108    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760111    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760113    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760116    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760119    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760121    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760124    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760127    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760130    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760132    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760135    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760137    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760140    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:11:37.763456 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760142    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760147    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760150    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760152    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760155    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760158    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760161    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760163    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760166    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760169    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760172    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760175    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760177    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760180    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760182    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760186    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760188    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760191    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760193    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760196    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:11:37.764031 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760198    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760201    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760203    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760206    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760208    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760211    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760213    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760216    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760219    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760221    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760224    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760227    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760229    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760233    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760236    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760239    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760241    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760245    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760248    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:11:37.764516 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760251    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760254    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760256    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760259    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760261    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760264    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760267    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760269    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760272    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760274    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760277    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760279    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760282    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760287    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760291    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760294    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760296    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760299    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760302    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:11:37.765016 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760305    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760308    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760311    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760313    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760316    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760318    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760321    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760325    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760327    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760330    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760332    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760335    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760337    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760340    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760342    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760345    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760349    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760352    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.760355    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760426    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 07 10:11:37.765477 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760464    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760470    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760475    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760479    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760482    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760486    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760491    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760494    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760497    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760501    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760504    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760507    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760510    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760513    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760516    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760524    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760527    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760532    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760534    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760539    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760543    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760546    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760550    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 07 10:11:37.765995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760553    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760557    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760560    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760563    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760566    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760569    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760572    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760575    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760578    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760581    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760584    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760587    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760590    2562 flags.go:64] FLAG: --enable-server="true"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760593    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760597    2562 flags.go:64] FLAG: --event-burst="100"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760600    2562 flags.go:64] FLAG: --event-qps="50"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760603    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760606    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760608    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760612    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760615    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760618    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760621    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760624    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760627    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 07 10:11:37.766564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760630    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760633    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760636    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760640    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760643    2562 flags.go:64] FLAG: --feature-gates=""
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760647    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760650    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760653    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760656    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760659    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760662    2562 flags.go:64] FLAG: --help="false"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760665    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-141-62.ec2.internal"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760668    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760671    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760673    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760677    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760680    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760683    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760686    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760688    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760691    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760694    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760697    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760699    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 07 10:11:37.767177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760702    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760705    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760708    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760711    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760714    2562 flags.go:64] FLAG: --lock-file=""
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760716    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760719    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760722    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760727    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760730    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760733    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760736    2562 flags.go:64] FLAG: --logging-format="text"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760740    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760744    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760746    2562 flags.go:64] FLAG: --manifest-url=""
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760749    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760754    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760757    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760775    2562 flags.go:64] FLAG: --max-pods="110"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760778    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760781    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760784    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760787    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760790    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760793    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 07 10:11:37.767809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760796    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760803    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760806    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760809    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760812    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760815    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760821    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760823    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760826    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760830    2562 flags.go:64] FLAG: --port="10250"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760833    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760835    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-048f3523b3be4042e"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760838    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760842    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760845    2562 flags.go:64] FLAG: --register-node="true"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760848    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760851    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760855    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760857    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760862    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760870    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760873    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760876    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760879    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760882    2562 flags.go:64] FLAG: --runonce="false"
Sep 07 10:11:37.768422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760885    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760888    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760891    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760894    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760897    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760900    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760903    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760905    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760908    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760911    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760914    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760916    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760919    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760922    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760925    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760930    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760933    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760935    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760940    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760942    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760945    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760948    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760951    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760954    2562 flags.go:64] FLAG: --v="2"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760957    2562 flags.go:64] FLAG: --version="false"
Sep 07 10:11:37.769029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760962    2562 flags.go:64] FLAG: --vmodule=""
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760967    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.760970    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761093    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761097    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761108    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761110    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761113    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761116    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761119    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761121    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761124    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761127    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761129    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761132    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761134    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761137    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761140    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761142    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761145    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:11:37.769678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761147    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761150    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761153    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761155    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761158    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761160    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761163    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761165    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761168    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761172    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761175    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761178    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761182    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761187    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761190    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761193    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761196    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761199    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761201    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:11:37.770177 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761204    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761206    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761209    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761211    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761214    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761216    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761220    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761222    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761225    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761227    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761230    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761233    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761235    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761238    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761241    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761243    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761246    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761248    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761251    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761253    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:11:37.770630 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761256    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761258    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761261    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761263    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761266    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761270    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761274    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761277    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761280    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761283    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761285    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761288    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761290    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761293    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761295    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761298    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761301    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761303    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761306    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761308    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:11:37.771119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761311    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761313    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761316    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761326    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761329    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761332    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761334    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761339    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761342    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761345    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761348    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761350    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761353    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761356    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761358    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761361    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761364    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761370    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761376    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:11:37.771599 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761379    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:11:37.772083 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761382    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:11:37.772083 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761385    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:11:37.772083 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.761388    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:11:37.772083 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.761890    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 10:11:37.772083 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.768838    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 07 10:11:37.772083 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.768855    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 07 10:11:37.772083 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.768875    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 07 10:11:37.772083 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.768886    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 07 10:11:37.772083 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768940    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:11:37.772083 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768945    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:11:37.772083 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768948    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:11:37.772083 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768951    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:11:37.772083 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768954    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:11:37.772083 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768957    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:11:37.772083 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768960    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768963    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768967    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768971    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768974    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768977    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768980    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768983    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768986    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768989    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768992    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768994    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.768997    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769000    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769003    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769006    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769008    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769011    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769013    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:11:37.772493 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769016    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769018    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769021    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769023    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769028    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769031    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769034    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769037    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769040    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769042    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769045    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769048    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769051    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769053    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769056    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769058    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769061    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769063    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769066    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769069    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:11:37.772993 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769071    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769074    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769076    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769079    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769081    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769083    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769086    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769089    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769091    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769095    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769099    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769102    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769105    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769107    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769110    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769113    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769115    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769119    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769122    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:11:37.773550 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769124    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769128    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769131    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769134    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769137    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769139    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769142    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769145    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769148    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769150    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769153    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769155    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769158    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769160    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769163    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769165    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769168    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769170    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769173    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:11:37.774147 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769175    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:11:37.774629 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769178    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:11:37.774629 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769181    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:11:37.774629 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769184    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:11:37.774629 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769187    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:11:37.774629 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769189    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:11:37.774629 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769192    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:11:37.774629 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769194    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:11:37.774629 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769197    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:11:37.774629 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769200    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:11:37.774629 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769202    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:11:37.774629 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769205    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:11:37.774629 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769208    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:11:37.774629 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769210    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:11:37.774629 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769213    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:11:37.774629 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769215    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:11:37.775019 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.769220    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 10:11:37.775019 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769342    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:11:37.775019 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769347    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:11:37.775019 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769351    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:11:37.775019 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769354    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:11:37.775019 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769358    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:11:37.775019 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769361    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:11:37.775019 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769365    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:11:37.775019 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769369    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:11:37.775019 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769373    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:11:37.775019 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769376    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:11:37.775019 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769379    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:11:37.775019 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769382    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:11:37.775019 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769385    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769387    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769390    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769393    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769395    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769398    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769401    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769404    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769407    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769409    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769412    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769414    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769417    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769419    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769422    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769424    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769427    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769430    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769433    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769435    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:11:37.775449 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769438    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769440    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769443    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769445    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769448    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769450    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769453    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769455    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769458    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769461    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769463    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769466    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769469    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769471    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769474    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769476    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769479    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769481    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769484    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769487    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:11:37.775979 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769489    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769492    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769494    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769497    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769499    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769503    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769506    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769509    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769512    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769520    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769523    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769525    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769528    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769531    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769534    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769537    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769539    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769542    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769545    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769547    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:11:37.776786 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769550    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769552    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769555    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769558    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769561    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769563    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769566    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769568    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769571    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769574    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769577    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769579    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769582    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769585    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769587    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769590    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769592    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769595    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769598    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769600    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:11:37.777367 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769603    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:11:37.778117 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769605    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:11:37.778117 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769608    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:11:37.778117 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769611    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:11:37.778117 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769613    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:11:37.778117 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769616    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:11:37.778117 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:37.769619    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:11:37.778117 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.769623    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 10:11:37.778117 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.769725    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 07 10:11:37.778117 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.772061    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 07 10:11:37.778117 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.772992    2562 server.go:1019] "Starting client certificate rotation"
Sep 07 10:11:37.778117 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.773113    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 10:11:37.778117 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.773886    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 10:11:37.778117 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.775376    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 10:11:37.778600 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.778077    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 10:11:37.789792 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.789772    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 07 10:11:37.792672 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.792655    2562 log.go:25] "Validated CRI v1 image API"
Sep 07 10:11:37.793491 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.793474    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 07 10:11:37.796214 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.796193    2562 fs.go:135] Filesystem UUIDs: map[56baf68c-452c-41a5-8e78-7366b6081e14:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bab6c2bb-2faa-4e40-b986-279f2d359742:/dev/nvme0n1p4]
Sep 07 10:11:37.796289 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.796214    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 07 10:11:37.800879 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.800752    2562 manager.go:216] Machine: {Timestamp:2026-09-07 10:11:37.80026539 +0000 UTC m=+0.297955309 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101681 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26057a3e8f389f48830620ff3b4edc SystemUUID:ec26057a-3e8f-389f-4883-0620ff3b4edc BootID:0c02a088-c96a-4ecc-a6b9-6429edce4b94 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ee:5c:f7:52:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ee:5c:f7:52:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:d5:a0:79:ea:76 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 07 10:11:37.800879 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.800871    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 07 10:11:37.801026 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.800949    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 07 10:11:37.801551 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.801533    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 10:11:37.802495 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.802463    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 07 10:11:37.802654 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.802503    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-62.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 07 10:11:37.803158 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.803146    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 07 10:11:37.803210 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.803163    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 07 10:11:37.803210 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.803174    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 10:11:37.804147 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.804134    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 10:11:37.808673 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.808660    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 07 10:11:37.812382 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.812368    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 10:11:37.812510 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.812500    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 07 10:11:37.812574 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.812539    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 07 10:11:37.812574 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.812554    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 07 10:11:37.812668 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.812576    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 07 10:11:37.812668 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.812590    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 07 10:11:37.812668 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.812602    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 07 10:11:37.813013 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.813001    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 10:11:37.813082 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.813024    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 10:11:37.816039 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.816022    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 07 10:11:37.818152 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.818131    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 07 10:11:37.818226 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.818174    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 07 10:11:37.818915 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.818903    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 07 10:11:37.818989 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.818923    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 07 10:11:37.818989 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.818933    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 07 10:11:37.818989 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.818942    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 07 10:11:37.818989 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.818954    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 07 10:11:37.818989 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.818964    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 07 10:11:37.818989 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.818984    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 07 10:11:37.819219 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.818992    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 07 10:11:37.819219 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.819002    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 07 10:11:37.819219 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.819011    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 07 10:11:37.819219 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.819039    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 07 10:11:37.819219 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.819048    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 07 10:11:37.820190 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.820137    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 07 10:11:37.820386 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.820374    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 07 10:11:37.822779 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:37.822718    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-62.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 07 10:11:37.822932 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:37.822384    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 07 10:11:37.823612 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.823580    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-62.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 07 10:11:37.825072 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.825055    2562 server.go:1268] "Started kubelet"
Sep 07 10:11:37.825232 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.825201    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 07 10:11:37.825307 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.825188    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 07 10:11:37.825365 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.825341    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 07 10:11:37.826010 ip-10-0-141-62 systemd[1]: Started Kubernetes Kubelet.
Sep 07 10:11:37.828055 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.826502    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 07 10:11:37.828055 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.827314    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 07 10:11:37.829953 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.829105    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wjzpp"
Sep 07 10:11:37.829953 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:37.829379    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 07 10:11:37.831652 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.831385    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 07 10:11:37.832120 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.832098    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 07 10:11:37.832309 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:37.832183    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-62.ec2.internal\" not found"
Sep 07 10:11:37.832944 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.832928    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 07 10:11:37.833062 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.833051    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 07 10:11:37.833129 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.833069    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 07 10:11:37.833180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.833152    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 07 10:11:37.833180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.833163    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 07 10:11:37.833263 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.833242    2562 factory.go:153] Registering CRI-O factory
Sep 07 10:11:37.833306 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.833275    2562 factory.go:223] Registration of the crio container factory successfully
Sep 07 10:11:37.833364 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.833328    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 07 10:11:37.833364 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.833336    2562 factory.go:55] Registering systemd factory
Sep 07 10:11:37.833364 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.833342    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 07 10:11:37.833364 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.833355    2562 factory.go:103] Registering Raw factory
Sep 07 10:11:37.833364 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.833366    2562 manager.go:1196] Started watching for new ooms in manager
Sep 07 10:11:37.834289 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.834265    2562 manager.go:349] Starting recovery of all containers
Sep 07 10:11:37.835017 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.834994    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wjzpp"
Sep 07 10:11:37.836898 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:37.836877    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-62.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 07 10:11:37.837002 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:37.836980    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 07 10:11:37.838082 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:37.837106    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-62.ec2.internal.18d30233a4a41e85  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-62.ec2.internal,UID:ip-10-0-141-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-62.ec2.internal,},FirstTimestamp:2026-09-07 10:11:37.825025669 +0000 UTC m=+0.322715570,LastTimestamp:2026-09-07 10:11:37.825025669 +0000 UTC m=+0.322715570,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-62.ec2.internal,}"
Sep 07 10:11:37.845156 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.844986    2562 manager.go:354] Recovery completed
Sep 07 10:11:37.851001 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.850981    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:11:37.853461 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.853440    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:11:37.853536 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.853474    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:11:37.853536 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.853489    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:11:37.854094 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.854077    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 07 10:11:37.854165 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.854095    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 07 10:11:37.854165 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.854115    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 10:11:37.854165 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.854134    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:11:37.854870 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.854852    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:11:37.854936 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.854885    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:11:37.854936 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.854899    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:11:37.855412 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.855398    2562 policy_none.go:49] "None policy: Start"
Sep 07 10:11:37.855445 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.855419    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 07 10:11:37.855445 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.855435    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 07 10:11:37.856530 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.856519    2562 policy_none.go:47] "Start"
Sep 07 10:11:37.895369 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.895354    2562 manager.go:337] "Starting Device Plugin manager"
Sep 07 10:11:37.895484 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:37.895413    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 07 10:11:37.895484 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.895426    2562 server.go:87] "Starting device plugin registration server"
Sep 07 10:11:37.895825 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.895813    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 07 10:11:37.895872 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.895830    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 07 10:11:37.896035 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.896019    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 07 10:11:37.896119 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.896107    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 07 10:11:37.896174 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.896120    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 07 10:11:37.896597 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:37.896563    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 07 10:11:37.896701 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:37.896614    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-62.ec2.internal\" not found"
Sep 07 10:11:37.956453 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.956417    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 07 10:11:37.957801 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.957787    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 07 10:11:37.957914 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.957846    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 07 10:11:37.957914 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.957875    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 07 10:11:37.958011 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:37.957920    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 07 10:11:37.960141 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.960120    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:11:37.996873 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.996841    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:11:37.997948 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.997929    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:11:37.998046 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.997964    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:11:37.998046 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.997977    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:11:37.998046 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:37.998006    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.006055 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.005975    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.006055 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:38.006008    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-62.ec2.internal\": node \"ip-10-0-141-62.ec2.internal\" not found"
Sep 07 10:11:38.055120 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:38.055092    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-62.ec2.internal\" not found"
Sep 07 10:11:38.058183 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.058154    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-62.ec2.internal"]
Sep 07 10:11:38.058245 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.058235    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:11:38.060010 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.059988    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:11:38.060108 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.060017    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:11:38.060108 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.060028    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:11:38.061587 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.061575    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:11:38.062313 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.062298    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.062385 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.062334    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:11:38.062385 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.062356    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:11:38.062385 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.062380    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:11:38.062473 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.062392    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:11:38.062983 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.062970    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:11:38.063052 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.062992    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:11:38.063052 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.063001    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:11:38.063863 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.063850    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.063912 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.063874    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:11:38.064557 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.064544    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:11:38.064603 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.064570    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:11:38.064603 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.064580    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:11:38.087478 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:38.087451    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-62.ec2.internal\" not found" node="ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.091994 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:38.091978    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-62.ec2.internal\" not found" node="ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.134738 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.134706    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d9182b821ba5135386976925f2d8116-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal\" (UID: \"5d9182b821ba5135386976925f2d8116\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.134738 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.134738    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/236f13cb4cc1d42288493688c2e3dab9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-62.ec2.internal\" (UID: \"236f13cb4cc1d42288493688c2e3dab9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.134929 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.134756    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d9182b821ba5135386976925f2d8116-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal\" (UID: \"5d9182b821ba5135386976925f2d8116\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.155692 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:38.155666    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-62.ec2.internal\" not found"
Sep 07 10:11:38.235326 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.235298    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d9182b821ba5135386976925f2d8116-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal\" (UID: \"5d9182b821ba5135386976925f2d8116\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.235435 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.235333    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/236f13cb4cc1d42288493688c2e3dab9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-62.ec2.internal\" (UID: \"236f13cb4cc1d42288493688c2e3dab9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.235435 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.235349    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d9182b821ba5135386976925f2d8116-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal\" (UID: \"5d9182b821ba5135386976925f2d8116\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.235435 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.235399    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d9182b821ba5135386976925f2d8116-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal\" (UID: \"5d9182b821ba5135386976925f2d8116\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.235529 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.235451    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d9182b821ba5135386976925f2d8116-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal\" (UID: \"5d9182b821ba5135386976925f2d8116\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.235529 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.235476    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/236f13cb4cc1d42288493688c2e3dab9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-62.ec2.internal\" (UID: \"236f13cb4cc1d42288493688c2e3dab9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.256484 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:38.256423    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-62.ec2.internal\" not found"
Sep 07 10:11:38.357322 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:38.357292    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-62.ec2.internal\" not found"
Sep 07 10:11:38.390501 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.390465    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.395405 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.395386    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-62.ec2.internal"
Sep 07 10:11:38.458123 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:38.458084    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-62.ec2.internal\" not found"
Sep 07 10:11:38.559881 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:38.559802    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-62.ec2.internal\" not found"
Sep 07 10:11:38.661641 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:38.661604    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-62.ec2.internal\" not found"
Sep 07 10:11:38.727524 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.727494    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:11:38.763502 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:38.763476    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-62.ec2.internal\" not found"
Sep 07 10:11:38.772831 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.772759    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 07 10:11:38.772965 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.772939    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 10:11:38.773012 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.772971    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 10:11:38.836817 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.836723    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 10:06:38 +0000 UTC" deadline="2028-04-13 06:18:17.41697858 +0000 UTC"
Sep 07 10:11:38.836817 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.836755    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14012h6m38.580226239s"
Sep 07 10:11:38.864399 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:38.864369    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-62.ec2.internal\" not found"
Sep 07 10:11:38.965346 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:38.965312    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-62.ec2.internal\" not found"
Sep 07 10:11:38.965765 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:38.965735    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9182b821ba5135386976925f2d8116.slice/crio-07caba923b7c17634b07ed3e8f0bd59dccca43379d3c61cbde5dafe4cc620d71 WatchSource:0}: Error finding container 07caba923b7c17634b07ed3e8f0bd59dccca43379d3c61cbde5dafe4cc620d71: Status 404 returned error can't find the container with id 07caba923b7c17634b07ed3e8f0bd59dccca43379d3c61cbde5dafe4cc620d71
Sep 07 10:11:38.966277 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:38.966254    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236f13cb4cc1d42288493688c2e3dab9.slice/crio-4b77e1f6818f1d4731810d60b24c2bf90757c38d1dabb15ed1abed296d57ff13 WatchSource:0}: Error finding container 4b77e1f6818f1d4731810d60b24c2bf90757c38d1dabb15ed1abed296d57ff13: Status 404 returned error can't find the container with id 4b77e1f6818f1d4731810d60b24c2bf90757c38d1dabb15ed1abed296d57ff13
Sep 07 10:11:38.972681 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:38.972664    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 10:11:39.010414 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.010391    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:11:39.033049 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.033022    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal"
Sep 07 10:11:39.044607 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.044586    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 10:11:39.046448 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.046435    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-62.ec2.internal"
Sep 07 10:11:39.055627 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.055605    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 10:11:39.171228 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.171141    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:11:39.814368 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.814333    2562 apiserver.go:52] "Watching apiserver"
Sep 07 10:11:39.822433 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.822403    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 07 10:11:39.822980 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.822937    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-czjtg","kube-system/global-pull-secret-syncer-tk7lm","kube-system/konnectivity-agent-xgmhs","kube-system/kube-apiserver-proxy-ip-10-0-141-62.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal","openshift-multus/multus-tmj6t","openshift-network-diagnostics/network-check-target-q2724","openshift-ovn-kubernetes/ovnkube-node-959sz","openshift-cluster-node-tuning-operator/tuned-z9jpp","openshift-dns/node-resolver-qjnz6","openshift-image-registry/node-ca-v86rj","openshift-multus/multus-additional-cni-plugins-6kdt9","openshift-multus/network-metrics-daemon-l7n5z"]
Sep 07 10:11:39.825842 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.825817    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.828322 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.828251    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 07 10:11:39.828420 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.828373    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 07 10:11:39.828484 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.828380    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.828530 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.828499    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hzrdz\""
Sep 07 10:11:39.830990 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.830950    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xgmhs"
Sep 07 10:11:39.831086 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.831026    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 07 10:11:39.831150 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.831083    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 07 10:11:39.831786 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.831391    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 07 10:11:39.831786 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.831448    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 07 10:11:39.831786 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.831449    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-68tkj\""
Sep 07 10:11:39.831786 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.831513    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 07 10:11:39.832424 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.832407    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 07 10:11:39.834120 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.833929    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 07 10:11:39.834120 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.833991    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nhwjb\""
Sep 07 10:11:39.834120 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.833996    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 07 10:11:39.834120 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.834093    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:39.834375 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:39.834200    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:11:39.836608 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.836586    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:39.840198 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.839156    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:39.840198 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:39.839224    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:11:39.840198 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.839831    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 07 10:11:39.840198 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.840039    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 07 10:11:39.840475 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.840218    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 07 10:11:39.840475 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.840380    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l7t4m\""
Sep 07 10:11:39.842756 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.841926    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.844341 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.844319    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 07 10:11:39.845022 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.844621    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.845022 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.844758    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nw5cn\""
Sep 07 10:11:39.848990 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.847440    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 10:11:39.848990 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.848532    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 07 10:11:39.848990 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.848627    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.848990 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.848660    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.848990 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.848697    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.848990 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.848742    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcqr8\" (UniqueName: \"kubernetes.io/projected/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-kube-api-access-rcqr8\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.848990 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.848796    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-var-lib-kubelet\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.848990 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.848827    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 07 10:11:39.848990 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.848921    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 07 10:11:39.848990 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.848830    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8284b473-d11f-48e6-860e-277051c2086e-tmp\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.848990 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.848968    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-cnibin\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849013    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ba3d830-3508-49a6-8d41-36409f735098-konnectivity-ca\") pod \"konnectivity-agent-xgmhs\" (UID: \"6ba3d830-3508-49a6-8d41-36409f735098\") " pod="kube-system/konnectivity-agent-xgmhs"
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849139    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs\") pod \"network-metrics-daemon-l7n5z\" (UID: \"7254bd39-6628-42fd-96e9-6a01b723c275\") " pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849177    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrpp2\" (UniqueName: \"kubernetes.io/projected/7254bd39-6628-42fd-96e9-6a01b723c275-kube-api-access-nrpp2\") pod \"network-metrics-daemon-l7n5z\" (UID: \"7254bd39-6628-42fd-96e9-6a01b723c275\") " pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849226    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849236    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-sysconfig\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849274    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-sysctl-d\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849305    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-host\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849386    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849425    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ba3d830-3508-49a6-8d41-36409f735098-agent-certs\") pod \"konnectivity-agent-xgmhs\" (UID: \"6ba3d830-3508-49a6-8d41-36409f735098\") " pod="kube-system/konnectivity-agent-xgmhs"
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849558    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-modprobe-d\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.848842    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849615    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-sysctl-conf\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849669    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-run\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849722    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8284b473-d11f-48e6-860e-277051c2086e-etc-tuned\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849762    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbfxd\" (UniqueName: \"kubernetes.io/projected/8284b473-d11f-48e6-860e-277051c2086e-kube-api-access-qbfxd\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849771    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849811    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-system-cni-dir\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849837    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 07 10:11:39.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.849860    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-kubernetes\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.852092 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:39.849867    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:11:39.852092 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.850013    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-systemd\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.852092 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.850059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-sys\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.852092 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.850157    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-lib-modules\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.852092 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.851132    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-czjtg"
Sep 07 10:11:39.852092 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.851306    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8wpgm\""
Sep 07 10:11:39.853848 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.853822    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 07 10:11:39.854119 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.854099    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 07 10:11:39.854440 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.854398    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 07 10:11:39.854561 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.854463    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-48zjp\""
Sep 07 10:11:39.854734 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.854680    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qjnz6"
Sep 07 10:11:39.857039 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.857015    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v86rj"
Sep 07 10:11:39.858119 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.858094    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 07 10:11:39.858322 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.858300    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 07 10:11:39.858423 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.858312    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7rw82\""
Sep 07 10:11:39.860533 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.860510    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 07 10:11:39.860744 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.860723    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 07 10:11:39.860857 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.860823    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 07 10:11:39.861022 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.861003    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-csphj\""
Sep 07 10:11:39.934401 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.934376    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 07 10:11:39.951137 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-run-k8s-cni-cncf-io\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.951275 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-hostroot\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.951275 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951178    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-run-openvswitch\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.951275 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951196    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbqdp\" (UniqueName: \"kubernetes.io/projected/ac0c45e5-27f8-4c26-929a-001e4b644d46-kube-api-access-cbqdp\") pod \"node-resolver-qjnz6\" (UID: \"ac0c45e5-27f8-4c26-929a-001e4b644d46\") " pod="openshift-dns/node-resolver-qjnz6"
Sep 07 10:11:39.951275 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951213    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz2rs\" (UniqueName: \"kubernetes.io/projected/70b33bfb-c309-4092-9e67-f09a30673a7d-kube-api-access-pz2rs\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.951275 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951241    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.951275 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951268    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcqr8\" (UniqueName: \"kubernetes.io/projected/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-kube-api-access-rcqr8\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.951569 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951294    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrpp2\" (UniqueName: \"kubernetes.io/projected/7254bd39-6628-42fd-96e9-6a01b723c275-kube-api-access-nrpp2\") pod \"network-metrics-daemon-l7n5z\" (UID: \"7254bd39-6628-42fd-96e9-6a01b723c275\") " pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:39.951569 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951319    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.951569 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951345    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-socket-dir\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:39.951569 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951369    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2nl5\" (UniqueName: \"kubernetes.io/projected/86e562c2-10c5-463c-a528-2d9f9c54737d-kube-api-access-r2nl5\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:39.951770 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-cnibin\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.951770 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951626    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-kubelet\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.951770 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951646    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-cnibin\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.951770 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951687    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-run-systemd\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.951770 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951732    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-cni-netd\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.952013 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951792    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-ovnkube-config\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.952013 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951848    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-device-dir\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:39.952013 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951887    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6xd6\" (UniqueName: \"kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6\") pod \"network-check-target-q2724\" (UID: \"948c00ef-80eb-4ff1-95c8-81d24b2a4d57\") " pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:39.952013 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951915    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-system-cni-dir\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.952013 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951943    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.952013 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.951956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-host\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.952248 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952045    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-host\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.952248 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952100    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.952248 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952136    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70b33bfb-c309-4092-9e67-f09a30673a7d-cni-binary-copy\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.952248 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952163    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-run-multus-certs\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.952248 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952193    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0bf9977-9b65-4b22-958e-46b52f65079e-kubelet-config\") pod \"global-pull-secret-syncer-tk7lm\" (UID: \"e0bf9977-9b65-4b22-958e-46b52f65079e\") " pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:39.952462 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-modprobe-d\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.952462 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952295    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-cni-bin\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.952462 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952324    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-systemd-units\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.952462 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952403    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-run-netns\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.952462 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952435    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa5caa93-f632-4e94-bfe6-4c2a54d9bf71-iptables-alerter-script\") pod \"iptables-alerter-czjtg\" (UID: \"aa5caa93-f632-4e94-bfe6-4c2a54d9bf71\") " pod="openshift-network-operator/iptables-alerter-czjtg"
Sep 07 10:11:39.952462 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952436    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-modprobe-d\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.952776 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-multus-socket-dir-parent\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.952776 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952521    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8284b473-d11f-48e6-860e-277051c2086e-tmp\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.952776 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952568    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.952776 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952577    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac0c45e5-27f8-4c26-929a-001e4b644d46-hosts-file\") pod \"node-resolver-qjnz6\" (UID: \"ac0c45e5-27f8-4c26-929a-001e4b644d46\") " pod="openshift-dns/node-resolver-qjnz6"
Sep 07 10:11:39.952776 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952609    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-registration-dir\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:39.952776 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952650    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-multus-conf-dir\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.952776 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952684    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ba3d830-3508-49a6-8d41-36409f735098-agent-certs\") pod \"konnectivity-agent-xgmhs\" (UID: \"6ba3d830-3508-49a6-8d41-36409f735098\") " pod="kube-system/konnectivity-agent-xgmhs"
Sep 07 10:11:39.952776 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952736    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-etc-openvswitch\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.952776 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952758    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-node-log\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.953180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:39.953180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952864    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-cnibin\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.953180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952917    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70b33bfb-c309-4092-9e67-f09a30673a7d-multus-daemon-config\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.953180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952955    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.953180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.952986    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbfxd\" (UniqueName: \"kubernetes.io/projected/8284b473-d11f-48e6-860e-277051c2086e-kube-api-access-qbfxd\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.953180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-system-cni-dir\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.953180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953072    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-run-ovn\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.953180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953099    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0bf9977-9b65-4b22-958e-46b52f65079e-dbus\") pod \"global-pull-secret-syncer-tk7lm\" (UID: \"e0bf9977-9b65-4b22-958e-46b52f65079e\") " pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:39.953180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-kubernetes\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.953180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953154    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-sys\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.953180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-lib-modules\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953203    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-log-socket\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953214    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bk9ww"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953233    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953280    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-env-overrides\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953311    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-kubernetes\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953316    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-sys\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953355    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f37b4a59-9c92-41fd-9561-ed2d017c6751-serviceca\") pod \"node-ca-v86rj\" (UID: \"f37b4a59-9c92-41fd-9561-ed2d017c6751\") " pod="openshift-image-registry/node-ca-v86rj"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953381    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-sys-fs\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953410    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret\") pod \"global-pull-secret-syncer-tk7lm\" (UID: \"e0bf9977-9b65-4b22-958e-46b52f65079e\") " pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953445    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-lib-modules\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953458    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953485    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzvmt\" (UniqueName: \"kubernetes.io/projected/aa5caa93-f632-4e94-bfe6-4c2a54d9bf71-kube-api-access-zzvmt\") pod \"iptables-alerter-czjtg\" (UID: \"aa5caa93-f632-4e94-bfe6-4c2a54d9bf71\") " pod="openshift-network-operator/iptables-alerter-czjtg"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953514    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pxk8\" (UniqueName: \"kubernetes.io/projected/f37b4a59-9c92-41fd-9561-ed2d017c6751-kube-api-access-7pxk8\") pod \"node-ca-v86rj\" (UID: \"f37b4a59-9c92-41fd-9561-ed2d017c6751\") " pod="openshift-image-registry/node-ca-v86rj"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953543    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-var-lib-kubelet\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953487    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-system-cni-dir\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953588    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-etc-kubernetes\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.953619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953616    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-sysconfig\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953644    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-sysctl-d\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953688    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-sysconfig\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953725    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-ovnkube-script-lib\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f37b4a59-9c92-41fd-9561-ed2d017c6751-host\") pod \"node-ca-v86rj\" (UID: \"f37b4a59-9c92-41fd-9561-ed2d017c6751\") " pod="openshift-image-registry/node-ca-v86rj"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953829    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-sysctl-conf\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953857    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-sysctl-d\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953870    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-run\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953909    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8284b473-d11f-48e6-860e-277051c2086e-etc-tuned\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-ovn-node-metrics-cert\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953971    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-sysctl-conf\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953992    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-etc-selinux\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.953987    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954017    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-multus-cni-dir\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954039    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-run\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954066    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-systemd\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.954345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954115    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-var-lib-openvswitch\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.955134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954138    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ckqb\" (UniqueName: \"kubernetes.io/projected/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-kube-api-access-2ckqb\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.955134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954162    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-run-netns\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.955134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954161    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-etc-systemd\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.955134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-var-lib-cni-bin\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.955134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954250    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-var-lib-cni-multus\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:39.955134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954276    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-var-lib-kubelet\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.955134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954294    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ba3d830-3508-49a6-8d41-36409f735098-konnectivity-ca\") pod \"konnectivity-agent-xgmhs\" (UID: \"6ba3d830-3508-49a6-8d41-36409f735098\") " pod="kube-system/konnectivity-agent-xgmhs"
Sep 07 10:11:39.955134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954306    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.955134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954313    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs\") pod \"network-metrics-daemon-l7n5z\" (UID: \"7254bd39-6628-42fd-96e9-6a01b723c275\") " pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:39.955134 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:39.954399    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:39.955134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954453    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8284b473-d11f-48e6-860e-277051c2086e-var-lib-kubelet\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.955134 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:39.954496    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs podName:7254bd39-6628-42fd-96e9-6a01b723c275 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:40.454464668 +0000 UTC m=+2.952154562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs") pod "network-metrics-daemon-l7n5z" (UID: "7254bd39-6628-42fd-96e9-6a01b723c275") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:39.955134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954692    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-slash\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:39.955134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954739    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa5caa93-f632-4e94-bfe6-4c2a54d9bf71-host-slash\") pod \"iptables-alerter-czjtg\" (UID: \"aa5caa93-f632-4e94-bfe6-4c2a54d9bf71\") " pod="openshift-network-operator/iptables-alerter-czjtg"
Sep 07 10:11:39.955134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954765    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0c45e5-27f8-4c26-929a-001e4b644d46-tmp-dir\") pod \"node-resolver-qjnz6\" (UID: \"ac0c45e5-27f8-4c26-929a-001e4b644d46\") " pod="openshift-dns/node-resolver-qjnz6"
Sep 07 10:11:39.955134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.954942    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ba3d830-3508-49a6-8d41-36409f735098-konnectivity-ca\") pod \"konnectivity-agent-xgmhs\" (UID: \"6ba3d830-3508-49a6-8d41-36409f735098\") " pod="kube-system/konnectivity-agent-xgmhs"
Sep 07 10:11:39.957404 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.957378    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8284b473-d11f-48e6-860e-277051c2086e-tmp\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.957668 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.957643    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ba3d830-3508-49a6-8d41-36409f735098-agent-certs\") pod \"konnectivity-agent-xgmhs\" (UID: \"6ba3d830-3508-49a6-8d41-36409f735098\") " pod="kube-system/konnectivity-agent-xgmhs"
Sep 07 10:11:39.957839 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.957802    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8284b473-d11f-48e6-860e-277051c2086e-etc-tuned\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.961147 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.961092    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcqr8\" (UniqueName: \"kubernetes.io/projected/757b7332-c3c9-4849-a0b7-cf8a4d46b3c1-kube-api-access-rcqr8\") pod \"multus-additional-cni-plugins-6kdt9\" (UID: \"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1\") " pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:39.961499 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.961479    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbfxd\" (UniqueName: \"kubernetes.io/projected/8284b473-d11f-48e6-860e-277051c2086e-kube-api-access-qbfxd\") pod \"tuned-z9jpp\" (UID: \"8284b473-d11f-48e6-860e-277051c2086e\") " pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:39.962134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.962115    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bk9ww"
Sep 07 10:11:39.962914 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.962868    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-62.ec2.internal" event={"ID":"236f13cb4cc1d42288493688c2e3dab9","Type":"ContainerStarted","Data":"4b77e1f6818f1d4731810d60b24c2bf90757c38d1dabb15ed1abed296d57ff13"}
Sep 07 10:11:39.964220 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.964200    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal" event={"ID":"5d9182b821ba5135386976925f2d8116","Type":"ContainerStarted","Data":"07caba923b7c17634b07ed3e8f0bd59dccca43379d3c61cbde5dafe4cc620d71"}
Sep 07 10:11:39.966943 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:39.966923    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrpp2\" (UniqueName: \"kubernetes.io/projected/7254bd39-6628-42fd-96e9-6a01b723c275-kube-api-access-nrpp2\") pod \"network-metrics-daemon-l7n5z\" (UID: \"7254bd39-6628-42fd-96e9-6a01b723c275\") " pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:40.055824 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.055792    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6xd6\" (UniqueName: \"kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6\") pod \"network-check-target-q2724\" (UID: \"948c00ef-80eb-4ff1-95c8-81d24b2a4d57\") " pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:40.056010 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.055830    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-system-cni-dir\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.056010 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.055860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70b33bfb-c309-4092-9e67-f09a30673a7d-cni-binary-copy\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.056010 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.055885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-run-multus-certs\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.056010 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.055924    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0bf9977-9b65-4b22-958e-46b52f65079e-kubelet-config\") pod \"global-pull-secret-syncer-tk7lm\" (UID: \"e0bf9977-9b65-4b22-958e-46b52f65079e\") " pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:40.056010 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.055947    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-cni-bin\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.056010 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.055949    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-system-cni-dir\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.056010 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.055995    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-cni-bin\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.056409 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056015    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-run-multus-certs\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.056409 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056053    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0bf9977-9b65-4b22-958e-46b52f65079e-kubelet-config\") pod \"global-pull-secret-syncer-tk7lm\" (UID: \"e0bf9977-9b65-4b22-958e-46b52f65079e\") " pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:40.056409 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056131    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-systemd-units\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.056409 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056170    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-run-netns\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.056409 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056187    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-systemd-units\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.056409 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056199    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa5caa93-f632-4e94-bfe6-4c2a54d9bf71-iptables-alerter-script\") pod \"iptables-alerter-czjtg\" (UID: \"aa5caa93-f632-4e94-bfe6-4c2a54d9bf71\") " pod="openshift-network-operator/iptables-alerter-czjtg"
Sep 07 10:11:40.056409 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056226    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-multus-socket-dir-parent\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.056409 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056235    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-run-netns\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.056409 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056265    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac0c45e5-27f8-4c26-929a-001e4b644d46-hosts-file\") pod \"node-resolver-qjnz6\" (UID: \"ac0c45e5-27f8-4c26-929a-001e4b644d46\") " pod="openshift-dns/node-resolver-qjnz6"
Sep 07 10:11:40.056409 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056288    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-multus-socket-dir-parent\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.056409 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056293    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-registration-dir\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:40.056409 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056319    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-multus-conf-dir\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.056409 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056363    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-multus-conf-dir\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.056409 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056411    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac0c45e5-27f8-4c26-929a-001e4b644d46-hosts-file\") pod \"node-resolver-qjnz6\" (UID: \"ac0c45e5-27f8-4c26-929a-001e4b644d46\") " pod="openshift-dns/node-resolver-qjnz6"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056430    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-registration-dir\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-etc-openvswitch\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056488    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-node-log\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056493    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-etc-openvswitch\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056494    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70b33bfb-c309-4092-9e67-f09a30673a7d-cni-binary-copy\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056534    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056542    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-node-log\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-cnibin\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056605    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056645    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70b33bfb-c309-4092-9e67-f09a30673a7d-multus-daemon-config\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056697    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-cnibin\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056715    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-run-ovn\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0bf9977-9b65-4b22-958e-46b52f65079e-dbus\") pod \"global-pull-secret-syncer-tk7lm\" (UID: \"e0bf9977-9b65-4b22-958e-46b52f65079e\") " pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056816    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-run-ovn\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056821    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-log-socket\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056827    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa5caa93-f632-4e94-bfe6-4c2a54d9bf71-iptables-alerter-script\") pod \"iptables-alerter-czjtg\" (UID: \"aa5caa93-f632-4e94-bfe6-4c2a54d9bf71\") " pod="openshift-network-operator/iptables-alerter-czjtg"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056855    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-env-overrides\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.057123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056860    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-log-socket\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f37b4a59-9c92-41fd-9561-ed2d017c6751-serviceca\") pod \"node-ca-v86rj\" (UID: \"f37b4a59-9c92-41fd-9561-ed2d017c6751\") " pod="openshift-image-registry/node-ca-v86rj"
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056901    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0bf9977-9b65-4b22-958e-46b52f65079e-dbus\") pod \"global-pull-secret-syncer-tk7lm\" (UID: \"e0bf9977-9b65-4b22-958e-46b52f65079e\") " pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-sys-fs\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056950    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret\") pod \"global-pull-secret-syncer-tk7lm\" (UID: \"e0bf9977-9b65-4b22-958e-46b52f65079e\") " pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.056984    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-sys-fs\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057007    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzvmt\" (UniqueName: \"kubernetes.io/projected/aa5caa93-f632-4e94-bfe6-4c2a54d9bf71-kube-api-access-zzvmt\") pod \"iptables-alerter-czjtg\" (UID: \"aa5caa93-f632-4e94-bfe6-4c2a54d9bf71\") " pod="openshift-network-operator/iptables-alerter-czjtg"
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pxk8\" (UniqueName: \"kubernetes.io/projected/f37b4a59-9c92-41fd-9561-ed2d017c6751-kube-api-access-7pxk8\") pod \"node-ca-v86rj\" (UID: \"f37b4a59-9c92-41fd-9561-ed2d017c6751\") " pod="openshift-image-registry/node-ca-v86rj"
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-var-lib-kubelet\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:40.057078    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-etc-kubernetes\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057077    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057113    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-ovnkube-script-lib\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:40.057136    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret podName:e0bf9977-9b65-4b22-958e-46b52f65079e nodeName:}" failed. No retries permitted until 2026-09-07 10:11:40.557117263 +0000 UTC m=+3.054807166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret") pod "global-pull-secret-syncer-tk7lm" (UID: "e0bf9977-9b65-4b22-958e-46b52f65079e") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057136    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70b33bfb-c309-4092-9e67-f09a30673a7d-multus-daemon-config\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057173    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f37b4a59-9c92-41fd-9561-ed2d017c6751-host\") pod \"node-ca-v86rj\" (UID: \"f37b4a59-9c92-41fd-9561-ed2d017c6751\") " pod="openshift-image-registry/node-ca-v86rj"
Sep 07 10:11:40.058079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057205    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-ovn-node-metrics-cert\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-etc-selinux\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057254    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-multus-cni-dir\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057289    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-var-lib-openvswitch\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057315    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ckqb\" (UniqueName: \"kubernetes.io/projected/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-kube-api-access-2ckqb\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057325    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f37b4a59-9c92-41fd-9561-ed2d017c6751-serviceca\") pod \"node-ca-v86rj\" (UID: \"f37b4a59-9c92-41fd-9561-ed2d017c6751\") " pod="openshift-image-registry/node-ca-v86rj"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057340    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-run-netns\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057349    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-env-overrides\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057380    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-var-lib-cni-bin\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057385    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-var-lib-kubelet\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057401    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-var-lib-openvswitch\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057419    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-var-lib-cni-multus\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f37b4a59-9c92-41fd-9561-ed2d017c6751-host\") pod \"node-ca-v86rj\" (UID: \"f37b4a59-9c92-41fd-9561-ed2d017c6751\") " pod="openshift-image-registry/node-ca-v86rj"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057488    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-slash\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057491    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-var-lib-cni-multus\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057502    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-etc-selinux\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057514    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-var-lib-cni-bin\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057457    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-slash\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.058871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa5caa93-f632-4e94-bfe6-4c2a54d9bf71-host-slash\") pod \"iptables-alerter-czjtg\" (UID: \"aa5caa93-f632-4e94-bfe6-4c2a54d9bf71\") " pod="openshift-network-operator/iptables-alerter-czjtg"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057546    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-run-netns\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057558    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0c45e5-27f8-4c26-929a-001e4b644d46-tmp-dir\") pod \"node-resolver-qjnz6\" (UID: \"ac0c45e5-27f8-4c26-929a-001e4b644d46\") " pod="openshift-dns/node-resolver-qjnz6"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057578    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-run-k8s-cni-cncf-io\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057584    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-etc-kubernetes\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057547    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-multus-cni-dir\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057586    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa5caa93-f632-4e94-bfe6-4c2a54d9bf71-host-slash\") pod \"iptables-alerter-czjtg\" (UID: \"aa5caa93-f632-4e94-bfe6-4c2a54d9bf71\") " pod="openshift-network-operator/iptables-alerter-czjtg"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057647    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-ovnkube-script-lib\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057689    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-hostroot\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057710    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-host-run-k8s-cni-cncf-io\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057727    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-run-openvswitch\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057751    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqdp\" (UniqueName: \"kubernetes.io/projected/ac0c45e5-27f8-4c26-929a-001e4b644d46-kube-api-access-cbqdp\") pod \"node-resolver-qjnz6\" (UID: \"ac0c45e5-27f8-4c26-929a-001e4b644d46\") " pod="openshift-dns/node-resolver-qjnz6"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057762    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70b33bfb-c309-4092-9e67-f09a30673a7d-hostroot\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057791    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz2rs\" (UniqueName: \"kubernetes.io/projected/70b33bfb-c309-4092-9e67-f09a30673a7d-kube-api-access-pz2rs\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057911    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0c45e5-27f8-4c26-929a-001e4b644d46-tmp-dir\") pod \"node-resolver-qjnz6\" (UID: \"ac0c45e5-27f8-4c26-929a-001e4b644d46\") " pod="openshift-dns/node-resolver-qjnz6"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.057964    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-run-openvswitch\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.058006    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.058046    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-socket-dir\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:40.059602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.058049    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.060280 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.058076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2nl5\" (UniqueName: \"kubernetes.io/projected/86e562c2-10c5-463c-a528-2d9f9c54737d-kube-api-access-r2nl5\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:40.060280 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.058103    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-kubelet\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.060280 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.058129    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-run-systemd\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.060280 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.058174    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-run-systemd\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.060280 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.058208    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-socket-dir\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:40.060280 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.058219    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-kubelet\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.060280 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.058249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-cni-netd\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.060280 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.058274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-ovnkube-config\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.060280 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.058299    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-device-dir\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:40.060280 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.058330    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-host-cni-netd\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.060280 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.058374    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/86e562c2-10c5-463c-a528-2d9f9c54737d-device-dir\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:40.060280 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.058751    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-ovnkube-config\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.060280 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.060030    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-ovn-node-metrics-cert\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.061966 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:40.061944    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:11:40.062097 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:40.061970    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:11:40.062097 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:40.061984    2562 projected.go:196] Error preparing data for projected volume kube-api-access-q6xd6 for pod openshift-network-diagnostics/network-check-target-q2724: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:40.062097 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:40.062051    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6 podName:948c00ef-80eb-4ff1-95c8-81d24b2a4d57 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:40.562031138 +0000 UTC m=+3.059721029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q6xd6" (UniqueName: "kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6") pod "network-check-target-q2724" (UID: "948c00ef-80eb-4ff1-95c8-81d24b2a4d57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:40.071509 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.071449    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pxk8\" (UniqueName: \"kubernetes.io/projected/f37b4a59-9c92-41fd-9561-ed2d017c6751-kube-api-access-7pxk8\") pod \"node-ca-v86rj\" (UID: \"f37b4a59-9c92-41fd-9561-ed2d017c6751\") " pod="openshift-image-registry/node-ca-v86rj"
Sep 07 10:11:40.072436 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.072402    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbqdp\" (UniqueName: \"kubernetes.io/projected/ac0c45e5-27f8-4c26-929a-001e4b644d46-kube-api-access-cbqdp\") pod \"node-resolver-qjnz6\" (UID: \"ac0c45e5-27f8-4c26-929a-001e4b644d46\") " pod="openshift-dns/node-resolver-qjnz6"
Sep 07 10:11:40.072891 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.072871    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ckqb\" (UniqueName: \"kubernetes.io/projected/db9c06dd-e0b0-41c6-8f43-3fedcde253c0-kube-api-access-2ckqb\") pod \"ovnkube-node-959sz\" (UID: \"db9c06dd-e0b0-41c6-8f43-3fedcde253c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.072984 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.072904    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzvmt\" (UniqueName: \"kubernetes.io/projected/aa5caa93-f632-4e94-bfe6-4c2a54d9bf71-kube-api-access-zzvmt\") pod \"iptables-alerter-czjtg\" (UID: \"aa5caa93-f632-4e94-bfe6-4c2a54d9bf71\") " pod="openshift-network-operator/iptables-alerter-czjtg"
Sep 07 10:11:40.073523 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.073498    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz2rs\" (UniqueName: \"kubernetes.io/projected/70b33bfb-c309-4092-9e67-f09a30673a7d-kube-api-access-pz2rs\") pod \"multus-tmj6t\" (UID: \"70b33bfb-c309-4092-9e67-f09a30673a7d\") " pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.074255 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.074232    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2nl5\" (UniqueName: \"kubernetes.io/projected/86e562c2-10c5-463c-a528-2d9f9c54737d-kube-api-access-r2nl5\") pod \"aws-ebs-csi-driver-node-zbtcl\" (UID: \"86e562c2-10c5-463c-a528-2d9f9c54737d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:40.139495 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.139452    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z9jpp"
Sep 07 10:11:40.153190 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.153170    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6kdt9"
Sep 07 10:11:40.164255 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.164231    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xgmhs"
Sep 07 10:11:40.169103 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.169079    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl"
Sep 07 10:11:40.177067 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.177048    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:11:40.185002 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.184980    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tmj6t"
Sep 07 10:11:40.192705 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.192686    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-czjtg"
Sep 07 10:11:40.197687 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.197669    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:11:40.199720 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.199699    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qjnz6"
Sep 07 10:11:40.206794 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.206776    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v86rj"
Sep 07 10:11:40.461955 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.461858    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs\") pod \"network-metrics-daemon-l7n5z\" (UID: \"7254bd39-6628-42fd-96e9-6a01b723c275\") " pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:40.462132 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:40.462000    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:40.462132 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:40.462085    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs podName:7254bd39-6628-42fd-96e9-6a01b723c275 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:41.46206247 +0000 UTC m=+3.959752364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs") pod "network-metrics-daemon-l7n5z" (UID: "7254bd39-6628-42fd-96e9-6a01b723c275") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:40.562659 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.562616    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret\") pod \"global-pull-secret-syncer-tk7lm\" (UID: \"e0bf9977-9b65-4b22-958e-46b52f65079e\") " pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:40.562843 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.562705    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6xd6\" (UniqueName: \"kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6\") pod \"network-check-target-q2724\" (UID: \"948c00ef-80eb-4ff1-95c8-81d24b2a4d57\") " pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:40.562843 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:40.562789    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:40.562971 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:40.562849    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:11:40.562971 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:40.562872    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:11:40.562971 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:40.562888    2562 projected.go:196] Error preparing data for projected volume kube-api-access-q6xd6 for pod openshift-network-diagnostics/network-check-target-q2724: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:40.562971 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:40.562873    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret podName:e0bf9977-9b65-4b22-958e-46b52f65079e nodeName:}" failed. No retries permitted until 2026-09-07 10:11:41.562850507 +0000 UTC m=+4.060540414 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret") pod "global-pull-secret-syncer-tk7lm" (UID: "e0bf9977-9b65-4b22-958e-46b52f65079e") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:40.562971 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:40.562948    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6 podName:948c00ef-80eb-4ff1-95c8-81d24b2a4d57 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:41.56293238 +0000 UTC m=+4.060622275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6xd6" (UniqueName: "kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6") pod "network-check-target-q2724" (UID: "948c00ef-80eb-4ff1-95c8-81d24b2a4d57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:40.930849 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:40.930658    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757b7332_c3c9_4849_a0b7_cf8a4d46b3c1.slice/crio-eda512e613700b5fd4093a5f95b05f71a5c272bd61d696171b198deff7063a9b WatchSource:0}: Error finding container eda512e613700b5fd4093a5f95b05f71a5c272bd61d696171b198deff7063a9b: Status 404 returned error can't find the container with id eda512e613700b5fd4093a5f95b05f71a5c272bd61d696171b198deff7063a9b
Sep 07 10:11:40.932119 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:40.932097    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba3d830_3508_49a6_8d41_36409f735098.slice/crio-b2e8d5196ed03f0e3ef67f9724f82fea28b06940ad0f1b55d8b6176d1a5723db WatchSource:0}: Error finding container b2e8d5196ed03f0e3ef67f9724f82fea28b06940ad0f1b55d8b6176d1a5723db: Status 404 returned error can't find the container with id b2e8d5196ed03f0e3ef67f9724f82fea28b06940ad0f1b55d8b6176d1a5723db
Sep 07 10:11:40.933250 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:40.932879    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b33bfb_c309_4092_9e67_f09a30673a7d.slice/crio-92bfc2817513cd7058d41127e9d02a57e91165bf86b10eb63bb5eda28fceb625 WatchSource:0}: Error finding container 92bfc2817513cd7058d41127e9d02a57e91165bf86b10eb63bb5eda28fceb625: Status 404 returned error can't find the container with id 92bfc2817513cd7058d41127e9d02a57e91165bf86b10eb63bb5eda28fceb625
Sep 07 10:11:40.936070 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:40.936047    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9c06dd_e0b0_41c6_8f43_3fedcde253c0.slice/crio-bdbb7e45a1e67c6e22465f6d7c70a4d12b9eaea1be7337771e87721dc04ba354 WatchSource:0}: Error finding container bdbb7e45a1e67c6e22465f6d7c70a4d12b9eaea1be7337771e87721dc04ba354: Status 404 returned error can't find the container with id bdbb7e45a1e67c6e22465f6d7c70a4d12b9eaea1be7337771e87721dc04ba354
Sep 07 10:11:40.936697 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:40.936603    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0c45e5_27f8_4c26_929a_001e4b644d46.slice/crio-cbd3f1aacb99b506d5f37b0d421a613da5e63adec5ae63c120cf7eb85bfc9f73 WatchSource:0}: Error finding container cbd3f1aacb99b506d5f37b0d421a613da5e63adec5ae63c120cf7eb85bfc9f73: Status 404 returned error can't find the container with id cbd3f1aacb99b506d5f37b0d421a613da5e63adec5ae63c120cf7eb85bfc9f73
Sep 07 10:11:40.937981 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:40.937960    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf37b4a59_9c92_41fd_9561_ed2d017c6751.slice/crio-73ba14c53d211ab1cdaeb4ee7291f20ac8015bcdfad302424710495690b2c39d WatchSource:0}: Error finding container 73ba14c53d211ab1cdaeb4ee7291f20ac8015bcdfad302424710495690b2c39d: Status 404 returned error can't find the container with id 73ba14c53d211ab1cdaeb4ee7291f20ac8015bcdfad302424710495690b2c39d
Sep 07 10:11:40.938913 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:11:40.938878    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5caa93_f632_4e94_bfe6_4c2a54d9bf71.slice/crio-a9f417c3174b09de665d2c1f8b7220d35c2e2309ad7312701c6e62d6a0416c8c WatchSource:0}: Error finding container a9f417c3174b09de665d2c1f8b7220d35c2e2309ad7312701c6e62d6a0416c8c: Status 404 returned error can't find the container with id a9f417c3174b09de665d2c1f8b7220d35c2e2309ad7312701c6e62d6a0416c8c
Sep 07 10:11:40.966317 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.966287    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 10:06:40 +0000 UTC" deadline="2028-02-13 00:45:15.552337818 +0000 UTC"
Sep 07 10:11:40.966317 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.966316    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12566h33m34.58602557s"
Sep 07 10:11:40.968002 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.967978    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z9jpp" event={"ID":"8284b473-d11f-48e6-860e-277051c2086e","Type":"ContainerStarted","Data":"6b3d7c41a3ef7f85016b519a1ff6f8f9254b456a1a18f08152b2121a60d74ec1"}
Sep 07 10:11:40.974597 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.974577    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl" event={"ID":"86e562c2-10c5-463c-a528-2d9f9c54737d","Type":"ContainerStarted","Data":"0021ea8ce4e18870dd692938cca12416c5039bb3865f856e4f9a5e2cfb1af66f"}
Sep 07 10:11:40.979126 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.979104    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-czjtg" event={"ID":"aa5caa93-f632-4e94-bfe6-4c2a54d9bf71","Type":"ContainerStarted","Data":"a9f417c3174b09de665d2c1f8b7220d35c2e2309ad7312701c6e62d6a0416c8c"}
Sep 07 10:11:40.985878 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.985849    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qjnz6" event={"ID":"ac0c45e5-27f8-4c26-929a-001e4b644d46","Type":"ContainerStarted","Data":"cbd3f1aacb99b506d5f37b0d421a613da5e63adec5ae63c120cf7eb85bfc9f73"}
Sep 07 10:11:40.987073 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.987053    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-959sz" event={"ID":"db9c06dd-e0b0-41c6-8f43-3fedcde253c0","Type":"ContainerStarted","Data":"bdbb7e45a1e67c6e22465f6d7c70a4d12b9eaea1be7337771e87721dc04ba354"}
Sep 07 10:11:40.988146 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.988123    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tmj6t" event={"ID":"70b33bfb-c309-4092-9e67-f09a30673a7d","Type":"ContainerStarted","Data":"92bfc2817513cd7058d41127e9d02a57e91165bf86b10eb63bb5eda28fceb625"}
Sep 07 10:11:40.989358 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.989334    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v86rj" event={"ID":"f37b4a59-9c92-41fd-9561-ed2d017c6751","Type":"ContainerStarted","Data":"73ba14c53d211ab1cdaeb4ee7291f20ac8015bcdfad302424710495690b2c39d"}
Sep 07 10:11:40.990407 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.990384    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xgmhs" event={"ID":"6ba3d830-3508-49a6-8d41-36409f735098","Type":"ContainerStarted","Data":"b2e8d5196ed03f0e3ef67f9724f82fea28b06940ad0f1b55d8b6176d1a5723db"}
Sep 07 10:11:40.991496 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:40.991478    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kdt9" event={"ID":"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1","Type":"ContainerStarted","Data":"eda512e613700b5fd4093a5f95b05f71a5c272bd61d696171b198deff7063a9b"}
Sep 07 10:11:41.174715 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:41.174692    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:11:41.469830 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:41.469736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs\") pod \"network-metrics-daemon-l7n5z\" (UID: \"7254bd39-6628-42fd-96e9-6a01b723c275\") " pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:41.469996 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:41.469913    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:41.469996 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:41.469979    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs podName:7254bd39-6628-42fd-96e9-6a01b723c275 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:43.469959092 +0000 UTC m=+5.967648995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs") pod "network-metrics-daemon-l7n5z" (UID: "7254bd39-6628-42fd-96e9-6a01b723c275") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:41.570868 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:41.570831    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6xd6\" (UniqueName: \"kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6\") pod \"network-check-target-q2724\" (UID: \"948c00ef-80eb-4ff1-95c8-81d24b2a4d57\") " pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:41.571056 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:41.570899    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret\") pod \"global-pull-secret-syncer-tk7lm\" (UID: \"e0bf9977-9b65-4b22-958e-46b52f65079e\") " pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:41.571056 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:41.571018    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:41.571165 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:41.571155    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:11:41.571215 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:41.571173    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:11:41.571215 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:41.571187    2562 projected.go:196] Error preparing data for projected volume kube-api-access-q6xd6 for pod openshift-network-diagnostics/network-check-target-q2724: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:41.571315 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:41.571242    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6 podName:948c00ef-80eb-4ff1-95c8-81d24b2a4d57 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:43.571223316 +0000 UTC m=+6.068913206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6xd6" (UniqueName: "kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6") pod "network-check-target-q2724" (UID: "948c00ef-80eb-4ff1-95c8-81d24b2a4d57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:41.571315 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:41.571271    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret podName:e0bf9977-9b65-4b22-958e-46b52f65079e nodeName:}" failed. No retries permitted until 2026-09-07 10:11:43.571259645 +0000 UTC m=+6.068949551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret") pod "global-pull-secret-syncer-tk7lm" (UID: "e0bf9977-9b65-4b22-958e-46b52f65079e") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:41.960906 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:41.960880    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:41.961549 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:41.961004    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:11:41.961549 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:41.961384    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:41.961549 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:41.961479    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:11:41.961719 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:41.961595    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:41.961719 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:41.961691    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:11:41.966675 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:41.966461    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 10:06:40 +0000 UTC" deadline="2028-04-29 04:57:12.108267718 +0000 UTC"
Sep 07 10:11:41.966675 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:41.966507    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14394h45m30.141764359s"
Sep 07 10:11:42.006037 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:42.006000    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-62.ec2.internal" event={"ID":"236f13cb4cc1d42288493688c2e3dab9","Type":"ContainerStarted","Data":"e260575286403f0f30abef26bbadaf4240126212f9580bed186dd0654dd45cd6"}
Sep 07 10:11:42.011571 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:42.011478    2562 generic.go:356] "Generic (PLEG): container finished" podID="5d9182b821ba5135386976925f2d8116" containerID="b91dd38c0990cd45018feccfc86a27391025156cf8ae2a7a5b515022d1ca8b86" exitCode=0
Sep 07 10:11:42.011571 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:42.011528    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal" event={"ID":"5d9182b821ba5135386976925f2d8116","Type":"ContainerDied","Data":"b91dd38c0990cd45018feccfc86a27391025156cf8ae2a7a5b515022d1ca8b86"}
Sep 07 10:11:42.051823 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:42.050200    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-62.ec2.internal" podStartSLOduration=3.050179089 podStartE2EDuration="3.050179089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:42.029139044 +0000 UTC m=+4.526828954" watchObservedRunningTime="2026-09-07 10:11:42.050179089 +0000 UTC m=+4.547868999"
Sep 07 10:11:43.052062 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:43.051589    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal" event={"ID":"5d9182b821ba5135386976925f2d8116","Type":"ContainerStarted","Data":"59f86b813e446e210c3e21d88aec1011741d75d3c06b95097c770cee458102ac"}
Sep 07 10:11:43.080203 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:43.080147    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-62.ec2.internal" podStartSLOduration=4.080125944 podStartE2EDuration="4.080125944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:43.079860858 +0000 UTC m=+5.577550792" watchObservedRunningTime="2026-09-07 10:11:43.080125944 +0000 UTC m=+5.577815856"
Sep 07 10:11:43.489172 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:43.488598    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs\") pod \"network-metrics-daemon-l7n5z\" (UID: \"7254bd39-6628-42fd-96e9-6a01b723c275\") " pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:43.489172 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:43.488774    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:43.489172 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:43.488835    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs podName:7254bd39-6628-42fd-96e9-6a01b723c275 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:47.488817139 +0000 UTC m=+9.986507049 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs") pod "network-metrics-daemon-l7n5z" (UID: "7254bd39-6628-42fd-96e9-6a01b723c275") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:43.590498 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:43.589679    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret\") pod \"global-pull-secret-syncer-tk7lm\" (UID: \"e0bf9977-9b65-4b22-958e-46b52f65079e\") " pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:43.590498 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:43.589814    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6xd6\" (UniqueName: \"kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6\") pod \"network-check-target-q2724\" (UID: \"948c00ef-80eb-4ff1-95c8-81d24b2a4d57\") " pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:43.590498 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:43.589965    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:11:43.590498 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:43.589983    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:11:43.590498 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:43.589995    2562 projected.go:196] Error preparing data for projected volume kube-api-access-q6xd6 for pod openshift-network-diagnostics/network-check-target-q2724: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:43.590498 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:43.590050    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6 podName:948c00ef-80eb-4ff1-95c8-81d24b2a4d57 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:47.590032891 +0000 UTC m=+10.087722792 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6xd6" (UniqueName: "kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6") pod "network-check-target-q2724" (UID: "948c00ef-80eb-4ff1-95c8-81d24b2a4d57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:43.590498 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:43.590419    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:43.590498 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:43.590464    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret podName:e0bf9977-9b65-4b22-958e-46b52f65079e nodeName:}" failed. No retries permitted until 2026-09-07 10:11:47.590449223 +0000 UTC m=+10.088139148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret") pod "global-pull-secret-syncer-tk7lm" (UID: "e0bf9977-9b65-4b22-958e-46b52f65079e") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:43.959267 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:43.958477    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:43.959267 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:43.958624    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:11:43.959267 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:43.959016    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:43.959267 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:43.959132    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:11:43.962080 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:43.962033    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:43.962338 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:43.962303    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:11:45.964223 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:45.964190    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:45.964664 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:45.964323    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:11:45.964737 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:45.964684    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:45.964824 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:45.964800    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:11:45.964925 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:45.964852    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:45.964925 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:45.964914    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:11:47.525988 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:47.525956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs\") pod \"network-metrics-daemon-l7n5z\" (UID: \"7254bd39-6628-42fd-96e9-6a01b723c275\") " pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:47.526424 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:47.526145    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:47.526424 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:47.526221    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs podName:7254bd39-6628-42fd-96e9-6a01b723c275 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:55.526199815 +0000 UTC m=+18.023889720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs") pod "network-metrics-daemon-l7n5z" (UID: "7254bd39-6628-42fd-96e9-6a01b723c275") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:47.627475 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:47.627029    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6xd6\" (UniqueName: \"kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6\") pod \"network-check-target-q2724\" (UID: \"948c00ef-80eb-4ff1-95c8-81d24b2a4d57\") " pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:47.627475 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:47.627099    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret\") pod \"global-pull-secret-syncer-tk7lm\" (UID: \"e0bf9977-9b65-4b22-958e-46b52f65079e\") " pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:47.627475 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:47.627247    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:47.627475 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:47.627315    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret podName:e0bf9977-9b65-4b22-958e-46b52f65079e nodeName:}" failed. No retries permitted until 2026-09-07 10:11:55.627296947 +0000 UTC m=+18.124986854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret") pod "global-pull-secret-syncer-tk7lm" (UID: "e0bf9977-9b65-4b22-958e-46b52f65079e") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:47.627475 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:47.627314    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:11:47.627475 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:47.627340    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:11:47.627475 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:47.627353    2562 projected.go:196] Error preparing data for projected volume kube-api-access-q6xd6 for pod openshift-network-diagnostics/network-check-target-q2724: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:47.627475 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:47.627397    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6 podName:948c00ef-80eb-4ff1-95c8-81d24b2a4d57 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:55.627382451 +0000 UTC m=+18.125072339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6xd6" (UniqueName: "kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6") pod "network-check-target-q2724" (UID: "948c00ef-80eb-4ff1-95c8-81d24b2a4d57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:47.960379 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:47.959882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:47.960379 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:47.960000    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:11:47.960379 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:47.960087    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:47.960379 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:47.960210    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:11:47.960379 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:47.960259    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:47.960379 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:47.960344    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:11:49.959598 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:49.959350    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:49.960599 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:49.960208    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:49.960599 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:49.960323    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:11:49.960599 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:49.960344    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:49.960599 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:49.960468    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:11:49.960599 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:49.960557    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:11:51.958990 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:51.958950    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:51.959431 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:51.958950    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:51.959431 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:51.958955    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:51.959431 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:51.959133    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:11:51.959431 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:51.959263    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:11:51.959431 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:51.959351    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:11:53.958313 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:53.958275    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:53.958805 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:53.958275    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:53.958805 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:53.958431    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:11:53.958805 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:53.958285    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:53.958805 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:53.958484    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:11:53.958805 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:53.958579    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:11:55.589090 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:55.589054    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs\") pod \"network-metrics-daemon-l7n5z\" (UID: \"7254bd39-6628-42fd-96e9-6a01b723c275\") " pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:55.589531 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:55.589228    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:55.589531 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:55.589299    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs podName:7254bd39-6628-42fd-96e9-6a01b723c275 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:11.589277429 +0000 UTC m=+34.086967320 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs") pod "network-metrics-daemon-l7n5z" (UID: "7254bd39-6628-42fd-96e9-6a01b723c275") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:55.690268 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:55.690230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret\") pod \"global-pull-secret-syncer-tk7lm\" (UID: \"e0bf9977-9b65-4b22-958e-46b52f65079e\") " pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:55.690432 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:55.690297    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6xd6\" (UniqueName: \"kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6\") pod \"network-check-target-q2724\" (UID: \"948c00ef-80eb-4ff1-95c8-81d24b2a4d57\") " pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:55.690432 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:55.690392    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:55.690432 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:55.690406    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:11:55.690432 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:55.690428    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:11:55.690610 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:55.690442    2562 projected.go:196] Error preparing data for projected volume kube-api-access-q6xd6 for pod openshift-network-diagnostics/network-check-target-q2724: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:55.690610 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:55.690469    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret podName:e0bf9977-9b65-4b22-958e-46b52f65079e nodeName:}" failed. No retries permitted until 2026-09-07 10:12:11.690453435 +0000 UTC m=+34.188143343 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret") pod "global-pull-secret-syncer-tk7lm" (UID: "e0bf9977-9b65-4b22-958e-46b52f65079e") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:55.690610 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:55.690486    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6 podName:948c00ef-80eb-4ff1-95c8-81d24b2a4d57 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:11.690479907 +0000 UTC m=+34.188169794 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6xd6" (UniqueName: "kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6") pod "network-check-target-q2724" (UID: "948c00ef-80eb-4ff1-95c8-81d24b2a4d57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:55.958576 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:55.958484    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:55.958576 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:55.958536    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:55.958824 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:55.958484    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:55.958824 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:55.958625    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:11:55.958824 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:55.958718    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:11:55.958824 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:55.958809    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:11:57.958798 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:57.958745    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:57.959248 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:57.958875    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:11:57.959248 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:57.958882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:57.959248 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:57.958967    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:11:57.959248 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:57.958975    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:57.959248 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:57.959066    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:11:59.088838 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.088651    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z9jpp" event={"ID":"8284b473-d11f-48e6-860e-277051c2086e","Type":"ContainerStarted","Data":"125e46d9e55a3a0e110cd9812906848a2470a15bc6b463d54b8fa6801de8db6f"}
Sep 07 10:11:59.090110 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.090088    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl" event={"ID":"86e562c2-10c5-463c-a528-2d9f9c54737d","Type":"ContainerStarted","Data":"ea0a99563ef57cb400722cb71e449c79176378aa4669d5a3ececa43d2365efd6"}
Sep 07 10:11:59.091467 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.091448    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qjnz6" event={"ID":"ac0c45e5-27f8-4c26-929a-001e4b644d46","Type":"ContainerStarted","Data":"54603a008ce897d5957de5b94ecaae86f679214efde520031a3a43d1909cc069"}
Sep 07 10:11:59.093756 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.093735    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-959sz" event={"ID":"db9c06dd-e0b0-41c6-8f43-3fedcde253c0","Type":"ContainerStarted","Data":"69141fd03df75ccae5c6bc5005b1e0a6d2e4d42ddb5d33f897e0a4912eb8c04b"}
Sep 07 10:11:59.093852 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.093809    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-959sz" event={"ID":"db9c06dd-e0b0-41c6-8f43-3fedcde253c0","Type":"ContainerStarted","Data":"5c1c17be8ce372bc31d3e988d144b3e5d2c893a980136f935156026623db6d9e"}
Sep 07 10:11:59.093897 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.093852    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-959sz" event={"ID":"db9c06dd-e0b0-41c6-8f43-3fedcde253c0","Type":"ContainerStarted","Data":"b6f7057c4084744ddf6d86fe15aa742fdba8e1910db0e51b919d4f8b3db4059e"}
Sep 07 10:11:59.093897 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.093882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-959sz" event={"ID":"db9c06dd-e0b0-41c6-8f43-3fedcde253c0","Type":"ContainerStarted","Data":"00e4c8dc84b602d5ccacfb1d1df1cff0fbf750fb3d07f48eb11f5d462f700c8e"}
Sep 07 10:11:59.095080 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.095062    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tmj6t" event={"ID":"70b33bfb-c309-4092-9e67-f09a30673a7d","Type":"ContainerStarted","Data":"55f4d80a228507a7c585595dc08aac64f63e902c01fd4693b0c8a0ef91a192fb"}
Sep 07 10:11:59.096469 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.096451    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v86rj" event={"ID":"f37b4a59-9c92-41fd-9561-ed2d017c6751","Type":"ContainerStarted","Data":"d75430dcc45f1da196c09ef6f9805a77787bd4d13ff74b2832c493429c125fc0"}
Sep 07 10:11:59.097804 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.097757    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xgmhs" event={"ID":"6ba3d830-3508-49a6-8d41-36409f735098","Type":"ContainerStarted","Data":"7b85a2f8ace5b91a8b5ce43af58dba67af82145b95405a77ee8e6b00ab5e8a64"}
Sep 07 10:11:59.099167 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.099144    2562 generic.go:356] "Generic (PLEG): container finished" podID="757b7332-c3c9-4849-a0b7-cf8a4d46b3c1" containerID="ef5db5421cf849d060ef6d2af6b4946e5de1ae37250f9f6830d1e0bde18d5a66" exitCode=0
Sep 07 10:11:59.099258 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.099178    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kdt9" event={"ID":"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1","Type":"ContainerDied","Data":"ef5db5421cf849d060ef6d2af6b4946e5de1ae37250f9f6830d1e0bde18d5a66"}
Sep 07 10:11:59.103857 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.103824    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z9jpp" podStartSLOduration=3.48869278 podStartE2EDuration="21.103813s" totalImagesPullingTime="17.61512022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:59.103612225 +0000 UTC m=+21.601302134" watchObservedRunningTime="2026-09-07 10:11:59.103813 +0000 UTC m=+21.601502909"
Sep 07 10:11:59.131591 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.131552    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qjnz6" podStartSLOduration=3.515823445 podStartE2EDuration="21.131531983s" totalImagesPullingTime="17.615708538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:59.118183509 +0000 UTC m=+21.615873428" watchObservedRunningTime="2026-09-07 10:11:59.131531983 +0000 UTC m=+21.629221940"
Sep 07 10:11:59.131824 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.131802    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xgmhs" podStartSLOduration=10.922307659 podStartE2EDuration="21.131796788s" totalImagesPullingTime="10.209489129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:59.131184926 +0000 UTC m=+21.628874834" watchObservedRunningTime="2026-09-07 10:11:59.131796788 +0000 UTC m=+21.629486696"
Sep 07 10:11:59.147428 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.147389    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tmj6t" podStartSLOduration=3.484804279 podStartE2EDuration="21.147375526s" totalImagesPullingTime="17.662571247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:59.147162824 +0000 UTC m=+21.644852764" watchObservedRunningTime="2026-09-07 10:11:59.147375526 +0000 UTC m=+21.645065435"
Sep 07 10:11:59.161053 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.160993    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v86rj" podStartSLOduration=3.5580978119999997 podStartE2EDuration="21.16098019s" totalImagesPullingTime="17.602882378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:59.160474706 +0000 UTC m=+21.658164641" watchObservedRunningTime="2026-09-07 10:11:59.16098019 +0000 UTC m=+21.658670099"
Sep 07 10:11:59.958325 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.958297    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:11:59.958449 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.958415    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:11:59.958449 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:59.958416    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:11:59.958619 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:59.958519    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:11:59.958619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:11:59.958298    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:11:59.958720 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:11:59.958622    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:12:00.044655 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:00.044605    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 07 10:12:00.103778 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:00.103679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl" event={"ID":"86e562c2-10c5-463c-a528-2d9f9c54737d","Type":"ContainerStarted","Data":"d28db51fc2d6908b53d12475d3ff528212cea1fb8d6d736f73cf4cc12d33b385"}
Sep 07 10:12:00.105424 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:00.105392    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-czjtg" event={"ID":"aa5caa93-f632-4e94-bfe6-4c2a54d9bf71","Type":"ContainerStarted","Data":"73f9852f9fe571e8cbb5efce3210315636b34aef7f9c9c205cd085195456ff71"}
Sep 07 10:12:00.109080 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:00.109043    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-959sz" event={"ID":"db9c06dd-e0b0-41c6-8f43-3fedcde253c0","Type":"ContainerStarted","Data":"ed0679cbe257d7dbdeae204d19ce92c27447d4b5d5b00fdb8d859311e7d066c4"}
Sep 07 10:12:00.109199 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:00.109119    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-959sz" event={"ID":"db9c06dd-e0b0-41c6-8f43-3fedcde253c0","Type":"ContainerStarted","Data":"b2c4935782de3c5a8d85412de5567649003a2b185c5906f2a6ffcb601621b573"}
Sep 07 10:12:00.119482 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:00.119438    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-czjtg" podStartSLOduration=4.505168074 podStartE2EDuration="22.119419872s" totalImagesPullingTime="17.614251798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:00.119001676 +0000 UTC m=+22.616691611" watchObservedRunningTime="2026-09-07 10:12:00.119419872 +0000 UTC m=+22.617109783"
Sep 07 10:12:00.906620 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:00.906502    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T10:12:00.044653121Z","UUID":"b2006eb8-1fed-4be8-bf2d-6be9738609fa","Handler":null,"Name":"","Endpoint":""}
Sep 07 10:12:00.908153 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:00.908118    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 07 10:12:00.908276 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:00.908168    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 07 10:12:01.113986 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:01.113740    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl" event={"ID":"86e562c2-10c5-463c-a528-2d9f9c54737d","Type":"ContainerStarted","Data":"6ecc2ed15a0575690e0240ede3a5bfe0add0b1ed950975df68808992514b8465"}
Sep 07 10:12:01.132044 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:01.131985    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtcl" podStartSLOduration=3.508525698 podStartE2EDuration="23.131963158s" totalImagesPullingTime="19.62343746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:01.131131723 +0000 UTC m=+23.628821652" watchObservedRunningTime="2026-09-07 10:12:01.131963158 +0000 UTC m=+23.629653069"
Sep 07 10:12:01.958591 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:01.958557    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:12:01.958591 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:01.958591    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:12:01.958850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:01.958557    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:12:01.958850 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:01.958681    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:12:01.958850 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:01.958825    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:12:01.959006 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:01.958876    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:12:02.029103 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:02.029070    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xgmhs"
Sep 07 10:12:02.029750 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:02.029728    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xgmhs"
Sep 07 10:12:02.120119 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:02.120080    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-959sz" event={"ID":"db9c06dd-e0b0-41c6-8f43-3fedcde253c0","Type":"ContainerStarted","Data":"86e1fe67b8dc34bbfa5543473211718a875016a9c01fbb158e3244a77f7512bf"}
Sep 07 10:12:02.120613 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:02.120438    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xgmhs"
Sep 07 10:12:02.120964 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:02.120947    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xgmhs"
Sep 07 10:12:03.958561 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:03.958483    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:12:03.959004 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:03.958483    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:12:03.959004 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:03.958600    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:12:03.959004 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:03.958601    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:12:03.959004 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:03.958672    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:12:03.959004 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:03.958787    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:12:05.129851 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:05.129595    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-959sz" event={"ID":"db9c06dd-e0b0-41c6-8f43-3fedcde253c0","Type":"ContainerStarted","Data":"df7dba8804fc09dcf0aff286668086349a82c27a8e2795cc60371d8e6c7c6b86"}
Sep 07 10:12:05.130334 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:05.129898    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:12:05.130334 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:05.129950    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:12:05.131589 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:05.131555    2562 generic.go:356] "Generic (PLEG): container finished" podID="757b7332-c3c9-4849-a0b7-cf8a4d46b3c1" containerID="5de86008274993b8b39a698492954363f1768801827c6715735c51969872364f" exitCode=0
Sep 07 10:12:05.131676 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:05.131589    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kdt9" event={"ID":"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1","Type":"ContainerDied","Data":"5de86008274993b8b39a698492954363f1768801827c6715735c51969872364f"}
Sep 07 10:12:05.145465 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:05.145437    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:12:05.159685 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:05.159637    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-959sz" podStartSLOduration=9.337333788 podStartE2EDuration="27.159622129s" totalImagesPullingTime="17.822288341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:05.159057613 +0000 UTC m=+27.656747541" watchObservedRunningTime="2026-09-07 10:12:05.159622129 +0000 UTC m=+27.657312038"
Sep 07 10:12:05.958565 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:05.958533    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:12:05.958947 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:05.958533    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:12:05.958947 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:05.958675    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:12:05.958947 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:05.958695    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:12:05.958947 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:05.958922    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:12:05.959183 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:05.958967    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:12:06.133954 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:06.133920    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:12:06.152246 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:06.152212    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:12:06.278732 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:06.278650    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tk7lm"]
Sep 07 10:12:06.278913 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:06.278757    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:12:06.278913 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:06.278875    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:12:06.282317 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:06.282282    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l7n5z"]
Sep 07 10:12:06.282465 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:06.282404    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:12:06.282549 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:06.282526    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:12:06.282898 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:06.282865    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q2724"]
Sep 07 10:12:06.283025 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:06.282955    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:12:06.283094 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:06.283049    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:12:07.138281 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:07.138245    2562 generic.go:356] "Generic (PLEG): container finished" podID="757b7332-c3c9-4849-a0b7-cf8a4d46b3c1" containerID="9b69eda61a5bbf96ce7a6f91a0771ea3a7ccf9d8c74950000e8e2dd40988bfa8" exitCode=0
Sep 07 10:12:07.138662 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:07.138316    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kdt9" event={"ID":"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1","Type":"ContainerDied","Data":"9b69eda61a5bbf96ce7a6f91a0771ea3a7ccf9d8c74950000e8e2dd40988bfa8"}
Sep 07 10:12:07.959737 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:07.959579    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:12:07.959930 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:07.959650    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:12:07.959930 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:07.959837    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:12:07.959930 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:07.959682    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:12:07.959930 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:07.959899    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:12:07.960141 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:07.959932    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:12:08.144650 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:08.144605    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kdt9" event={"ID":"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1","Type":"ContainerStarted","Data":"55e85cf5277611aa8a05c4826f1245c75d437f1bc824261d7c6bf1ff70b0342e"}
Sep 07 10:12:09.149303 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:09.149264    2562 generic.go:356] "Generic (PLEG): container finished" podID="757b7332-c3c9-4849-a0b7-cf8a4d46b3c1" containerID="55e85cf5277611aa8a05c4826f1245c75d437f1bc824261d7c6bf1ff70b0342e" exitCode=0
Sep 07 10:12:09.149303 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:09.149306    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kdt9" event={"ID":"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1","Type":"ContainerDied","Data":"55e85cf5277611aa8a05c4826f1245c75d437f1bc824261d7c6bf1ff70b0342e"}
Sep 07 10:12:09.958701 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:09.958664    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:12:09.958900 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:09.958664    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:12:09.958900 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:09.958822    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7n5z" podUID="7254bd39-6628-42fd-96e9-6a01b723c275"
Sep 07 10:12:09.958900 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:09.958664    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:12:09.958900 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:09.958883    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2724" podUID="948c00ef-80eb-4ff1-95c8-81d24b2a4d57"
Sep 07 10:12:09.959101 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:09.958937    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk7lm" podUID="e0bf9977-9b65-4b22-958e-46b52f65079e"
Sep 07 10:12:11.312403 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.312375    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-62.ec2.internal" event="NodeReady"
Sep 07 10:12:11.312848 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.312533    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 07 10:12:11.354941 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.354871    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6j256"]
Sep 07 10:12:11.356838 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.356816    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:11.359152 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.359132    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 07 10:12:11.359279 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.359203    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6d8d2\""
Sep 07 10:12:11.359279 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.359234    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 07 10:12:11.362312 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.362292    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d2r97"]
Sep 07 10:12:11.364131 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.364111    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d2r97"
Sep 07 10:12:11.366568 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.366404    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gq9jw\""
Sep 07 10:12:11.366568 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.366446    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 07 10:12:11.367038 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.366917    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 07 10:12:11.367038 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.366940    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 07 10:12:11.368282 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.368227    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6j256"]
Sep 07 10:12:11.373663 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.373642    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d2r97"]
Sep 07 10:12:11.498649 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.498617    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da5527f2-1ae4-4b33-a1f9-dff0689388e8-cert\") pod \"ingress-canary-d2r97\" (UID: \"da5527f2-1ae4-4b33-a1f9-dff0689388e8\") " pod="openshift-ingress-canary/ingress-canary-d2r97"
Sep 07 10:12:11.498649 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.498659    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-tmp-dir\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:11.498923 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.498745    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sbfp\" (UniqueName: \"kubernetes.io/projected/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-kube-api-access-8sbfp\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:11.498923 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.498791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdhr7\" (UniqueName: \"kubernetes.io/projected/da5527f2-1ae4-4b33-a1f9-dff0689388e8-kube-api-access-jdhr7\") pod \"ingress-canary-d2r97\" (UID: \"da5527f2-1ae4-4b33-a1f9-dff0689388e8\") " pod="openshift-ingress-canary/ingress-canary-d2r97"
Sep 07 10:12:11.498923 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.498819    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-config-volume\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:11.498923 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.498847    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:11.599341 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.599289    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs\") pod \"network-metrics-daemon-l7n5z\" (UID: \"7254bd39-6628-42fd-96e9-6a01b723c275\") " pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:12:11.599341 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.599334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da5527f2-1ae4-4b33-a1f9-dff0689388e8-cert\") pod \"ingress-canary-d2r97\" (UID: \"da5527f2-1ae4-4b33-a1f9-dff0689388e8\") " pod="openshift-ingress-canary/ingress-canary-d2r97"
Sep 07 10:12:11.599341 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.599361    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-tmp-dir\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:11.599812 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:11.599450    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:12:11.599812 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:11.599519    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs podName:7254bd39-6628-42fd-96e9-6a01b723c275 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:43.599499616 +0000 UTC m=+66.097189514 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs") pod "network-metrics-daemon-l7n5z" (UID: "7254bd39-6628-42fd-96e9-6a01b723c275") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:12:11.599812 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.599446    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sbfp\" (UniqueName: \"kubernetes.io/projected/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-kube-api-access-8sbfp\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:11.599812 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.599583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdhr7\" (UniqueName: \"kubernetes.io/projected/da5527f2-1ae4-4b33-a1f9-dff0689388e8-kube-api-access-jdhr7\") pod \"ingress-canary-d2r97\" (UID: \"da5527f2-1ae4-4b33-a1f9-dff0689388e8\") " pod="openshift-ingress-canary/ingress-canary-d2r97"
Sep 07 10:12:11.599812 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.599610    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-config-volume\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:11.599812 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.599644    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:11.599812 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.599737    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-tmp-dir\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:11.599812 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:11.599794    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 10:12:11.600499 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:11.599839    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls podName:f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:12.099823387 +0000 UTC m=+34.597513274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls") pod "dns-default-6j256" (UID: "f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20") : secret "dns-default-metrics-tls" not found
Sep 07 10:12:11.600499 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.600246    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-config-volume\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:11.604147 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.604124    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da5527f2-1ae4-4b33-a1f9-dff0689388e8-cert\") pod \"ingress-canary-d2r97\" (UID: \"da5527f2-1ae4-4b33-a1f9-dff0689388e8\") " pod="openshift-ingress-canary/ingress-canary-d2r97"
Sep 07 10:12:11.609157 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.609070    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sbfp\" (UniqueName: \"kubernetes.io/projected/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-kube-api-access-8sbfp\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:11.609314 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.609297    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdhr7\" (UniqueName: \"kubernetes.io/projected/da5527f2-1ae4-4b33-a1f9-dff0689388e8-kube-api-access-jdhr7\") pod \"ingress-canary-d2r97\" (UID: \"da5527f2-1ae4-4b33-a1f9-dff0689388e8\") " pod="openshift-ingress-canary/ingress-canary-d2r97"
Sep 07 10:12:11.675886 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.675853    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d2r97"
Sep 07 10:12:11.700738 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.700534    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6xd6\" (UniqueName: \"kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6\") pod \"network-check-target-q2724\" (UID: \"948c00ef-80eb-4ff1-95c8-81d24b2a4d57\") " pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:12:11.700861 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:11.700743    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:12:11.700861 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:11.700783    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:12:11.700861 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:11.700799    2562 projected.go:196] Error preparing data for projected volume kube-api-access-q6xd6 for pod openshift-network-diagnostics/network-check-target-q2724: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:12:11.700861 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.700809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret\") pod \"global-pull-secret-syncer-tk7lm\" (UID: \"e0bf9977-9b65-4b22-958e-46b52f65079e\") " pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:12:11.700861 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:11.700860    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6 podName:948c00ef-80eb-4ff1-95c8-81d24b2a4d57 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:43.700841955 +0000 UTC m=+66.198531856 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6xd6" (UniqueName: "kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6") pod "network-check-target-q2724" (UID: "948c00ef-80eb-4ff1-95c8-81d24b2a4d57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:12:11.701070 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:11.700925    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:12:11.701070 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:11.700975    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret podName:e0bf9977-9b65-4b22-958e-46b52f65079e nodeName:}" failed. No retries permitted until 2026-09-07 10:12:43.700959982 +0000 UTC m=+66.198649877 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret") pod "global-pull-secret-syncer-tk7lm" (UID: "e0bf9977-9b65-4b22-958e-46b52f65079e") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:12:11.959317 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.959285    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:12:11.959588 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.959419    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:12:11.959666 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.959419    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:12:11.963248 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.963213    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 10:12:11.964582 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.964491    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 10:12:11.964582 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.964528    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 10:12:11.964701 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.964667    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v9xn8\""
Sep 07 10:12:11.964701 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.964694    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cp5xp\""
Sep 07 10:12:11.964832 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.964799    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 10:12:11.967825 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:12:11.967804    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5527f2_1ae4_4b33_a1f9_dff0689388e8.slice/crio-d18ab4a4d6fdc211d700ead004102c024a8b640a37d68963a5309a90190d87ee WatchSource:0}: Error finding container d18ab4a4d6fdc211d700ead004102c024a8b640a37d68963a5309a90190d87ee: Status 404 returned error can't find the container with id d18ab4a4d6fdc211d700ead004102c024a8b640a37d68963a5309a90190d87ee
Sep 07 10:12:11.968424 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:11.967908    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d2r97"]
Sep 07 10:12:12.103387 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:12.103348    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:12.103551 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:12.103476    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 10:12:12.103551 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:12.103540    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls podName:f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:13.103516884 +0000 UTC m=+35.601206774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls") pod "dns-default-6j256" (UID: "f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20") : secret "dns-default-metrics-tls" not found
Sep 07 10:12:12.158497 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:12.158457    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d2r97" event={"ID":"da5527f2-1ae4-4b33-a1f9-dff0689388e8","Type":"ContainerStarted","Data":"d18ab4a4d6fdc211d700ead004102c024a8b640a37d68963a5309a90190d87ee"}
Sep 07 10:12:12.599888 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:12.599852    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzc8f"]
Sep 07 10:12:12.602326 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:12.602307    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzc8f"
Sep 07 10:12:12.604507 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:12.604491    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9zff6\""
Sep 07 10:12:12.605484 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:12.605469    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 07 10:12:12.605577 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:12.605470    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 07 10:12:12.610484 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:12.610461    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzc8f"]
Sep 07 10:12:12.708158 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:12.708122    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9drs\" (UniqueName: \"kubernetes.io/projected/39ac5c95-1a0d-479e-9427-33688fdf5751-kube-api-access-l9drs\") pod \"migrator-fdbb8fdff-kzc8f\" (UID: \"39ac5c95-1a0d-479e-9427-33688fdf5751\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzc8f"
Sep 07 10:12:12.808535 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:12.808494    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9drs\" (UniqueName: \"kubernetes.io/projected/39ac5c95-1a0d-479e-9427-33688fdf5751-kube-api-access-l9drs\") pod \"migrator-fdbb8fdff-kzc8f\" (UID: \"39ac5c95-1a0d-479e-9427-33688fdf5751\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzc8f"
Sep 07 10:12:12.817823 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:12.817787    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9drs\" (UniqueName: \"kubernetes.io/projected/39ac5c95-1a0d-479e-9427-33688fdf5751-kube-api-access-l9drs\") pod \"migrator-fdbb8fdff-kzc8f\" (UID: \"39ac5c95-1a0d-479e-9427-33688fdf5751\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzc8f"
Sep 07 10:12:12.912224 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:12.912145    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzc8f"
Sep 07 10:12:13.110996 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.110954    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:13.111174 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:13.111102    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 10:12:13.111174 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:13.111172    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls podName:f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:15.111154041 +0000 UTC m=+37.608843942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls") pod "dns-default-6j256" (UID: "f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20") : secret "dns-default-metrics-tls" not found
Sep 07 10:12:13.460917 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.460884    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4f5vn"]
Sep 07 10:12:13.463256 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.463237    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:13.465799 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.465779    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 07 10:12:13.465920 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.465807    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 07 10:12:13.465920 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.465886    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 07 10:12:13.466704 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.466684    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gp5c5\""
Sep 07 10:12:13.466811 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.466691    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 07 10:12:13.476438 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.476418    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4f5vn"]
Sep 07 10:12:13.614731 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.614707    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v8ss\" (UniqueName: \"kubernetes.io/projected/295fae86-047c-4eb7-a003-622609960fe9-kube-api-access-7v8ss\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:13.615173 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.614740    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:13.615173 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.614869    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/295fae86-047c-4eb7-a003-622609960fe9-data-volume\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:13.615173 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.614911    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/295fae86-047c-4eb7-a003-622609960fe9-crio-socket\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:13.615173 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.614938    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:13.716355 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.716285    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/295fae86-047c-4eb7-a003-622609960fe9-data-volume\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:13.716355 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.716346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/295fae86-047c-4eb7-a003-622609960fe9-crio-socket\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:13.716537 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.716385    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:13.716537 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.716437    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v8ss\" (UniqueName: \"kubernetes.io/projected/295fae86-047c-4eb7-a003-622609960fe9-kube-api-access-7v8ss\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:13.716537 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.716463    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:13.716698 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:13.716554    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:13.716698 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:13.716573    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 10:12:13.716698 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:13.716622    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls podName:295fae86-047c-4eb7-a003-622609960fe9 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:14.216602808 +0000 UTC m=+36.714292697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4f5vn" (UID: "295fae86-047c-4eb7-a003-622609960fe9") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:13.716698 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.716623    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/295fae86-047c-4eb7-a003-622609960fe9-crio-socket\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:13.716698 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:13.716641    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm podName:295fae86-047c-4eb7-a003-622609960fe9 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:14.21663221 +0000 UTC m=+36.714322097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4f5vn" (UID: "295fae86-047c-4eb7-a003-622609960fe9") : configmap "kube-rbac-proxy" not found
Sep 07 10:12:13.716698 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.716646    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/295fae86-047c-4eb7-a003-622609960fe9-data-volume\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:13.730301 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.730236    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v8ss\" (UniqueName: \"kubernetes.io/projected/295fae86-047c-4eb7-a003-622609960fe9-kube-api-access-7v8ss\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:13.834240 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:13.834092    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzc8f"]
Sep 07 10:12:13.839730 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:12:13.839694    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ac5c95_1a0d_479e_9427_33688fdf5751.slice/crio-295b04c3eadfcad19f16091cb47d925ec18a9a018149b9d61e7f6c7fb779a49b WatchSource:0}: Error finding container 295b04c3eadfcad19f16091cb47d925ec18a9a018149b9d61e7f6c7fb779a49b: Status 404 returned error can't find the container with id 295b04c3eadfcad19f16091cb47d925ec18a9a018149b9d61e7f6c7fb779a49b
Sep 07 10:12:14.171290 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:14.171191    2562 generic.go:356] "Generic (PLEG): container finished" podID="757b7332-c3c9-4849-a0b7-cf8a4d46b3c1" containerID="657b967c93b5b44840226624828a97512389ec151f489548b16c937701311528" exitCode=0
Sep 07 10:12:14.171447 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:14.171292    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kdt9" event={"ID":"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1","Type":"ContainerDied","Data":"657b967c93b5b44840226624828a97512389ec151f489548b16c937701311528"}
Sep 07 10:12:14.172852 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:14.172824    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzc8f" event={"ID":"39ac5c95-1a0d-479e-9427-33688fdf5751","Type":"ContainerStarted","Data":"295b04c3eadfcad19f16091cb47d925ec18a9a018149b9d61e7f6c7fb779a49b"}
Sep 07 10:12:14.221631 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:14.221595    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:14.221800 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:14.221673    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:14.221800 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:14.221729    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:14.221897 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:14.221809    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 10:12:14.221897 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:14.221817    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls podName:295fae86-047c-4eb7-a003-622609960fe9 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:15.221793288 +0000 UTC m=+37.719483195 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4f5vn" (UID: "295fae86-047c-4eb7-a003-622609960fe9") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:14.221897 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:14.221872    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm podName:295fae86-047c-4eb7-a003-622609960fe9 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:15.221858876 +0000 UTC m=+37.719548765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4f5vn" (UID: "295fae86-047c-4eb7-a003-622609960fe9") : configmap "kube-rbac-proxy" not found
Sep 07 10:12:15.130265 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.129992    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:15.130265 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:15.130170    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 10:12:15.130891 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:15.130333    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls podName:f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:19.130312702 +0000 UTC m=+41.628002604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls") pod "dns-default-6j256" (UID: "f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20") : secret "dns-default-metrics-tls" not found
Sep 07 10:12:15.178806 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.178758    2562 generic.go:356] "Generic (PLEG): container finished" podID="757b7332-c3c9-4849-a0b7-cf8a4d46b3c1" containerID="9108c1cd1c18d16cbca868dc2ac2e46f5ac9cc0bf07dab5dc45615858b778cfa" exitCode=0
Sep 07 10:12:15.178970 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.178837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kdt9" event={"ID":"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1","Type":"ContainerDied","Data":"9108c1cd1c18d16cbca868dc2ac2e46f5ac9cc0bf07dab5dc45615858b778cfa"}
Sep 07 10:12:15.217004 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.216970    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-thh29"]
Sep 07 10:12:15.230726 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.230690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:15.230924 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.230755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:15.230924 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:15.230825    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:15.230924 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:15.230842    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 10:12:15.230924 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:15.230892    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls podName:295fae86-047c-4eb7-a003-622609960fe9 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:17.230876645 +0000 UTC m=+39.728566533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4f5vn" (UID: "295fae86-047c-4eb7-a003-622609960fe9") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:15.230924 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:15.230910    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm podName:295fae86-047c-4eb7-a003-622609960fe9 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:17.230901567 +0000 UTC m=+39.728591454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4f5vn" (UID: "295fae86-047c-4eb7-a003-622609960fe9") : configmap "kube-rbac-proxy" not found
Sep 07 10:12:15.231953 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.231926    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-thh29"]
Sep 07 10:12:15.232081 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.232037    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-thh29"
Sep 07 10:12:15.234887 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.234865    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 07 10:12:15.235036 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.235024    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 07 10:12:15.235138 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.235072    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 07 10:12:15.235138 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.235098    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hc9wp\""
Sep 07 10:12:15.235249 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.235139    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 07 10:12:15.331824 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.331791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e41375da-803f-4de3-9351-b89e0b93dc91-signing-cabundle\") pod \"service-ca-75d99c484-thh29\" (UID: \"e41375da-803f-4de3-9351-b89e0b93dc91\") " pod="openshift-service-ca/service-ca-75d99c484-thh29"
Sep 07 10:12:15.332028 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.331853    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e41375da-803f-4de3-9351-b89e0b93dc91-signing-key\") pod \"service-ca-75d99c484-thh29\" (UID: \"e41375da-803f-4de3-9351-b89e0b93dc91\") " pod="openshift-service-ca/service-ca-75d99c484-thh29"
Sep 07 10:12:15.332028 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.331889    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvsdk\" (UniqueName: \"kubernetes.io/projected/e41375da-803f-4de3-9351-b89e0b93dc91-kube-api-access-rvsdk\") pod \"service-ca-75d99c484-thh29\" (UID: \"e41375da-803f-4de3-9351-b89e0b93dc91\") " pod="openshift-service-ca/service-ca-75d99c484-thh29"
Sep 07 10:12:15.433221 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.433114    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e41375da-803f-4de3-9351-b89e0b93dc91-signing-cabundle\") pod \"service-ca-75d99c484-thh29\" (UID: \"e41375da-803f-4de3-9351-b89e0b93dc91\") " pod="openshift-service-ca/service-ca-75d99c484-thh29"
Sep 07 10:12:15.433221 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.433182    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e41375da-803f-4de3-9351-b89e0b93dc91-signing-key\") pod \"service-ca-75d99c484-thh29\" (UID: \"e41375da-803f-4de3-9351-b89e0b93dc91\") " pod="openshift-service-ca/service-ca-75d99c484-thh29"
Sep 07 10:12:15.433221 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.433210    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvsdk\" (UniqueName: \"kubernetes.io/projected/e41375da-803f-4de3-9351-b89e0b93dc91-kube-api-access-rvsdk\") pod \"service-ca-75d99c484-thh29\" (UID: \"e41375da-803f-4de3-9351-b89e0b93dc91\") " pod="openshift-service-ca/service-ca-75d99c484-thh29"
Sep 07 10:12:15.433968 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.433945    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e41375da-803f-4de3-9351-b89e0b93dc91-signing-cabundle\") pod \"service-ca-75d99c484-thh29\" (UID: \"e41375da-803f-4de3-9351-b89e0b93dc91\") " pod="openshift-service-ca/service-ca-75d99c484-thh29"
Sep 07 10:12:15.436444 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.436422    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e41375da-803f-4de3-9351-b89e0b93dc91-signing-key\") pod \"service-ca-75d99c484-thh29\" (UID: \"e41375da-803f-4de3-9351-b89e0b93dc91\") " pod="openshift-service-ca/service-ca-75d99c484-thh29"
Sep 07 10:12:15.441527 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.441500    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvsdk\" (UniqueName: \"kubernetes.io/projected/e41375da-803f-4de3-9351-b89e0b93dc91-kube-api-access-rvsdk\") pod \"service-ca-75d99c484-thh29\" (UID: \"e41375da-803f-4de3-9351-b89e0b93dc91\") " pod="openshift-service-ca/service-ca-75d99c484-thh29"
Sep 07 10:12:15.545151 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:15.545121    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-thh29"
Sep 07 10:12:16.228386 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:16.228363    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qjnz6_ac0c45e5-27f8-4c26-929a-001e4b644d46/dns-node-resolver/0.log"
Sep 07 10:12:16.517206 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:16.517165    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-thh29"]
Sep 07 10:12:16.520524 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:12:16.520489    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41375da_803f_4de3_9351_b89e0b93dc91.slice/crio-23a2630b45ffc093a00341203cfd2bde026392686c2c27803d6fc4a4ee0cd4af WatchSource:0}: Error finding container 23a2630b45ffc093a00341203cfd2bde026392686c2c27803d6fc4a4ee0cd4af: Status 404 returned error can't find the container with id 23a2630b45ffc093a00341203cfd2bde026392686c2c27803d6fc4a4ee0cd4af
Sep 07 10:12:17.186429 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:17.186390    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-thh29" event={"ID":"e41375da-803f-4de3-9351-b89e0b93dc91","Type":"ContainerStarted","Data":"23a2630b45ffc093a00341203cfd2bde026392686c2c27803d6fc4a4ee0cd4af"}
Sep 07 10:12:17.190337 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:17.190309    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kdt9" event={"ID":"757b7332-c3c9-4849-a0b7-cf8a4d46b3c1","Type":"ContainerStarted","Data":"dbe2e274dabda2f0f7f95bc94087f9be16c05f05ccad2bcb669f45c5365e01b7"}
Sep 07 10:12:17.192180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:17.192151    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d2r97" event={"ID":"da5527f2-1ae4-4b33-a1f9-dff0689388e8","Type":"ContainerStarted","Data":"99f4671adb41cf2bac74b1aece0b0833dec0910e546dea8e5d8962005b37a011"}
Sep 07 10:12:17.194084 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:17.194060    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzc8f" event={"ID":"39ac5c95-1a0d-479e-9427-33688fdf5751","Type":"ContainerStarted","Data":"c2231d6b38a5322bbf189ddcfac246b28870f807592b2627756905a4d00b9a35"}
Sep 07 10:12:17.194208 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:17.194101    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzc8f" event={"ID":"39ac5c95-1a0d-479e-9427-33688fdf5751","Type":"ContainerStarted","Data":"6131fae691bffd13f29698d87e5f438cbc6f3b6e5d52977cce9e815fb4026355"}
Sep 07 10:12:17.213083 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:17.213020    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6kdt9" podStartSLOduration=9.433377372 podStartE2EDuration="39.212999619s" totalImagesPullingTime="29.779622247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:17.211450221 +0000 UTC m=+39.709140131" watchObservedRunningTime="2026-09-07 10:12:17.212999619 +0000 UTC m=+39.710689532"
Sep 07 10:12:17.232266 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:17.232208    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d2r97" podStartSLOduration=1.960354459 podStartE2EDuration="6.232184271s" totalImagesPullingTime="4.271829812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:17.230835247 +0000 UTC m=+39.728525175" watchObservedRunningTime="2026-09-07 10:12:17.232184271 +0000 UTC m=+39.729874182"
Sep 07 10:12:17.246722 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:17.246683    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:17.246905 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:17.246755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:17.246905 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:17.246849    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 10:12:17.247025 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:17.246919    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm podName:295fae86-047c-4eb7-a003-622609960fe9 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:21.246902032 +0000 UTC m=+43.744594279 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4f5vn" (UID: "295fae86-047c-4eb7-a003-622609960fe9") : configmap "kube-rbac-proxy" not found
Sep 07 10:12:17.247025 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:17.246859    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:17.247025 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:17.246996    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls podName:295fae86-047c-4eb7-a003-622609960fe9 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:21.246979813 +0000 UTC m=+43.744669701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4f5vn" (UID: "295fae86-047c-4eb7-a003-622609960fe9") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:17.251382 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:17.251340    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kzc8f" podStartSLOduration=2.849787889 podStartE2EDuration="5.251325335s" totalImagesPullingTime="2.401537446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:17.251155746 +0000 UTC m=+39.748845679" watchObservedRunningTime="2026-09-07 10:12:17.251325335 +0000 UTC m=+39.749015243"
Sep 07 10:12:17.428230 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:17.428199    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v86rj_f37b4a59-9c92-41fd-9561-ed2d017c6751/node-ca/0.log"
Sep 07 10:12:17.828619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:17.828590    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d2r97_da5527f2-1ae4-4b33-a1f9-dff0689388e8/serve-healthcheck-canary/0.log"
Sep 07 10:12:18.428147 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:18.428115    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kzc8f_39ac5c95-1a0d-479e-9427-33688fdf5751/migrator/0.log"
Sep 07 10:12:18.628598 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:18.628565    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kzc8f_39ac5c95-1a0d-479e-9427-33688fdf5751/graceful-termination/0.log"
Sep 07 10:12:19.163023 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:19.162979    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:19.163193 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:19.163131    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 10:12:19.163270 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:19.163193    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls podName:f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:27.163179272 +0000 UTC m=+49.660869159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls") pod "dns-default-6j256" (UID: "f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20") : secret "dns-default-metrics-tls" not found
Sep 07 10:12:19.201512 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:19.201476    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-thh29" event={"ID":"e41375da-803f-4de3-9351-b89e0b93dc91","Type":"ContainerStarted","Data":"fd6d65eb3d0986e4e1c93b571130175dd7c67e180c090eec886def0d42026644"}
Sep 07 10:12:19.217417 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:19.217292    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-thh29" podStartSLOduration=1.774782433 podStartE2EDuration="4.217275645s" totalImagesPullingTime="2.442493212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:19.216229777 +0000 UTC m=+41.713919706" watchObservedRunningTime="2026-09-07 10:12:19.217275645 +0000 UTC m=+41.714965554"
Sep 07 10:12:21.282075 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:21.282040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:21.282526 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:21.282151    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:21.282526 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:21.282178    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 10:12:21.282526 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:21.282246    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:21.282526 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:21.282264    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm podName:295fae86-047c-4eb7-a003-622609960fe9 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:29.282244661 +0000 UTC m=+51.779934568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4f5vn" (UID: "295fae86-047c-4eb7-a003-622609960fe9") : configmap "kube-rbac-proxy" not found
Sep 07 10:12:21.282526 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:21.282283    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls podName:295fae86-047c-4eb7-a003-622609960fe9 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:29.282272577 +0000 UTC m=+51.779962464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4f5vn" (UID: "295fae86-047c-4eb7-a003-622609960fe9") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:27.227415 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:27.227374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:27.229813 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:27.229785    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20-metrics-tls\") pod \"dns-default-6j256\" (UID: \"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20\") " pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:27.269482 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:27.269443    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:27.565882 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:27.565812    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6j256"]
Sep 07 10:12:27.568888 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:12:27.568861    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f159ed_ccc9_40b9_8fc4_f1feb8fe7f20.slice/crio-1157b247a25caaafc41e22d626628e53ffacebe12a1ab3e688fdb6041d40b08f WatchSource:0}: Error finding container 1157b247a25caaafc41e22d626628e53ffacebe12a1ab3e688fdb6041d40b08f: Status 404 returned error can't find the container with id 1157b247a25caaafc41e22d626628e53ffacebe12a1ab3e688fdb6041d40b08f
Sep 07 10:12:28.226038 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:28.225992    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6j256" event={"ID":"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20","Type":"ContainerStarted","Data":"1157b247a25caaafc41e22d626628e53ffacebe12a1ab3e688fdb6041d40b08f"}
Sep 07 10:12:29.344754 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:29.344713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:29.345155 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:29.344819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:29.345155 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:29.344914    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:29.345155 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:29.344977    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls podName:295fae86-047c-4eb7-a003-622609960fe9 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:45.344963486 +0000 UTC m=+67.842653373 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4f5vn" (UID: "295fae86-047c-4eb7-a003-622609960fe9") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:29.345271 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:29.345165    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/295fae86-047c-4eb7-a003-622609960fe9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:30.233679 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:30.233645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6j256" event={"ID":"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20","Type":"ContainerStarted","Data":"ab450210f8132fb394301d1638c63badb1acc4a91cfcabebd87df2a03ffe7f12"}
Sep 07 10:12:30.233874 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:30.233696    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6j256" event={"ID":"f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20","Type":"ContainerStarted","Data":"30caf710fbabad8f47a7a0f5caa85c7f8c967f97e777e6b4e7a6b4b9c4be859e"}
Sep 07 10:12:30.233874 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:30.233788    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:30.250118 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:30.250015    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6j256" podStartSLOduration=17.096529541 podStartE2EDuration="19.249999452s" totalImagesPullingTime="2.153469911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:30.249803978 +0000 UTC m=+52.747493899" watchObservedRunningTime="2026-09-07 10:12:30.249999452 +0000 UTC m=+52.747689361"
Sep 07 10:12:38.089113 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.089082    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"]
Sep 07 10:12:38.092005 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.091981    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.094198 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.094169    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 07 10:12:38.094698 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.094675    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 07 10:12:38.094826 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.094753    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 07 10:12:38.095215 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.095195    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 07 10:12:38.095215 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.095212    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 07 10:12:38.095364 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.095219    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 07 10:12:38.096134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.096115    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 07 10:12:38.106301 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.106271    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"]
Sep 07 10:12:38.154982 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.154954    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-959sz"
Sep 07 10:12:38.210961 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.210930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-hub\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.211124 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.211005    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.211171 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.211135    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjxw\" (UniqueName: \"kubernetes.io/projected/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-kube-api-access-sfjxw\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.211260 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.211241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-ca\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.211327 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.211279    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.211327 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.211318    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.312093 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.312058    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.312276 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.312112    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjxw\" (UniqueName: \"kubernetes.io/projected/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-kube-api-access-sfjxw\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.312276 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.312170    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-ca\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.312276 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.312197    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.312276 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.312225    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.312481 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.312372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-hub\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.313133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.313098    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.314682 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.314657    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.314890 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.314868    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-ca\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.314973 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.314891    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-hub\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.315052 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.315034    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.327977 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.327952    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjxw\" (UniqueName: \"kubernetes.io/projected/0ef034e2-697f-4e4c-85d7-bcdddf6dfff2-kube-api-access-sfjxw\") pod \"cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2\" (UID: \"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.412612 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.412511    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"
Sep 07 10:12:38.679143 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:38.679064    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2"]
Sep 07 10:12:38.681817 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:12:38.681748    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ef034e2_697f_4e4c_85d7_bcdddf6dfff2.slice/crio-9093210edd28b9677278fb17a868c6fe509c64212360693a2c2cf8ceb2ce5814 WatchSource:0}: Error finding container 9093210edd28b9677278fb17a868c6fe509c64212360693a2c2cf8ceb2ce5814: Status 404 returned error can't find the container with id 9093210edd28b9677278fb17a868c6fe509c64212360693a2c2cf8ceb2ce5814
Sep 07 10:12:39.263797 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:39.263741    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2" event={"ID":"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2","Type":"ContainerStarted","Data":"9093210edd28b9677278fb17a868c6fe509c64212360693a2c2cf8ceb2ce5814"}
Sep 07 10:12:40.240222 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:40.239945    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6j256"
Sep 07 10:12:43.282986 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.282947    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2" event={"ID":"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2","Type":"ContainerStarted","Data":"d284b6f6766c4614885f927875c496d48cc7e70f8614a6bb1a656bca047d8fe0"}
Sep 07 10:12:43.283364 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.283000    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2" event={"ID":"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2","Type":"ContainerStarted","Data":"bc9f9319e9fb426c7fa441239f1494adbdc0fb02baecfbaab04f46239af494ff"}
Sep 07 10:12:43.283364 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.283018    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2" event={"ID":"0ef034e2-697f-4e4c-85d7-bcdddf6dfff2","Type":"ContainerStarted","Data":"a01f0970ccf0ba822b42d75855ae79a71a591f36c2c2d444575cc7bd24e3641d"}
Sep 07 10:12:43.300449 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.300395    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d4dd4b7c-mnfd2" podStartSLOduration=1.697301107 podStartE2EDuration="5.300378338s" totalImagesPullingTime="3.603077231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:43.299131125 +0000 UTC m=+65.796821046" watchObservedRunningTime="2026-09-07 10:12:43.300378338 +0000 UTC m=+65.798068247"
Sep 07 10:12:43.655813 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.655695    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs\") pod \"network-metrics-daemon-l7n5z\" (UID: \"7254bd39-6628-42fd-96e9-6a01b723c275\") " pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:12:43.657939 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.657918    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 10:12:43.668765 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.668746    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7254bd39-6628-42fd-96e9-6a01b723c275-metrics-certs\") pod \"network-metrics-daemon-l7n5z\" (UID: \"7254bd39-6628-42fd-96e9-6a01b723c275\") " pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:12:43.756235 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.756196    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6xd6\" (UniqueName: \"kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6\") pod \"network-check-target-q2724\" (UID: \"948c00ef-80eb-4ff1-95c8-81d24b2a4d57\") " pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:12:43.756411 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.756262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret\") pod \"global-pull-secret-syncer-tk7lm\" (UID: \"e0bf9977-9b65-4b22-958e-46b52f65079e\") " pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:12:43.758716 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.758690    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 10:12:43.758716 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.758710    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 10:12:43.768522 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.768503    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 10:12:43.768713 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.768696    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bf9977-9b65-4b22-958e-46b52f65079e-original-pull-secret\") pod \"global-pull-secret-syncer-tk7lm\" (UID: \"e0bf9977-9b65-4b22-958e-46b52f65079e\") " pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:12:43.779576 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.779545    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6xd6\" (UniqueName: \"kubernetes.io/projected/948c00ef-80eb-4ff1-95c8-81d24b2a4d57-kube-api-access-q6xd6\") pod \"network-check-target-q2724\" (UID: \"948c00ef-80eb-4ff1-95c8-81d24b2a4d57\") " pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:12:43.782356 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.782338    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk7lm"
Sep 07 10:12:43.791707 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.791685    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cp5xp\""
Sep 07 10:12:43.797597 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.797579    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v9xn8\""
Sep 07 10:12:43.800437 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.800424    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7n5z"
Sep 07 10:12:43.806208 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:43.806192    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:12:44.070481 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:44.070333    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tk7lm"]
Sep 07 10:12:44.072838 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:12:44.072812    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bf9977_9b65_4b22_958e_46b52f65079e.slice/crio-ce27ffce98e5942b6b5d2be742f3b5fcb8cbae28270e9022197530d117125c0b WatchSource:0}: Error finding container ce27ffce98e5942b6b5d2be742f3b5fcb8cbae28270e9022197530d117125c0b: Status 404 returned error can't find the container with id ce27ffce98e5942b6b5d2be742f3b5fcb8cbae28270e9022197530d117125c0b
Sep 07 10:12:44.098985 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:44.098962    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l7n5z"]
Sep 07 10:12:44.101609 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:12:44.101585    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7254bd39_6628_42fd_96e9_6a01b723c275.slice/crio-722a2f4b8aea33f39e5d03b0a059f8ec345edac85fef4717bd6e0a5bece89f7a WatchSource:0}: Error finding container 722a2f4b8aea33f39e5d03b0a059f8ec345edac85fef4717bd6e0a5bece89f7a: Status 404 returned error can't find the container with id 722a2f4b8aea33f39e5d03b0a059f8ec345edac85fef4717bd6e0a5bece89f7a
Sep 07 10:12:44.129023 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:44.128998    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q2724"]
Sep 07 10:12:44.131232 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:12:44.131212    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948c00ef_80eb_4ff1_95c8_81d24b2a4d57.slice/crio-794af610ec628689ad5a9357dd048d007e83b1e2fa87580d6f097d0105b6fc18 WatchSource:0}: Error finding container 794af610ec628689ad5a9357dd048d007e83b1e2fa87580d6f097d0105b6fc18: Status 404 returned error can't find the container with id 794af610ec628689ad5a9357dd048d007e83b1e2fa87580d6f097d0105b6fc18
Sep 07 10:12:44.291001 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:44.290958    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l7n5z" event={"ID":"7254bd39-6628-42fd-96e9-6a01b723c275","Type":"ContainerStarted","Data":"722a2f4b8aea33f39e5d03b0a059f8ec345edac85fef4717bd6e0a5bece89f7a"}
Sep 07 10:12:44.292264 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:44.292238    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tk7lm" event={"ID":"e0bf9977-9b65-4b22-958e-46b52f65079e","Type":"ContainerStarted","Data":"ce27ffce98e5942b6b5d2be742f3b5fcb8cbae28270e9022197530d117125c0b"}
Sep 07 10:12:44.293272 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:44.293252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q2724" event={"ID":"948c00ef-80eb-4ff1-95c8-81d24b2a4d57","Type":"ContainerStarted","Data":"794af610ec628689ad5a9357dd048d007e83b1e2fa87580d6f097d0105b6fc18"}
Sep 07 10:12:45.369745 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:45.369711    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:45.372278 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:45.372250    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/295fae86-047c-4eb7-a003-622609960fe9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4f5vn\" (UID: \"295fae86-047c-4eb7-a003-622609960fe9\") " pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:45.576677 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:45.576586    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gp5c5\""
Sep 07 10:12:45.584691 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:45.584658    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4f5vn"
Sep 07 10:12:45.875112 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:45.875015    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4f5vn"]
Sep 07 10:12:46.302634 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:46.302597    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4f5vn" event={"ID":"295fae86-047c-4eb7-a003-622609960fe9","Type":"ContainerStarted","Data":"d6195a823310daaf38bbe9c57a908d523694559b92b2ed884a69818ca6e702fd"}
Sep 07 10:12:46.302817 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:46.302660    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4f5vn" event={"ID":"295fae86-047c-4eb7-a003-622609960fe9","Type":"ContainerStarted","Data":"8d30574139385ede4910a694b1bdbf6d2d8fc7f9d7429168a8165a15da172298"}
Sep 07 10:12:46.304754 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:46.304728    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l7n5z" event={"ID":"7254bd39-6628-42fd-96e9-6a01b723c275","Type":"ContainerStarted","Data":"dbe94c2c66bcdd454343f14a4c7558335b0d47c1f6cbfd89f3b079c7cbc5d2eb"}
Sep 07 10:12:46.304906 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:46.304795    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l7n5z" event={"ID":"7254bd39-6628-42fd-96e9-6a01b723c275","Type":"ContainerStarted","Data":"ecc3d08f4411df43c74887ebf1f024d3431e7f128afd808ce1ebeeef1db60e76"}
Sep 07 10:12:46.320690 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:46.320627    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l7n5z" podStartSLOduration=67.136468208 podStartE2EDuration="1m8.320603545s" totalImagesPullingTime="1.184135337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:46.319266471 +0000 UTC m=+68.816956409" watchObservedRunningTime="2026-09-07 10:12:46.320603545 +0000 UTC m=+68.818293454"
Sep 07 10:12:49.319975 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:49.319933    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q2724" event={"ID":"948c00ef-80eb-4ff1-95c8-81d24b2a4d57","Type":"ContainerStarted","Data":"f5640827115f31de6c3a9132d1be7dcc37906d8b83aeae5728b240268d6ffcc8"}
Sep 07 10:12:49.320449 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:49.320051    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:12:49.322042 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:49.322009    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4f5vn" event={"ID":"295fae86-047c-4eb7-a003-622609960fe9","Type":"ContainerStarted","Data":"e03b65e1db73dd75a144c02305b3048ae0bc727094317ee3dccd11f9374911da"}
Sep 07 10:12:49.323609 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:49.323580    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tk7lm" event={"ID":"e0bf9977-9b65-4b22-958e-46b52f65079e","Type":"ContainerStarted","Data":"ce5bfc86d7431dce0e1b1784df91f9d9f37ca9c355ecd7749444095d41486bce"}
Sep 07 10:12:49.335411 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:49.335336    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q2724" podStartSLOduration=66.633975616 podStartE2EDuration="1m11.335316483s" totalImagesPullingTime="4.701340867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:49.334442981 +0000 UTC m=+71.832132910" watchObservedRunningTime="2026-09-07 10:12:49.335316483 +0000 UTC m=+71.833006393"
Sep 07 10:12:49.349794 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:49.349244    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tk7lm" podStartSLOduration=66.5740584 podStartE2EDuration="1m11.349227507s" totalImagesPullingTime="4.775169107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:49.348990823 +0000 UTC m=+71.846680734" watchObservedRunningTime="2026-09-07 10:12:49.349227507 +0000 UTC m=+71.846917417"
Sep 07 10:12:50.331682 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:50.331645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4f5vn" event={"ID":"295fae86-047c-4eb7-a003-622609960fe9","Type":"ContainerStarted","Data":"2e939a636049b2fc866945a0f75b7e187d8a224b7c57d849e4db9c5a9e347fcf"}
Sep 07 10:12:50.359168 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:50.359112    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4f5vn" podStartSLOduration=33.213362337 podStartE2EDuration="37.359095468s" totalImagesPullingTime="4.145733131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:12:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:50.357584695 +0000 UTC m=+72.855274616" watchObservedRunningTime="2026-09-07 10:12:50.359095468 +0000 UTC m=+72.856785376"
Sep 07 10:12:51.069679 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.069642    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-flj7x"]
Sep 07 10:12:51.089789 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.089746    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-flj7x"]
Sep 07 10:12:51.089926 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.089881    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x"
Sep 07 10:12:51.093483 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.093455    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-sggjv\""
Sep 07 10:12:51.093626 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.093482    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 07 10:12:51.093626 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.093504    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 07 10:12:51.093626 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.093480    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 07 10:12:51.093626 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.093609    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 07 10:12:51.093936 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.093912    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 07 10:12:51.213753 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.213714    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/25edec58-0c81-443b-8fd7-1a40acd6d7ea-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-flj7x\" (UID: \"25edec58-0c81-443b-8fd7-1a40acd6d7ea\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x"
Sep 07 10:12:51.213753 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.213751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ltgd\" (UniqueName: \"kubernetes.io/projected/25edec58-0c81-443b-8fd7-1a40acd6d7ea-kube-api-access-5ltgd\") pod \"prometheus-operator-6f4946db56-flj7x\" (UID: \"25edec58-0c81-443b-8fd7-1a40acd6d7ea\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x"
Sep 07 10:12:51.213977 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.213804    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25edec58-0c81-443b-8fd7-1a40acd6d7ea-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-flj7x\" (UID: \"25edec58-0c81-443b-8fd7-1a40acd6d7ea\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x"
Sep 07 10:12:51.213977 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.213838    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25edec58-0c81-443b-8fd7-1a40acd6d7ea-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-flj7x\" (UID: \"25edec58-0c81-443b-8fd7-1a40acd6d7ea\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x"
Sep 07 10:12:51.314144 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.314104    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25edec58-0c81-443b-8fd7-1a40acd6d7ea-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-flj7x\" (UID: \"25edec58-0c81-443b-8fd7-1a40acd6d7ea\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x"
Sep 07 10:12:51.314144 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.314147    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25edec58-0c81-443b-8fd7-1a40acd6d7ea-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-flj7x\" (UID: \"25edec58-0c81-443b-8fd7-1a40acd6d7ea\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x"
Sep 07 10:12:51.314354 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.314187    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/25edec58-0c81-443b-8fd7-1a40acd6d7ea-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-flj7x\" (UID: \"25edec58-0c81-443b-8fd7-1a40acd6d7ea\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x"
Sep 07 10:12:51.314354 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.314232    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ltgd\" (UniqueName: \"kubernetes.io/projected/25edec58-0c81-443b-8fd7-1a40acd6d7ea-kube-api-access-5ltgd\") pod \"prometheus-operator-6f4946db56-flj7x\" (UID: \"25edec58-0c81-443b-8fd7-1a40acd6d7ea\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x"
Sep 07 10:12:51.314462 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:51.314354    2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 07 10:12:51.314462 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:12:51.314447    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25edec58-0c81-443b-8fd7-1a40acd6d7ea-prometheus-operator-tls podName:25edec58-0c81-443b-8fd7-1a40acd6d7ea nodeName:}" failed. No retries permitted until 2026-09-07 10:12:51.814423206 +0000 UTC m=+74.312113113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/25edec58-0c81-443b-8fd7-1a40acd6d7ea-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-flj7x" (UID: "25edec58-0c81-443b-8fd7-1a40acd6d7ea") : secret "prometheus-operator-tls" not found
Sep 07 10:12:51.314780 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.314740    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25edec58-0c81-443b-8fd7-1a40acd6d7ea-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-flj7x\" (UID: \"25edec58-0c81-443b-8fd7-1a40acd6d7ea\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x"
Sep 07 10:12:51.316715 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.316690    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25edec58-0c81-443b-8fd7-1a40acd6d7ea-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-flj7x\" (UID: \"25edec58-0c81-443b-8fd7-1a40acd6d7ea\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x"
Sep 07 10:12:51.325574 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.325516    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ltgd\" (UniqueName: \"kubernetes.io/projected/25edec58-0c81-443b-8fd7-1a40acd6d7ea-kube-api-access-5ltgd\") pod \"prometheus-operator-6f4946db56-flj7x\" (UID: \"25edec58-0c81-443b-8fd7-1a40acd6d7ea\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x"
Sep 07 10:12:51.834158 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.834122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/25edec58-0c81-443b-8fd7-1a40acd6d7ea-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-flj7x\" (UID: \"25edec58-0c81-443b-8fd7-1a40acd6d7ea\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x"
Sep 07 10:12:51.836469 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.836442    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/25edec58-0c81-443b-8fd7-1a40acd6d7ea-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-flj7x\" (UID: \"25edec58-0c81-443b-8fd7-1a40acd6d7ea\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x"
Sep 07 10:12:51.998814 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:51.998772    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x"
Sep 07 10:12:52.264893 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:52.264863    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-flj7x"]
Sep 07 10:12:52.267564 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:12:52.267538    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25edec58_0c81_443b_8fd7_1a40acd6d7ea.slice/crio-0ab44767041be12854d5dcecc75967a87caad3f7aff044851a2bea37c71c1bba WatchSource:0}: Error finding container 0ab44767041be12854d5dcecc75967a87caad3f7aff044851a2bea37c71c1bba: Status 404 returned error can't find the container with id 0ab44767041be12854d5dcecc75967a87caad3f7aff044851a2bea37c71c1bba
Sep 07 10:12:52.342619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:52.342587    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x" event={"ID":"25edec58-0c81-443b-8fd7-1a40acd6d7ea","Type":"ContainerStarted","Data":"0ab44767041be12854d5dcecc75967a87caad3f7aff044851a2bea37c71c1bba"}
Sep 07 10:12:54.350904 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:54.350865    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x" event={"ID":"25edec58-0c81-443b-8fd7-1a40acd6d7ea","Type":"ContainerStarted","Data":"ac9db0a0454defce28c34b1ceb5d80161210be2ea76fa122283efc0952985e03"}
Sep 07 10:12:54.351272 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:54.350918    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x" event={"ID":"25edec58-0c81-443b-8fd7-1a40acd6d7ea","Type":"ContainerStarted","Data":"d042647dd5217e9bef2d0acdfdf272f564a947ae80c0d5eeb3361d8466497504"}
Sep 07 10:12:54.367115 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:54.367069    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-flj7x" podStartSLOduration=2.007411967 podStartE2EDuration="3.367050831s" totalImagesPullingTime="1.359638864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:54.365910844 +0000 UTC m=+76.863600773" watchObservedRunningTime="2026-09-07 10:12:54.367050831 +0000 UTC m=+76.864740741"
Sep 07 10:12:56.518194 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.518159    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tk2c8"]
Sep 07 10:12:56.521093 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.521071    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.523401 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.523376    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 07 10:12:56.523401 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.523387    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s5st7\""
Sep 07 10:12:56.523584 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.523548    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 07 10:12:56.523638 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.523586    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 07 10:12:56.668080 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.668041    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-sys\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.668080 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.668084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-metrics-client-ca\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.668321 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.668114    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-node-exporter-tls\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.668321 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.668140    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-node-exporter-wtmp\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.668321 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.668163    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-node-exporter-textfile\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.668321 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.668187    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-root\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.668321 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.668211    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-node-exporter-accelerators-collector-config\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.668321 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.668267    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.668605 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.668330    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr6h6\" (UniqueName: \"kubernetes.io/projected/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-kube-api-access-wr6h6\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.769321 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.769243    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-node-exporter-tls\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.769535 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.769515    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-node-exporter-wtmp\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.769657 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.769644    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-node-exporter-textfile\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.769745 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.769723    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-node-exporter-wtmp\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.769856 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.769841    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-root\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.769967 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.769946    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-node-exporter-accelerators-collector-config\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.770102 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.769979    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.770102 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.770011    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr6h6\" (UniqueName: \"kubernetes.io/projected/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-kube-api-access-wr6h6\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.770102 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.770059    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-sys\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.770102 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.770074    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-node-exporter-textfile\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.770102 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.770100    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-metrics-client-ca\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.771062 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.769889    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-root\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.771062 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.770633    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-metrics-client-ca\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.771062 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.771016    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-node-exporter-accelerators-collector-config\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.771354 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.771319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-sys\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.772488 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.772462    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-node-exporter-tls\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.774836 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.774813    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.779009 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.778984    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr6h6\" (UniqueName: \"kubernetes.io/projected/e7ef93be-7d82-4003-ac92-7f2a9eb75eef-kube-api-access-wr6h6\") pod \"node-exporter-tk2c8\" (UID: \"e7ef93be-7d82-4003-ac92-7f2a9eb75eef\") " pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.831066 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:56.831039    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tk2c8"
Sep 07 10:12:56.841812 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:12:56.841786    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ef93be_7d82_4003_ac92_7f2a9eb75eef.slice/crio-8016cf7bb880a3cf183fbca983e411f65ed1c6da7acf32a93406f262130f1384 WatchSource:0}: Error finding container 8016cf7bb880a3cf183fbca983e411f65ed1c6da7acf32a93406f262130f1384: Status 404 returned error can't find the container with id 8016cf7bb880a3cf183fbca983e411f65ed1c6da7acf32a93406f262130f1384
Sep 07 10:12:57.363641 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:57.363604    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tk2c8" event={"ID":"e7ef93be-7d82-4003-ac92-7f2a9eb75eef","Type":"ContainerStarted","Data":"8016cf7bb880a3cf183fbca983e411f65ed1c6da7acf32a93406f262130f1384"}
Sep 07 10:12:58.369001 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:58.368965    2562 generic.go:356] "Generic (PLEG): container finished" podID="e7ef93be-7d82-4003-ac92-7f2a9eb75eef" containerID="1b1c9c932b5da6de5cf0c466f7a0aba8463cbf0955f5f082eb4ddd018dda503f" exitCode=0
Sep 07 10:12:58.369378 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:58.369053    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tk2c8" event={"ID":"e7ef93be-7d82-4003-ac92-7f2a9eb75eef","Type":"ContainerDied","Data":"1b1c9c932b5da6de5cf0c466f7a0aba8463cbf0955f5f082eb4ddd018dda503f"}
Sep 07 10:12:59.375584 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:59.375548    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tk2c8" event={"ID":"e7ef93be-7d82-4003-ac92-7f2a9eb75eef","Type":"ContainerStarted","Data":"c8090fcb305b78cd7b801c054654436414576e6f6b3d2305c9ff94225460c242"}
Sep 07 10:12:59.376081 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:59.375617    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tk2c8" event={"ID":"e7ef93be-7d82-4003-ac92-7f2a9eb75eef","Type":"ContainerStarted","Data":"69b7ad2e22ffefc2d5673d610192b0659743b34e0448c6e772c5a6d33022b091"}
Sep 07 10:12:59.395039 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:12:59.394990    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tk2c8" podStartSLOduration=2.394562369 podStartE2EDuration="3.39497599s" totalImagesPullingTime="938.899541ms" totalInitContainerRuntime="61.51408ms" isStatefulPod=true podCreationTimestamp="2026-09-07 10:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:59.393313011 +0000 UTC m=+81.891002942" watchObservedRunningTime="2026-09-07 10:12:59.39497599 +0000 UTC m=+81.892665899"
Sep 07 10:13:00.846633 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:00.846523    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7c58fd999c-nwwxd"]
Sep 07 10:13:00.849643 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:00.849626    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:00.852254 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:00.852228    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 07 10:13:00.853561 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:00.853539    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 07 10:13:00.853561 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:00.853552    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 07 10:13:00.853723 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:00.853539    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dbqlh\""
Sep 07 10:13:00.853723 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:00.853611    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 07 10:13:00.853909 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:00.853893    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1qubknj7n0mle\""
Sep 07 10:13:00.858657 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:00.858634    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c58fd999c-nwwxd"]
Sep 07 10:13:01.001961 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.001926    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/92c0c47f-d30b-48f8-8619-e558dd51c3e2-secret-metrics-server-tls\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.002140 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.001968    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92c0c47f-d30b-48f8-8619-e558dd51c3e2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.002140 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.001998    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/92c0c47f-d30b-48f8-8619-e558dd51c3e2-client-ca-bundle\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.002140 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.002019    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/92c0c47f-d30b-48f8-8619-e558dd51c3e2-secret-metrics-server-client-certs\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.002140 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.002038    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/92c0c47f-d30b-48f8-8619-e558dd51c3e2-audit-log\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.002140 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.002056    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/92c0c47f-d30b-48f8-8619-e558dd51c3e2-metrics-server-audit-profiles\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.002140 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.002075    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zlq5\" (UniqueName: \"kubernetes.io/projected/92c0c47f-d30b-48f8-8619-e558dd51c3e2-kube-api-access-6zlq5\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.103445 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.103360    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/92c0c47f-d30b-48f8-8619-e558dd51c3e2-audit-log\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.103445 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.103399    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/92c0c47f-d30b-48f8-8619-e558dd51c3e2-metrics-server-audit-profiles\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.103445 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.103423    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zlq5\" (UniqueName: \"kubernetes.io/projected/92c0c47f-d30b-48f8-8619-e558dd51c3e2-kube-api-access-6zlq5\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.103651 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.103465    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/92c0c47f-d30b-48f8-8619-e558dd51c3e2-secret-metrics-server-tls\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.103651 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.103483    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92c0c47f-d30b-48f8-8619-e558dd51c3e2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.103651 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.103501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/92c0c47f-d30b-48f8-8619-e558dd51c3e2-client-ca-bundle\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.103651 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.103523    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/92c0c47f-d30b-48f8-8619-e558dd51c3e2-secret-metrics-server-client-certs\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.103947 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.103922    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/92c0c47f-d30b-48f8-8619-e558dd51c3e2-audit-log\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.104371 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.104345    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92c0c47f-d30b-48f8-8619-e558dd51c3e2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.104507 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.104489    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/92c0c47f-d30b-48f8-8619-e558dd51c3e2-metrics-server-audit-profiles\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.106046 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.106017    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/92c0c47f-d30b-48f8-8619-e558dd51c3e2-secret-metrics-server-tls\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.106148 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.106101    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/92c0c47f-d30b-48f8-8619-e558dd51c3e2-client-ca-bundle\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.106197 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.106159    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/92c0c47f-d30b-48f8-8619-e558dd51c3e2-secret-metrics-server-client-certs\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.110953 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.110930    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zlq5\" (UniqueName: \"kubernetes.io/projected/92c0c47f-d30b-48f8-8619-e558dd51c3e2-kube-api-access-6zlq5\") pod \"metrics-server-7c58fd999c-nwwxd\" (UID: \"92c0c47f-d30b-48f8-8619-e558dd51c3e2\") " pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.159362 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.159329    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:01.442435 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:01.442299    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c58fd999c-nwwxd"]
Sep 07 10:13:01.445592 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:13:01.445567    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c0c47f_d30b_48f8_8619_e558dd51c3e2.slice/crio-ec93df25370d4ee81b624e00378cec215407ee9992f75a306a037f812c680b53 WatchSource:0}: Error finding container ec93df25370d4ee81b624e00378cec215407ee9992f75a306a037f812c680b53: Status 404 returned error can't find the container with id ec93df25370d4ee81b624e00378cec215407ee9992f75a306a037f812c680b53
Sep 07 10:13:02.388570 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:02.388526    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd" event={"ID":"92c0c47f-d30b-48f8-8619-e558dd51c3e2","Type":"ContainerStarted","Data":"ec93df25370d4ee81b624e00378cec215407ee9992f75a306a037f812c680b53"}
Sep 07 10:13:03.393985 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:03.393906    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd" event={"ID":"92c0c47f-d30b-48f8-8619-e558dd51c3e2","Type":"ContainerStarted","Data":"f7dfad8be1e700d4382d91aeaf0e0cbdb1fc06c0aa54acf17e9a91fa69eeac43"}
Sep 07 10:13:03.412482 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:03.412428    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd" podStartSLOduration=1.909949137 podStartE2EDuration="3.412411344s" totalImagesPullingTime="1.502462207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:13:03.411188499 +0000 UTC m=+85.908878430" watchObservedRunningTime="2026-09-07 10:13:03.412411344 +0000 UTC m=+85.910101254"
Sep 07 10:13:20.334865 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:20.334837    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q2724"
Sep 07 10:13:21.159720 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:21.159672    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:21.159946 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:21.159742    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:41.165541 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:41.165511    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:13:41.169418 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:13:41.169394    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7c58fd999c-nwwxd"
Sep 07 10:15:56.036877 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:15:56.036841    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t7pqh"]
Sep 07 10:15:56.039894 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:15:56.039877    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t7pqh"
Sep 07 10:15:56.042539 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:15:56.042514    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 07 10:15:56.042951 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:15:56.042931    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 07 10:15:56.043095 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:15:56.043078    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-d52t6\""
Sep 07 10:15:56.052811 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:15:56.052788    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t7pqh"]
Sep 07 10:15:56.120588 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:15:56.120550    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee17e350-63d5-44ff-bb85-484b10356af3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t7pqh\" (UID: \"ee17e350-63d5-44ff-bb85-484b10356af3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t7pqh"
Sep 07 10:15:56.120588 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:15:56.120594    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r6sv\" (UniqueName: \"kubernetes.io/projected/ee17e350-63d5-44ff-bb85-484b10356af3-kube-api-access-2r6sv\") pod \"cert-manager-operator-controller-manager-55c444bf86-t7pqh\" (UID: \"ee17e350-63d5-44ff-bb85-484b10356af3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t7pqh"
Sep 07 10:15:56.221390 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:15:56.221347    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee17e350-63d5-44ff-bb85-484b10356af3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t7pqh\" (UID: \"ee17e350-63d5-44ff-bb85-484b10356af3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t7pqh"
Sep 07 10:15:56.221589 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:15:56.221400    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r6sv\" (UniqueName: \"kubernetes.io/projected/ee17e350-63d5-44ff-bb85-484b10356af3-kube-api-access-2r6sv\") pod \"cert-manager-operator-controller-manager-55c444bf86-t7pqh\" (UID: \"ee17e350-63d5-44ff-bb85-484b10356af3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t7pqh"
Sep 07 10:15:56.221820 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:15:56.221798    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee17e350-63d5-44ff-bb85-484b10356af3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t7pqh\" (UID: \"ee17e350-63d5-44ff-bb85-484b10356af3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t7pqh"
Sep 07 10:15:56.231950 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:15:56.231916    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r6sv\" (UniqueName: \"kubernetes.io/projected/ee17e350-63d5-44ff-bb85-484b10356af3-kube-api-access-2r6sv\") pod \"cert-manager-operator-controller-manager-55c444bf86-t7pqh\" (UID: \"ee17e350-63d5-44ff-bb85-484b10356af3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t7pqh"
Sep 07 10:15:56.349131 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:15:56.349038    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t7pqh"
Sep 07 10:15:56.623565 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:15:56.623469    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t7pqh"]
Sep 07 10:15:56.627527 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:15:56.627496    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee17e350_63d5_44ff_bb85_484b10356af3.slice/crio-733a619f387fcc7d343bbda25f21621d8c5372a1ea490acf014e5993fc57622f WatchSource:0}: Error finding container 733a619f387fcc7d343bbda25f21621d8c5372a1ea490acf014e5993fc57622f: Status 404 returned error can't find the container with id 733a619f387fcc7d343bbda25f21621d8c5372a1ea490acf014e5993fc57622f
Sep 07 10:15:56.972087 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:15:56.972053    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t7pqh" event={"ID":"ee17e350-63d5-44ff-bb85-484b10356af3","Type":"ContainerStarted","Data":"733a619f387fcc7d343bbda25f21621d8c5372a1ea490acf014e5993fc57622f"}
Sep 07 10:15:59.985546 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:15:59.985508    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t7pqh" event={"ID":"ee17e350-63d5-44ff-bb85-484b10356af3","Type":"ContainerStarted","Data":"eda3f21e4865ee61c4baa1df3aa6841bc1e5bfaf4e6f6a0a3e00434ab482d32a"}
Sep 07 10:16:00.007493 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:00.007442    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t7pqh" podStartSLOduration=1.554123133 podStartE2EDuration="4.007426553s" totalImagesPullingTime="2.45330342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:16:00.0050784 +0000 UTC m=+262.502768329" watchObservedRunningTime="2026-09-07 10:16:00.007426553 +0000 UTC m=+262.505116461"
Sep 07 10:16:00.718405 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:00.718372    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7"]
Sep 07 10:16:00.720527 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:00.720499    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7"
Sep 07 10:16:00.722996 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:00.722973    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 10:16:00.723730 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:00.723716    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m28xb\""
Sep 07 10:16:00.723795 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:00.723745    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 10:16:00.736886 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:00.736866    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7"]
Sep 07 10:16:00.858024 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:00.857987    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3de1c6a3-bc7b-4593-b180-3aa55469b64b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7\" (UID: \"3de1c6a3-bc7b-4593-b180-3aa55469b64b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7"
Sep 07 10:16:00.858196 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:00.858053    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6m6l\" (UniqueName: \"kubernetes.io/projected/3de1c6a3-bc7b-4593-b180-3aa55469b64b-kube-api-access-w6m6l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7\" (UID: \"3de1c6a3-bc7b-4593-b180-3aa55469b64b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7"
Sep 07 10:16:00.858196 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:00.858075    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3de1c6a3-bc7b-4593-b180-3aa55469b64b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7\" (UID: \"3de1c6a3-bc7b-4593-b180-3aa55469b64b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7"
Sep 07 10:16:00.958600 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:00.958569    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3de1c6a3-bc7b-4593-b180-3aa55469b64b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7\" (UID: \"3de1c6a3-bc7b-4593-b180-3aa55469b64b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7"
Sep 07 10:16:00.958751 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:00.958643    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6m6l\" (UniqueName: \"kubernetes.io/projected/3de1c6a3-bc7b-4593-b180-3aa55469b64b-kube-api-access-w6m6l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7\" (UID: \"3de1c6a3-bc7b-4593-b180-3aa55469b64b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7"
Sep 07 10:16:00.958751 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:00.958674    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3de1c6a3-bc7b-4593-b180-3aa55469b64b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7\" (UID: \"3de1c6a3-bc7b-4593-b180-3aa55469b64b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7"
Sep 07 10:16:00.959036 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:00.959014    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3de1c6a3-bc7b-4593-b180-3aa55469b64b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7\" (UID: \"3de1c6a3-bc7b-4593-b180-3aa55469b64b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7"
Sep 07 10:16:00.959073 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:00.959044    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3de1c6a3-bc7b-4593-b180-3aa55469b64b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7\" (UID: \"3de1c6a3-bc7b-4593-b180-3aa55469b64b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7"
Sep 07 10:16:00.968930 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:00.968870    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6m6l\" (UniqueName: \"kubernetes.io/projected/3de1c6a3-bc7b-4593-b180-3aa55469b64b-kube-api-access-w6m6l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7\" (UID: \"3de1c6a3-bc7b-4593-b180-3aa55469b64b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7"
Sep 07 10:16:01.028868 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:01.028836    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7"
Sep 07 10:16:01.331344 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:01.331136    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7"]
Sep 07 10:16:01.333801 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:16:01.333756    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de1c6a3_bc7b_4593_b180_3aa55469b64b.slice/crio-8bd71f7bbc82b96c3d23aa1f658110a03f7799a69bda508ce1451e05147e574b WatchSource:0}: Error finding container 8bd71f7bbc82b96c3d23aa1f658110a03f7799a69bda508ce1451e05147e574b: Status 404 returned error can't find the container with id 8bd71f7bbc82b96c3d23aa1f658110a03f7799a69bda508ce1451e05147e574b
Sep 07 10:16:01.993773 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:01.993719    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7" event={"ID":"3de1c6a3-bc7b-4593-b180-3aa55469b64b","Type":"ContainerStarted","Data":"8bd71f7bbc82b96c3d23aa1f658110a03f7799a69bda508ce1451e05147e574b"}
Sep 07 10:16:03.781378 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:03.781342    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2jj9m"]
Sep 07 10:16:03.783576 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:03.783555    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2jj9m"
Sep 07 10:16:03.786040 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:03.786009    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 07 10:16:03.786806 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:03.786788    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-x4g6k\""
Sep 07 10:16:03.786806 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:03.786798    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 07 10:16:03.793732 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:03.793709    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2jj9m"]
Sep 07 10:16:03.882402 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:03.882363    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/851cd0a2-6d4d-48a1-a578-a267cbdd6347-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2jj9m\" (UID: \"851cd0a2-6d4d-48a1-a578-a267cbdd6347\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2jj9m"
Sep 07 10:16:03.882570 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:03.882410    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7txh\" (UniqueName: \"kubernetes.io/projected/851cd0a2-6d4d-48a1-a578-a267cbdd6347-kube-api-access-g7txh\") pod \"cert-manager-cainjector-559cdd8c87-2jj9m\" (UID: \"851cd0a2-6d4d-48a1-a578-a267cbdd6347\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2jj9m"
Sep 07 10:16:03.983519 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:03.983482    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/851cd0a2-6d4d-48a1-a578-a267cbdd6347-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2jj9m\" (UID: \"851cd0a2-6d4d-48a1-a578-a267cbdd6347\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2jj9m"
Sep 07 10:16:03.983706 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:03.983534    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7txh\" (UniqueName: \"kubernetes.io/projected/851cd0a2-6d4d-48a1-a578-a267cbdd6347-kube-api-access-g7txh\") pod \"cert-manager-cainjector-559cdd8c87-2jj9m\" (UID: \"851cd0a2-6d4d-48a1-a578-a267cbdd6347\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2jj9m"
Sep 07 10:16:03.992072 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:03.992038    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/851cd0a2-6d4d-48a1-a578-a267cbdd6347-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2jj9m\" (UID: \"851cd0a2-6d4d-48a1-a578-a267cbdd6347\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2jj9m"
Sep 07 10:16:03.992205 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:03.992141    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7txh\" (UniqueName: \"kubernetes.io/projected/851cd0a2-6d4d-48a1-a578-a267cbdd6347-kube-api-access-g7txh\") pod \"cert-manager-cainjector-559cdd8c87-2jj9m\" (UID: \"851cd0a2-6d4d-48a1-a578-a267cbdd6347\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2jj9m"
Sep 07 10:16:04.094608 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:04.094524    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2jj9m"
Sep 07 10:16:04.367453 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:04.367422    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2jj9m"]
Sep 07 10:16:04.369967 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:16:04.369928    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod851cd0a2_6d4d_48a1_a578_a267cbdd6347.slice/crio-aa7e682ddc9143e34341f2ab1bfc994193c65721e02642a4445b02c6297dddc8 WatchSource:0}: Error finding container aa7e682ddc9143e34341f2ab1bfc994193c65721e02642a4445b02c6297dddc8: Status 404 returned error can't find the container with id aa7e682ddc9143e34341f2ab1bfc994193c65721e02642a4445b02c6297dddc8
Sep 07 10:16:05.008258 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:05.008223    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2jj9m" event={"ID":"851cd0a2-6d4d-48a1-a578-a267cbdd6347","Type":"ContainerStarted","Data":"aa7e682ddc9143e34341f2ab1bfc994193c65721e02642a4445b02c6297dddc8"}
Sep 07 10:16:09.027035 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:09.027002    2562 generic.go:356] "Generic (PLEG): container finished" podID="3de1c6a3-bc7b-4593-b180-3aa55469b64b" containerID="00059a91902b38a08896f66eeee4219a997c0f9c2b939fbac7b9d300be246482" exitCode=0
Sep 07 10:16:09.027708 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:09.027091    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7" event={"ID":"3de1c6a3-bc7b-4593-b180-3aa55469b64b","Type":"ContainerDied","Data":"00059a91902b38a08896f66eeee4219a997c0f9c2b939fbac7b9d300be246482"}
Sep 07 10:16:09.028997 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:09.028975    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2jj9m" event={"ID":"851cd0a2-6d4d-48a1-a578-a267cbdd6347","Type":"ContainerStarted","Data":"991ef3fb09b9c75e7d4a95cb0d64a780aa943b8e0dc4da454915bc20d273b619"}
Sep 07 10:16:09.064682 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:09.064618    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2jj9m" podStartSLOduration=1.8259325130000001 podStartE2EDuration="6.064601219s" totalImagesPullingTime="4.238668706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:16:09.06441866 +0000 UTC m=+271.562108570" watchObservedRunningTime="2026-09-07 10:16:09.064601219 +0000 UTC m=+271.562291129"
Sep 07 10:16:11.040342 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:11.040305    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7" event={"ID":"3de1c6a3-bc7b-4593-b180-3aa55469b64b","Type":"ContainerStarted","Data":"12c88b409ba31176d53a7533c25729b554ed4cd5ea1dc5231856108612e87d3b"}
Sep 07 10:16:12.046012 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:12.045978    2562 generic.go:356] "Generic (PLEG): container finished" podID="3de1c6a3-bc7b-4593-b180-3aa55469b64b" containerID="12c88b409ba31176d53a7533c25729b554ed4cd5ea1dc5231856108612e87d3b" exitCode=0
Sep 07 10:16:12.046487 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:12.046040    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7" event={"ID":"3de1c6a3-bc7b-4593-b180-3aa55469b64b","Type":"ContainerDied","Data":"12c88b409ba31176d53a7533c25729b554ed4cd5ea1dc5231856108612e87d3b"}
Sep 07 10:16:19.076553 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:19.076467    2562 generic.go:356] "Generic (PLEG): container finished" podID="3de1c6a3-bc7b-4593-b180-3aa55469b64b" containerID="3fce28363381ee58d34800a24b6053ae082111492899d4421e1c4aecf192d466" exitCode=0
Sep 07 10:16:19.076553 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:19.076504    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7" event={"ID":"3de1c6a3-bc7b-4593-b180-3aa55469b64b","Type":"ContainerDied","Data":"3fce28363381ee58d34800a24b6053ae082111492899d4421e1c4aecf192d466"}
Sep 07 10:16:20.201304 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.201281    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7"
Sep 07 10:16:20.320220 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.320187    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3de1c6a3-bc7b-4593-b180-3aa55469b64b-bundle\") pod \"3de1c6a3-bc7b-4593-b180-3aa55469b64b\" (UID: \"3de1c6a3-bc7b-4593-b180-3aa55469b64b\") "
Sep 07 10:16:20.320394 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.320239    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3de1c6a3-bc7b-4593-b180-3aa55469b64b-util\") pod \"3de1c6a3-bc7b-4593-b180-3aa55469b64b\" (UID: \"3de1c6a3-bc7b-4593-b180-3aa55469b64b\") "
Sep 07 10:16:20.320394 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.320316    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6m6l\" (UniqueName: \"kubernetes.io/projected/3de1c6a3-bc7b-4593-b180-3aa55469b64b-kube-api-access-w6m6l\") pod \"3de1c6a3-bc7b-4593-b180-3aa55469b64b\" (UID: \"3de1c6a3-bc7b-4593-b180-3aa55469b64b\") "
Sep 07 10:16:20.320618 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.320592    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de1c6a3-bc7b-4593-b180-3aa55469b64b-bundle" (OuterVolumeSpecName: "bundle") pod "3de1c6a3-bc7b-4593-b180-3aa55469b64b" (UID: "3de1c6a3-bc7b-4593-b180-3aa55469b64b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:16:20.322430 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.322405    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de1c6a3-bc7b-4593-b180-3aa55469b64b-kube-api-access-w6m6l" (OuterVolumeSpecName: "kube-api-access-w6m6l") pod "3de1c6a3-bc7b-4593-b180-3aa55469b64b" (UID: "3de1c6a3-bc7b-4593-b180-3aa55469b64b"). InnerVolumeSpecName "kube-api-access-w6m6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:16:20.324797 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.324772    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de1c6a3-bc7b-4593-b180-3aa55469b64b-util" (OuterVolumeSpecName: "util") pod "3de1c6a3-bc7b-4593-b180-3aa55469b64b" (UID: "3de1c6a3-bc7b-4593-b180-3aa55469b64b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:16:20.421753 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.421666    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3de1c6a3-bc7b-4593-b180-3aa55469b64b-util\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:16:20.421753 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.421694    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6m6l\" (UniqueName: \"kubernetes.io/projected/3de1c6a3-bc7b-4593-b180-3aa55469b64b-kube-api-access-w6m6l\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:16:20.421753 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.421704    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3de1c6a3-bc7b-4593-b180-3aa55469b64b-bundle\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:16:20.703556 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.703518    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lz6w5"]
Sep 07 10:16:20.703818 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.703804    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de1c6a3-bc7b-4593-b180-3aa55469b64b" containerName="util"
Sep 07 10:16:20.703877 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.703820    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de1c6a3-bc7b-4593-b180-3aa55469b64b" containerName="util"
Sep 07 10:16:20.703877 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.703829    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de1c6a3-bc7b-4593-b180-3aa55469b64b" containerName="pull"
Sep 07 10:16:20.703877 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.703835    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de1c6a3-bc7b-4593-b180-3aa55469b64b" containerName="pull"
Sep 07 10:16:20.703877 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.703842    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de1c6a3-bc7b-4593-b180-3aa55469b64b" containerName="extract"
Sep 07 10:16:20.703877 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.703848    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de1c6a3-bc7b-4593-b180-3aa55469b64b" containerName="extract"
Sep 07 10:16:20.704023 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.703894    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3de1c6a3-bc7b-4593-b180-3aa55469b64b" containerName="extract"
Sep 07 10:16:20.709995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.709962    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lz6w5"
Sep 07 10:16:20.712478 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.712453    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-dr4fh\""
Sep 07 10:16:20.716434 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.716390    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lz6w5"]
Sep 07 10:16:20.825494 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.825451    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgx6j\" (UniqueName: \"kubernetes.io/projected/88c73467-8996-4748-a92e-faef3f436048-kube-api-access-mgx6j\") pod \"cert-manager-55b4d954d5-lz6w5\" (UID: \"88c73467-8996-4748-a92e-faef3f436048\") " pod="cert-manager/cert-manager-55b4d954d5-lz6w5"
Sep 07 10:16:20.825494 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.825495    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88c73467-8996-4748-a92e-faef3f436048-bound-sa-token\") pod \"cert-manager-55b4d954d5-lz6w5\" (UID: \"88c73467-8996-4748-a92e-faef3f436048\") " pod="cert-manager/cert-manager-55b4d954d5-lz6w5"
Sep 07 10:16:20.926455 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.926410    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgx6j\" (UniqueName: \"kubernetes.io/projected/88c73467-8996-4748-a92e-faef3f436048-kube-api-access-mgx6j\") pod \"cert-manager-55b4d954d5-lz6w5\" (UID: \"88c73467-8996-4748-a92e-faef3f436048\") " pod="cert-manager/cert-manager-55b4d954d5-lz6w5"
Sep 07 10:16:20.926647 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.926463    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88c73467-8996-4748-a92e-faef3f436048-bound-sa-token\") pod \"cert-manager-55b4d954d5-lz6w5\" (UID: \"88c73467-8996-4748-a92e-faef3f436048\") " pod="cert-manager/cert-manager-55b4d954d5-lz6w5"
Sep 07 10:16:20.935074 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.935045    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88c73467-8996-4748-a92e-faef3f436048-bound-sa-token\") pod \"cert-manager-55b4d954d5-lz6w5\" (UID: \"88c73467-8996-4748-a92e-faef3f436048\") " pod="cert-manager/cert-manager-55b4d954d5-lz6w5"
Sep 07 10:16:20.935225 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:20.935199    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgx6j\" (UniqueName: \"kubernetes.io/projected/88c73467-8996-4748-a92e-faef3f436048-kube-api-access-mgx6j\") pod \"cert-manager-55b4d954d5-lz6w5\" (UID: \"88c73467-8996-4748-a92e-faef3f436048\") " pod="cert-manager/cert-manager-55b4d954d5-lz6w5"
Sep 07 10:16:21.019286 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:21.019172    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lz6w5"
Sep 07 10:16:21.086850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:21.086724    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7" event={"ID":"3de1c6a3-bc7b-4593-b180-3aa55469b64b","Type":"ContainerDied","Data":"8bd71f7bbc82b96c3d23aa1f658110a03f7799a69bda508ce1451e05147e574b"}
Sep 07 10:16:21.086850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:21.086757    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftltm7"
Sep 07 10:16:21.086850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:21.086821    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bd71f7bbc82b96c3d23aa1f658110a03f7799a69bda508ce1451e05147e574b"
Sep 07 10:16:21.288124 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:21.287933    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lz6w5"]
Sep 07 10:16:21.290639 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:16:21.290612    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88c73467_8996_4748_a92e_faef3f436048.slice/crio-e76039d6daf8b6728d0476a92cd39f1d95230b95117a46e3a624b7849a7889a8 WatchSource:0}: Error finding container e76039d6daf8b6728d0476a92cd39f1d95230b95117a46e3a624b7849a7889a8: Status 404 returned error can't find the container with id e76039d6daf8b6728d0476a92cd39f1d95230b95117a46e3a624b7849a7889a8
Sep 07 10:16:22.092097 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:22.092061    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lz6w5" event={"ID":"88c73467-8996-4748-a92e-faef3f436048","Type":"ContainerStarted","Data":"6a1830edd4a7132b43b1aecb643f50e2139510368dc5642ead293e0ff6570fcf"}
Sep 07 10:16:22.092097 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:22.092103    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lz6w5" event={"ID":"88c73467-8996-4748-a92e-faef3f436048","Type":"ContainerStarted","Data":"e76039d6daf8b6728d0476a92cd39f1d95230b95117a46e3a624b7849a7889a8"}
Sep 07 10:16:22.110579 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:22.110528    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-lz6w5" podStartSLOduration=2.110514187 podStartE2EDuration="2.110514187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:16:22.110073829 +0000 UTC m=+284.607763739" watchObservedRunningTime="2026-09-07 10:16:22.110514187 +0000 UTC m=+284.608204095"
Sep 07 10:16:26.202966 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:26.202930    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nsp2p"]
Sep 07 10:16:26.255317 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:26.255280    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nsp2p"]
Sep 07 10:16:26.255450 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:26.255374    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nsp2p"
Sep 07 10:16:26.258583 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:26.258558    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 07 10:16:26.258583 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:26.258558    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-p6bcw\""
Sep 07 10:16:26.259350 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:26.259329    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 07 10:16:26.366397 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:26.366361    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9d853ee-540f-4886-904f-b7e32d365e30-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nsp2p\" (UID: \"c9d853ee-540f-4886-904f-b7e32d365e30\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nsp2p"
Sep 07 10:16:26.366397 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:26.366401    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwrkk\" (UniqueName: \"kubernetes.io/projected/c9d853ee-540f-4886-904f-b7e32d365e30-kube-api-access-bwrkk\") pod \"openshift-lws-operator-fd8ccff4c-nsp2p\" (UID: \"c9d853ee-540f-4886-904f-b7e32d365e30\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nsp2p"
Sep 07 10:16:26.467375 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:26.467284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9d853ee-540f-4886-904f-b7e32d365e30-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nsp2p\" (UID: \"c9d853ee-540f-4886-904f-b7e32d365e30\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nsp2p"
Sep 07 10:16:26.467375 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:26.467328    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrkk\" (UniqueName: \"kubernetes.io/projected/c9d853ee-540f-4886-904f-b7e32d365e30-kube-api-access-bwrkk\") pod \"openshift-lws-operator-fd8ccff4c-nsp2p\" (UID: \"c9d853ee-540f-4886-904f-b7e32d365e30\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nsp2p"
Sep 07 10:16:26.467672 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:26.467638    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9d853ee-540f-4886-904f-b7e32d365e30-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nsp2p\" (UID: \"c9d853ee-540f-4886-904f-b7e32d365e30\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nsp2p"
Sep 07 10:16:26.474973 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:26.474953    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwrkk\" (UniqueName: \"kubernetes.io/projected/c9d853ee-540f-4886-904f-b7e32d365e30-kube-api-access-bwrkk\") pod \"openshift-lws-operator-fd8ccff4c-nsp2p\" (UID: \"c9d853ee-540f-4886-904f-b7e32d365e30\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nsp2p"
Sep 07 10:16:26.564521 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:26.564488    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nsp2p"
Sep 07 10:16:26.846226 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:26.846095    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nsp2p"]
Sep 07 10:16:26.849410 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:16:26.849374    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9d853ee_540f_4886_904f_b7e32d365e30.slice/crio-089c2a94af7682e1292fb7bd83321b7107c5a17491ecd48dd725d8d977d8b4fe WatchSource:0}: Error finding container 089c2a94af7682e1292fb7bd83321b7107c5a17491ecd48dd725d8d977d8b4fe: Status 404 returned error can't find the container with id 089c2a94af7682e1292fb7bd83321b7107c5a17491ecd48dd725d8d977d8b4fe
Sep 07 10:16:27.113088 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:27.113013    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nsp2p" event={"ID":"c9d853ee-540f-4886-904f-b7e32d365e30","Type":"ContainerStarted","Data":"089c2a94af7682e1292fb7bd83321b7107c5a17491ecd48dd725d8d977d8b4fe"}
Sep 07 10:16:30.126220 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:30.126184    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nsp2p" event={"ID":"c9d853ee-540f-4886-904f-b7e32d365e30","Type":"ContainerStarted","Data":"aa0bdf72c3416f3094752b4ddbf8a592515caa2e548ad964f9d38896de65e148"}
Sep 07 10:16:30.142224 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:30.142174    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nsp2p" podStartSLOduration=1.837418719 podStartE2EDuration="4.142154297s" totalImagesPullingTime="2.304735578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:16:30.142051345 +0000 UTC m=+292.639741255" watchObservedRunningTime="2026-09-07 10:16:30.142154297 +0000 UTC m=+292.639844208"
Sep 07 10:16:37.860225 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:37.860195    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 07 10:16:54.497811 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:54.497757    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4"]
Sep 07 10:16:54.500342 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:54.500320    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4"
Sep 07 10:16:54.503057 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:54.503035    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 10:16:54.503929 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:54.503912    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m28xb\""
Sep 07 10:16:54.504015 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:54.503950    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 10:16:54.510031 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:54.510005    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4"]
Sep 07 10:16:54.568297 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:54.568261    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4\" (UID: \"5e8de9d2-5a6a-47a9-808d-1330b7b780d7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4"
Sep 07 10:16:54.568466 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:54.568313    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkhsj\" (UniqueName: \"kubernetes.io/projected/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-kube-api-access-wkhsj\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4\" (UID: \"5e8de9d2-5a6a-47a9-808d-1330b7b780d7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4"
Sep 07 10:16:54.568466 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:54.568349    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4\" (UID: \"5e8de9d2-5a6a-47a9-808d-1330b7b780d7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4"
Sep 07 10:16:54.668698 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:54.668656    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4\" (UID: \"5e8de9d2-5a6a-47a9-808d-1330b7b780d7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4"
Sep 07 10:16:54.668897 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:54.668713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkhsj\" (UniqueName: \"kubernetes.io/projected/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-kube-api-access-wkhsj\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4\" (UID: \"5e8de9d2-5a6a-47a9-808d-1330b7b780d7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4"
Sep 07 10:16:54.668897 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:54.668731    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4\" (UID: \"5e8de9d2-5a6a-47a9-808d-1330b7b780d7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4"
Sep 07 10:16:54.669136 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:54.669111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4\" (UID: \"5e8de9d2-5a6a-47a9-808d-1330b7b780d7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4"
Sep 07 10:16:54.669200 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:54.669123    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4\" (UID: \"5e8de9d2-5a6a-47a9-808d-1330b7b780d7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4"
Sep 07 10:16:54.683583 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:54.683553    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkhsj\" (UniqueName: \"kubernetes.io/projected/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-kube-api-access-wkhsj\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4\" (UID: \"5e8de9d2-5a6a-47a9-808d-1330b7b780d7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4"
Sep 07 10:16:54.810399 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:54.810319    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4"
Sep 07 10:16:55.086199 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:55.086102    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4"]
Sep 07 10:16:55.089990 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:16:55.089961    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8de9d2_5a6a_47a9_808d_1330b7b780d7.slice/crio-29f441561ffe10671d9e89940ec892420a8f0f1e5895b342eb6d285cad12d378 WatchSource:0}: Error finding container 29f441561ffe10671d9e89940ec892420a8f0f1e5895b342eb6d285cad12d378: Status 404 returned error can't find the container with id 29f441561ffe10671d9e89940ec892420a8f0f1e5895b342eb6d285cad12d378
Sep 07 10:16:55.224908 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:55.224879    2562 generic.go:356] "Generic (PLEG): container finished" podID="5e8de9d2-5a6a-47a9-808d-1330b7b780d7" containerID="45dd8e18ea0397c2fe96f0b7efd7b364a21067a7dd398b75a8e8daaa97a317d5" exitCode=0
Sep 07 10:16:55.225066 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:55.224940    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4" event={"ID":"5e8de9d2-5a6a-47a9-808d-1330b7b780d7","Type":"ContainerDied","Data":"45dd8e18ea0397c2fe96f0b7efd7b364a21067a7dd398b75a8e8daaa97a317d5"}
Sep 07 10:16:55.225066 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:55.224981    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4" event={"ID":"5e8de9d2-5a6a-47a9-808d-1330b7b780d7","Type":"ContainerStarted","Data":"29f441561ffe10671d9e89940ec892420a8f0f1e5895b342eb6d285cad12d378"}
Sep 07 10:16:55.225428 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:55.225414    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 10:16:56.231369 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.231338    2562 generic.go:356] "Generic (PLEG): container finished" podID="5e8de9d2-5a6a-47a9-808d-1330b7b780d7" containerID="17efb9d5f6d570c3652ea0363d4a48c3d59fa3815a21ea485f5aff28599df066" exitCode=0
Sep 07 10:16:56.231804 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.231400    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4" event={"ID":"5e8de9d2-5a6a-47a9-808d-1330b7b780d7","Type":"ContainerDied","Data":"17efb9d5f6d570c3652ea0363d4a48c3d59fa3815a21ea485f5aff28599df066"}
Sep 07 10:16:56.249584 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.249556    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"]
Sep 07 10:16:56.252144 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.252128    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.254989 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.254895    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 07 10:16:56.255111 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.254998    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 07 10:16:56.255496 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.255479    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 07 10:16:56.255666 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.255644    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 07 10:16:56.256051 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.256035    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q77qk\""
Sep 07 10:16:56.267357 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.267336    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"]
Sep 07 10:16:56.281354 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.281327    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1e2706d-036b-464d-8c8b-e34513f33a39-tmp\") pod \"opendatahub-operator-controller-manager-685c7f779d-jpm2r\" (UID: \"b1e2706d-036b-464d-8c8b-e34513f33a39\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.281488 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.281371    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b1e2706d-036b-464d-8c8b-e34513f33a39-manifests\") pod \"opendatahub-operator-controller-manager-685c7f779d-jpm2r\" (UID: \"b1e2706d-036b-464d-8c8b-e34513f33a39\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.281488 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.281394    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcqs9\" (UniqueName: \"kubernetes.io/projected/b1e2706d-036b-464d-8c8b-e34513f33a39-kube-api-access-lcqs9\") pod \"opendatahub-operator-controller-manager-685c7f779d-jpm2r\" (UID: \"b1e2706d-036b-464d-8c8b-e34513f33a39\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.281488 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.281441    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b1e2706d-036b-464d-8c8b-e34513f33a39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-jpm2r\" (UID: \"b1e2706d-036b-464d-8c8b-e34513f33a39\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.281628 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.281526    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1e2706d-036b-464d-8c8b-e34513f33a39-webhook-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-jpm2r\" (UID: \"b1e2706d-036b-464d-8c8b-e34513f33a39\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.382363 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.382323    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1e2706d-036b-464d-8c8b-e34513f33a39-tmp\") pod \"opendatahub-operator-controller-manager-685c7f779d-jpm2r\" (UID: \"b1e2706d-036b-464d-8c8b-e34513f33a39\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.382363 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.382363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b1e2706d-036b-464d-8c8b-e34513f33a39-manifests\") pod \"opendatahub-operator-controller-manager-685c7f779d-jpm2r\" (UID: \"b1e2706d-036b-464d-8c8b-e34513f33a39\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.382571 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.382387    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcqs9\" (UniqueName: \"kubernetes.io/projected/b1e2706d-036b-464d-8c8b-e34513f33a39-kube-api-access-lcqs9\") pod \"opendatahub-operator-controller-manager-685c7f779d-jpm2r\" (UID: \"b1e2706d-036b-464d-8c8b-e34513f33a39\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.382571 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.382410    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b1e2706d-036b-464d-8c8b-e34513f33a39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-jpm2r\" (UID: \"b1e2706d-036b-464d-8c8b-e34513f33a39\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.382571 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.382439    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1e2706d-036b-464d-8c8b-e34513f33a39-webhook-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-jpm2r\" (UID: \"b1e2706d-036b-464d-8c8b-e34513f33a39\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.382756 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.382724    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b1e2706d-036b-464d-8c8b-e34513f33a39-manifests\") pod \"opendatahub-operator-controller-manager-685c7f779d-jpm2r\" (UID: \"b1e2706d-036b-464d-8c8b-e34513f33a39\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.384796 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.384744    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1e2706d-036b-464d-8c8b-e34513f33a39-tmp\") pod \"opendatahub-operator-controller-manager-685c7f779d-jpm2r\" (UID: \"b1e2706d-036b-464d-8c8b-e34513f33a39\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.385008 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.384984    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1e2706d-036b-464d-8c8b-e34513f33a39-webhook-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-jpm2r\" (UID: \"b1e2706d-036b-464d-8c8b-e34513f33a39\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.385075 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.385032    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b1e2706d-036b-464d-8c8b-e34513f33a39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-jpm2r\" (UID: \"b1e2706d-036b-464d-8c8b-e34513f33a39\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.401597 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.401565    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcqs9\" (UniqueName: \"kubernetes.io/projected/b1e2706d-036b-464d-8c8b-e34513f33a39-kube-api-access-lcqs9\") pod \"opendatahub-operator-controller-manager-685c7f779d-jpm2r\" (UID: \"b1e2706d-036b-464d-8c8b-e34513f33a39\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.570955 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.570865    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:16:56.844178 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:56.844147    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"]
Sep 07 10:16:56.845109 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:16:56.845080    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e2706d_036b_464d_8c8b_e34513f33a39.slice/crio-ea56e27207774c9fd98795fdc117333d00a65b0206146c504ee651a13af212b6 WatchSource:0}: Error finding container ea56e27207774c9fd98795fdc117333d00a65b0206146c504ee651a13af212b6: Status 404 returned error can't find the container with id ea56e27207774c9fd98795fdc117333d00a65b0206146c504ee651a13af212b6
Sep 07 10:16:57.237442 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:57.237400    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r" event={"ID":"b1e2706d-036b-464d-8c8b-e34513f33a39","Type":"ContainerStarted","Data":"ea56e27207774c9fd98795fdc117333d00a65b0206146c504ee651a13af212b6"}
Sep 07 10:16:57.239500 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:57.239469    2562 generic.go:356] "Generic (PLEG): container finished" podID="5e8de9d2-5a6a-47a9-808d-1330b7b780d7" containerID="32f4f7d6d62542fec407cccf53b1cf64280f9c71f2eadb4395fa68403e9963ea" exitCode=0
Sep 07 10:16:57.239619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:57.239528    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4" event={"ID":"5e8de9d2-5a6a-47a9-808d-1330b7b780d7","Type":"ContainerDied","Data":"32f4f7d6d62542fec407cccf53b1cf64280f9c71f2eadb4395fa68403e9963ea"}
Sep 07 10:16:59.777331 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:59.777305    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4"
Sep 07 10:16:59.806409 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:59.806381    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkhsj\" (UniqueName: \"kubernetes.io/projected/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-kube-api-access-wkhsj\") pod \"5e8de9d2-5a6a-47a9-808d-1330b7b780d7\" (UID: \"5e8de9d2-5a6a-47a9-808d-1330b7b780d7\") "
Sep 07 10:16:59.806572 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:59.806430    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-bundle\") pod \"5e8de9d2-5a6a-47a9-808d-1330b7b780d7\" (UID: \"5e8de9d2-5a6a-47a9-808d-1330b7b780d7\") "
Sep 07 10:16:59.807135 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:59.807106    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-bundle" (OuterVolumeSpecName: "bundle") pod "5e8de9d2-5a6a-47a9-808d-1330b7b780d7" (UID: "5e8de9d2-5a6a-47a9-808d-1330b7b780d7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:16:59.808366 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:59.808335    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-kube-api-access-wkhsj" (OuterVolumeSpecName: "kube-api-access-wkhsj") pod "5e8de9d2-5a6a-47a9-808d-1330b7b780d7" (UID: "5e8de9d2-5a6a-47a9-808d-1330b7b780d7"). InnerVolumeSpecName "kube-api-access-wkhsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:16:59.907190 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:59.907159    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-util\") pod \"5e8de9d2-5a6a-47a9-808d-1330b7b780d7\" (UID: \"5e8de9d2-5a6a-47a9-808d-1330b7b780d7\") "
Sep 07 10:16:59.907309 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:59.907289    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkhsj\" (UniqueName: \"kubernetes.io/projected/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-kube-api-access-wkhsj\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:16:59.907309 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:59.907301    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-bundle\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:16:59.912353 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:16:59.912315    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-util" (OuterVolumeSpecName: "util") pod "5e8de9d2-5a6a-47a9-808d-1330b7b780d7" (UID: "5e8de9d2-5a6a-47a9-808d-1330b7b780d7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:17:00.008058 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:00.008026    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8de9d2-5a6a-47a9-808d-1330b7b780d7-util\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:17:00.255367 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:00.255341    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4"
Sep 07 10:17:00.255539 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:00.255335    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09jvc4" event={"ID":"5e8de9d2-5a6a-47a9-808d-1330b7b780d7","Type":"ContainerDied","Data":"29f441561ffe10671d9e89940ec892420a8f0f1e5895b342eb6d285cad12d378"}
Sep 07 10:17:00.255539 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:00.255472    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29f441561ffe10671d9e89940ec892420a8f0f1e5895b342eb6d285cad12d378"
Sep 07 10:17:00.256966 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:00.256940    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r" event={"ID":"b1e2706d-036b-464d-8c8b-e34513f33a39","Type":"ContainerStarted","Data":"a320f4dcb9986c93dbe0eb5eecb15f6bc6823a82e73d13fae94f4ad0c927d269"}
Sep 07 10:17:01.262419 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:01.262384    2562 generic.go:356] "Generic (PLEG): container finished" podID="b1e2706d-036b-464d-8c8b-e34513f33a39" containerID="a320f4dcb9986c93dbe0eb5eecb15f6bc6823a82e73d13fae94f4ad0c927d269" exitCode=0
Sep 07 10:17:01.262820 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:01.262434    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r" event={"ID":"b1e2706d-036b-464d-8c8b-e34513f33a39","Type":"ContainerDied","Data":"a320f4dcb9986c93dbe0eb5eecb15f6bc6823a82e73d13fae94f4ad0c927d269"}
Sep 07 10:17:02.268649 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:02.268607    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r" event={"ID":"b1e2706d-036b-464d-8c8b-e34513f33a39","Type":"ContainerStarted","Data":"e9e4be73c906cdf98f85c1baba6ab164b192272747afcc6c8934e17e44281a77"}
Sep 07 10:17:02.269050 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:02.268746    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:17:02.289052 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:02.288998    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r" podStartSLOduration=1.809665729 podStartE2EDuration="6.28898221s" totalImagesPullingTime="3.331978554s" totalInitContainerRuntime="1.147337927s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:16:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:17:02.288362596 +0000 UTC m=+324.786052505" watchObservedRunningTime="2026-09-07 10:17:02.28898221 +0000 UTC m=+324.786672119"
Sep 07 10:17:13.274439 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:13.274406    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-jpm2r"
Sep 07 10:17:16.915073 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:16.915037    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k"]
Sep 07 10:17:16.915444 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:16.915293    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8de9d2-5a6a-47a9-808d-1330b7b780d7" containerName="extract"
Sep 07 10:17:16.915444 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:16.915303    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8de9d2-5a6a-47a9-808d-1330b7b780d7" containerName="extract"
Sep 07 10:17:16.915444 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:16.915313    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8de9d2-5a6a-47a9-808d-1330b7b780d7" containerName="pull"
Sep 07 10:17:16.915444 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:16.915319    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8de9d2-5a6a-47a9-808d-1330b7b780d7" containerName="pull"
Sep 07 10:17:16.915444 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:16.915326    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8de9d2-5a6a-47a9-808d-1330b7b780d7" containerName="util"
Sep 07 10:17:16.915444 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:16.915331    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8de9d2-5a6a-47a9-808d-1330b7b780d7" containerName="util"
Sep 07 10:17:16.915444 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:16.915380    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e8de9d2-5a6a-47a9-808d-1330b7b780d7" containerName="extract"
Sep 07 10:17:16.923366 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:16.923342    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k"
Sep 07 10:17:16.925936 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:16.925907    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 07 10:17:16.925936 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:16.925923    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 07 10:17:16.926948 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:16.926921    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zr766\""
Sep 07 10:17:16.927094 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:16.927070    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 07 10:17:16.927326 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:16.927307    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 07 10:17:16.929140 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:16.929118    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k"]
Sep 07 10:17:17.026875 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:17.026829    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf7bl\" (UniqueName: \"kubernetes.io/projected/8fa26e1a-d2b7-49fe-b0ab-ca422cd60929-kube-api-access-wf7bl\") pod \"kube-auth-proxy-7959f5887b-z7h6k\" (UID: \"8fa26e1a-d2b7-49fe-b0ab-ca422cd60929\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k"
Sep 07 10:17:17.027036 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:17.026886    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa26e1a-d2b7-49fe-b0ab-ca422cd60929-tls-certs\") pod \"kube-auth-proxy-7959f5887b-z7h6k\" (UID: \"8fa26e1a-d2b7-49fe-b0ab-ca422cd60929\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k"
Sep 07 10:17:17.027036 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:17.026973    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fa26e1a-d2b7-49fe-b0ab-ca422cd60929-tmp\") pod \"kube-auth-proxy-7959f5887b-z7h6k\" (UID: \"8fa26e1a-d2b7-49fe-b0ab-ca422cd60929\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k"
Sep 07 10:17:17.127934 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:17.127894    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fa26e1a-d2b7-49fe-b0ab-ca422cd60929-tmp\") pod \"kube-auth-proxy-7959f5887b-z7h6k\" (UID: \"8fa26e1a-d2b7-49fe-b0ab-ca422cd60929\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k"
Sep 07 10:17:17.128092 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:17.127944    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf7bl\" (UniqueName: \"kubernetes.io/projected/8fa26e1a-d2b7-49fe-b0ab-ca422cd60929-kube-api-access-wf7bl\") pod \"kube-auth-proxy-7959f5887b-z7h6k\" (UID: \"8fa26e1a-d2b7-49fe-b0ab-ca422cd60929\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k"
Sep 07 10:17:17.128092 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:17.127969    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa26e1a-d2b7-49fe-b0ab-ca422cd60929-tls-certs\") pod \"kube-auth-proxy-7959f5887b-z7h6k\" (UID: \"8fa26e1a-d2b7-49fe-b0ab-ca422cd60929\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k"
Sep 07 10:17:17.128092 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:17:17.128086    2562 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 07 10:17:17.128191 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:17:17.128165    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fa26e1a-d2b7-49fe-b0ab-ca422cd60929-tls-certs podName:8fa26e1a-d2b7-49fe-b0ab-ca422cd60929 nodeName:}" failed. No retries permitted until 2026-09-07 10:17:17.628150082 +0000 UTC m=+340.125839969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8fa26e1a-d2b7-49fe-b0ab-ca422cd60929-tls-certs") pod "kube-auth-proxy-7959f5887b-z7h6k" (UID: "8fa26e1a-d2b7-49fe-b0ab-ca422cd60929") : secret "kube-auth-proxy-tls" not found
Sep 07 10:17:17.130244 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:17.130213    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fa26e1a-d2b7-49fe-b0ab-ca422cd60929-tmp\") pod \"kube-auth-proxy-7959f5887b-z7h6k\" (UID: \"8fa26e1a-d2b7-49fe-b0ab-ca422cd60929\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k"
Sep 07 10:17:17.138655 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:17.138632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf7bl\" (UniqueName: \"kubernetes.io/projected/8fa26e1a-d2b7-49fe-b0ab-ca422cd60929-kube-api-access-wf7bl\") pod \"kube-auth-proxy-7959f5887b-z7h6k\" (UID: \"8fa26e1a-d2b7-49fe-b0ab-ca422cd60929\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k"
Sep 07 10:17:17.632493 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:17.632456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa26e1a-d2b7-49fe-b0ab-ca422cd60929-tls-certs\") pod \"kube-auth-proxy-7959f5887b-z7h6k\" (UID: \"8fa26e1a-d2b7-49fe-b0ab-ca422cd60929\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k"
Sep 07 10:17:17.635113 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:17.635083    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa26e1a-d2b7-49fe-b0ab-ca422cd60929-tls-certs\") pod \"kube-auth-proxy-7959f5887b-z7h6k\" (UID: \"8fa26e1a-d2b7-49fe-b0ab-ca422cd60929\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k"
Sep 07 10:17:17.835031 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:17.834992    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k"
Sep 07 10:17:18.016856 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.016826    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"]
Sep 07 10:17:18.020117 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.020100    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.022649 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.022626    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 07 10:17:18.023256 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.023228    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 07 10:17:18.023383 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.023366    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9x86w\""
Sep 07 10:17:18.023430 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.023397    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 07 10:17:18.023631 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.023616    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 07 10:17:18.033276 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.033253    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"]
Sep 07 10:17:18.036582 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.036559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.036682 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.036600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmjrb\" (UniqueName: \"kubernetes.io/projected/d096ebe3-efed-4ffb-9a95-792242cb09c5-kube-api-access-mmjrb\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.036682 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.036648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.036849 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.036805    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.036908 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.036891    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.036972 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.036923    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d096ebe3-efed-4ffb-9a95-792242cb09c5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.036972 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.036956    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.102951 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.102925    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k"]
Sep 07 10:17:18.105312 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:17:18.105284    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa26e1a_d2b7_49fe_b0ab_ca422cd60929.slice/crio-e9f2847fa24c06ba8f0fea1376bb98381a0408ca0191ad49debd2179dfa5183d WatchSource:0}: Error finding container e9f2847fa24c06ba8f0fea1376bb98381a0408ca0191ad49debd2179dfa5183d: Status 404 returned error can't find the container with id e9f2847fa24c06ba8f0fea1376bb98381a0408ca0191ad49debd2179dfa5183d
Sep 07 10:17:18.137587 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.137524    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.137587 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.137559    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjrb\" (UniqueName: \"kubernetes.io/projected/d096ebe3-efed-4ffb-9a95-792242cb09c5-kube-api-access-mmjrb\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.137587 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.137582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.137839 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.137602    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.137839 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.137643    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.137839 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.137661    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d096ebe3-efed-4ffb-9a95-792242cb09c5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.137839 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.137677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.138322 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.138294    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.140099 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.140071    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.140204 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.140119    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.140340 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.140323    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d096ebe3-efed-4ffb-9a95-792242cb09c5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.140466 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.140448    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.147079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.147037    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.148180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.148161    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjrb\" (UniqueName: \"kubernetes.io/projected/d096ebe3-efed-4ffb-9a95-792242cb09c5-kube-api-access-mmjrb\") pod \"istiod-openshift-gateway-7b7995d5cd-kcvg6\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.329685 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.329648    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:18.333335 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.333303    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k" event={"ID":"8fa26e1a-d2b7-49fe-b0ab-ca422cd60929","Type":"ContainerStarted","Data":"e9f2847fa24c06ba8f0fea1376bb98381a0408ca0191ad49debd2179dfa5183d"}
Sep 07 10:17:18.602364 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:18.602338    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"]
Sep 07 10:17:18.606266 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:17:18.606176    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd096ebe3_efed_4ffb_9a95_792242cb09c5.slice/crio-0b87866981540372715512557c670c2a2497ca875935a869ab8eca502be3dbec WatchSource:0}: Error finding container 0b87866981540372715512557c670c2a2497ca875935a869ab8eca502be3dbec: Status 404 returned error can't find the container with id 0b87866981540372715512557c670c2a2497ca875935a869ab8eca502be3dbec
Sep 07 10:17:19.340716 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:19.340675    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6" event={"ID":"d096ebe3-efed-4ffb-9a95-792242cb09c5","Type":"ContainerStarted","Data":"0b87866981540372715512557c670c2a2497ca875935a869ab8eca502be3dbec"}
Sep 07 10:17:21.352259 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:21.352219    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k" event={"ID":"8fa26e1a-d2b7-49fe-b0ab-ca422cd60929","Type":"ContainerStarted","Data":"5143b70291d6b069f69c93b1a999f050b9c2c9263afbbe85200cad02beb8fa86"}
Sep 07 10:17:21.369023 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:21.368934    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7959f5887b-z7h6k" podStartSLOduration=2.797770395 podStartE2EDuration="5.368918928s" totalImagesPullingTime="2.571148533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:17:21.368018418 +0000 UTC m=+343.865708352" watchObservedRunningTime="2026-09-07 10:17:21.368918928 +0000 UTC m=+343.866608836"
Sep 07 10:17:22.405366 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:22.405326    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 10:17:22.405629 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:22.405395    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 10:17:23.363570 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:23.363536    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6" event={"ID":"d096ebe3-efed-4ffb-9a95-792242cb09c5","Type":"ContainerStarted","Data":"79f3f00b59b4c47ff241a028ed3dc09a2566138103be10e320e27ae230dc36ed"}
Sep 07 10:17:23.363793 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:23.363773    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:23.365224 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:23.365190    2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-kcvg6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 10:17:23.365325 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:23.365263    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6" podUID="d096ebe3-efed-4ffb-9a95-792242cb09c5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 10:17:23.387201 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:23.387154    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6" podStartSLOduration=2.5902264280000002 podStartE2EDuration="6.387139639s" totalImagesPullingTime="3.796913211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:17:23.385666435 +0000 UTC m=+345.883356368" watchObservedRunningTime="2026-09-07 10:17:23.387139639 +0000 UTC m=+345.884829548"
Sep 07 10:17:24.369054 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:24.369028    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:26.792920 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:26.792886    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kjmb6"]
Sep 07 10:17:26.796199 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:26.796178    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-kjmb6"
Sep 07 10:17:26.798824 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:26.798805    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 07 10:17:26.806552 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:26.806529    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kjmb6"]
Sep 07 10:17:26.914688 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:26.914647    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/489fcc35-40b7-48d0-93f7-ecfb91e71d19-tls-certs\") pod \"model-serving-api-54d89bd5fc-kjmb6\" (UID: \"489fcc35-40b7-48d0-93f7-ecfb91e71d19\") " pod="opendatahub/model-serving-api-54d89bd5fc-kjmb6"
Sep 07 10:17:26.914896 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:26.914725    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h72lp\" (UniqueName: \"kubernetes.io/projected/489fcc35-40b7-48d0-93f7-ecfb91e71d19-kube-api-access-h72lp\") pod \"model-serving-api-54d89bd5fc-kjmb6\" (UID: \"489fcc35-40b7-48d0-93f7-ecfb91e71d19\") " pod="opendatahub/model-serving-api-54d89bd5fc-kjmb6"
Sep 07 10:17:27.015477 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:27.015438    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/489fcc35-40b7-48d0-93f7-ecfb91e71d19-tls-certs\") pod \"model-serving-api-54d89bd5fc-kjmb6\" (UID: \"489fcc35-40b7-48d0-93f7-ecfb91e71d19\") " pod="opendatahub/model-serving-api-54d89bd5fc-kjmb6"
Sep 07 10:17:27.015631 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:27.015501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h72lp\" (UniqueName: \"kubernetes.io/projected/489fcc35-40b7-48d0-93f7-ecfb91e71d19-kube-api-access-h72lp\") pod \"model-serving-api-54d89bd5fc-kjmb6\" (UID: \"489fcc35-40b7-48d0-93f7-ecfb91e71d19\") " pod="opendatahub/model-serving-api-54d89bd5fc-kjmb6"
Sep 07 10:17:27.017966 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:27.017941    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/489fcc35-40b7-48d0-93f7-ecfb91e71d19-tls-certs\") pod \"model-serving-api-54d89bd5fc-kjmb6\" (UID: \"489fcc35-40b7-48d0-93f7-ecfb91e71d19\") " pod="opendatahub/model-serving-api-54d89bd5fc-kjmb6"
Sep 07 10:17:27.023423 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:27.023389    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h72lp\" (UniqueName: \"kubernetes.io/projected/489fcc35-40b7-48d0-93f7-ecfb91e71d19-kube-api-access-h72lp\") pod \"model-serving-api-54d89bd5fc-kjmb6\" (UID: \"489fcc35-40b7-48d0-93f7-ecfb91e71d19\") " pod="opendatahub/model-serving-api-54d89bd5fc-kjmb6"
Sep 07 10:17:27.106697 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:27.106603    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-kjmb6"
Sep 07 10:17:27.374556 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:27.374521    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kjmb6"]
Sep 07 10:17:27.376642 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:17:27.376617    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489fcc35_40b7_48d0_93f7_ecfb91e71d19.slice/crio-bfb7f88c0e65f2077da8705aacee2031bf69c1119ac32536444aa53bd4bcf4bd WatchSource:0}: Error finding container bfb7f88c0e65f2077da8705aacee2031bf69c1119ac32536444aa53bd4bcf4bd: Status 404 returned error can't find the container with id bfb7f88c0e65f2077da8705aacee2031bf69c1119ac32536444aa53bd4bcf4bd
Sep 07 10:17:27.387488 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:27.382797    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-kjmb6" event={"ID":"489fcc35-40b7-48d0-93f7-ecfb91e71d19","Type":"ContainerStarted","Data":"bfb7f88c0e65f2077da8705aacee2031bf69c1119ac32536444aa53bd4bcf4bd"}
Sep 07 10:17:29.392033 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:29.391941    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-kjmb6" event={"ID":"489fcc35-40b7-48d0-93f7-ecfb91e71d19","Type":"ContainerStarted","Data":"ae5a775ff77bb92ba964e0672da644bd08e436ba90bc029a56ada91d3fcc6341"}
Sep 07 10:17:29.392372 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:29.392089    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-kjmb6"
Sep 07 10:17:29.409281 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:29.409225    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-kjmb6" podStartSLOduration=1.733716382 podStartE2EDuration="3.409209553s" totalImagesPullingTime="1.675493171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:17:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:17:29.408591789 +0000 UTC m=+351.906281698" watchObservedRunningTime="2026-09-07 10:17:29.409209553 +0000 UTC m=+351.906899461"
Sep 07 10:17:36.570329 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.570239    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fz2ql"]
Sep 07 10:17:36.574245 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.574218    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fz2ql"
Sep 07 10:17:36.578455 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.578428    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-fnc65\""
Sep 07 10:17:36.579138 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.579114    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 07 10:17:36.609020 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.608988    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hww6q"]
Sep 07 10:17:36.611581 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.611553    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fz2ql"]
Sep 07 10:17:36.611723 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.611666    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hww6q"
Sep 07 10:17:36.617754 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.617629    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 07 10:17:36.621745 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.621723    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-996q8\""
Sep 07 10:17:36.638796 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.638735    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hww6q"]
Sep 07 10:17:36.700849 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.700808    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84ac334d-8bfc-40cd-81cd-0e12f7179acc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hww6q\" (UID: \"84ac334d-8bfc-40cd-81cd-0e12f7179acc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hww6q"
Sep 07 10:17:36.700849 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.700851    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j796x\" (UniqueName: \"kubernetes.io/projected/84ac334d-8bfc-40cd-81cd-0e12f7179acc-kube-api-access-j796x\") pod \"llmisvc-controller-manager-777c6bd7d5-hww6q\" (UID: \"84ac334d-8bfc-40cd-81cd-0e12f7179acc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hww6q"
Sep 07 10:17:36.701053 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.700870    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2774810d-622e-4fce-9911-1e9684c90ab3-cert\") pod \"kserve-controller-manager-d6f4b65b8-fz2ql\" (UID: \"2774810d-622e-4fce-9911-1e9684c90ab3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fz2ql"
Sep 07 10:17:36.701053 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.700951    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqwpf\" (UniqueName: \"kubernetes.io/projected/2774810d-622e-4fce-9911-1e9684c90ab3-kube-api-access-cqwpf\") pod \"kserve-controller-manager-d6f4b65b8-fz2ql\" (UID: \"2774810d-622e-4fce-9911-1e9684c90ab3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fz2ql"
Sep 07 10:17:36.802034 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.801992    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84ac334d-8bfc-40cd-81cd-0e12f7179acc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hww6q\" (UID: \"84ac334d-8bfc-40cd-81cd-0e12f7179acc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hww6q"
Sep 07 10:17:36.802197 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.802045    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j796x\" (UniqueName: \"kubernetes.io/projected/84ac334d-8bfc-40cd-81cd-0e12f7179acc-kube-api-access-j796x\") pod \"llmisvc-controller-manager-777c6bd7d5-hww6q\" (UID: \"84ac334d-8bfc-40cd-81cd-0e12f7179acc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hww6q"
Sep 07 10:17:36.802197 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.802161    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2774810d-622e-4fce-9911-1e9684c90ab3-cert\") pod \"kserve-controller-manager-d6f4b65b8-fz2ql\" (UID: \"2774810d-622e-4fce-9911-1e9684c90ab3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fz2ql"
Sep 07 10:17:36.802286 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.802247    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqwpf\" (UniqueName: \"kubernetes.io/projected/2774810d-622e-4fce-9911-1e9684c90ab3-kube-api-access-cqwpf\") pod \"kserve-controller-manager-d6f4b65b8-fz2ql\" (UID: \"2774810d-622e-4fce-9911-1e9684c90ab3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fz2ql"
Sep 07 10:17:36.804532 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.804503    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2774810d-622e-4fce-9911-1e9684c90ab3-cert\") pod \"kserve-controller-manager-d6f4b65b8-fz2ql\" (UID: \"2774810d-622e-4fce-9911-1e9684c90ab3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fz2ql"
Sep 07 10:17:36.804639 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.804540    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84ac334d-8bfc-40cd-81cd-0e12f7179acc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hww6q\" (UID: \"84ac334d-8bfc-40cd-81cd-0e12f7179acc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hww6q"
Sep 07 10:17:36.813039 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.813018    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j796x\" (UniqueName: \"kubernetes.io/projected/84ac334d-8bfc-40cd-81cd-0e12f7179acc-kube-api-access-j796x\") pod \"llmisvc-controller-manager-777c6bd7d5-hww6q\" (UID: \"84ac334d-8bfc-40cd-81cd-0e12f7179acc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hww6q"
Sep 07 10:17:36.813158 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.813100    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqwpf\" (UniqueName: \"kubernetes.io/projected/2774810d-622e-4fce-9911-1e9684c90ab3-kube-api-access-cqwpf\") pod \"kserve-controller-manager-d6f4b65b8-fz2ql\" (UID: \"2774810d-622e-4fce-9911-1e9684c90ab3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fz2ql"
Sep 07 10:17:36.885352 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.885265    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fz2ql"
Sep 07 10:17:36.921438 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:36.921403    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hww6q"
Sep 07 10:17:37.169645 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:37.169621    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fz2ql"]
Sep 07 10:17:37.170858 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:17:37.170832    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2774810d_622e_4fce_9911_1e9684c90ab3.slice/crio-854b73b6d5bc6c81077931de7f7f81535fb64016e000450abc0eaa76965ad5ad WatchSource:0}: Error finding container 854b73b6d5bc6c81077931de7f7f81535fb64016e000450abc0eaa76965ad5ad: Status 404 returned error can't find the container with id 854b73b6d5bc6c81077931de7f7f81535fb64016e000450abc0eaa76965ad5ad
Sep 07 10:17:37.211467 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:37.211249    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hww6q"]
Sep 07 10:17:37.213885 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:17:37.213858    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod84ac334d_8bfc_40cd_81cd_0e12f7179acc.slice/crio-789606d1fdf39be4ee5c93181865453cd4a76ddf0da3aa6bb22319ee3b1688c2 WatchSource:0}: Error finding container 789606d1fdf39be4ee5c93181865453cd4a76ddf0da3aa6bb22319ee3b1688c2: Status 404 returned error can't find the container with id 789606d1fdf39be4ee5c93181865453cd4a76ddf0da3aa6bb22319ee3b1688c2
Sep 07 10:17:37.427887 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:37.427792    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hww6q" event={"ID":"84ac334d-8bfc-40cd-81cd-0e12f7179acc","Type":"ContainerStarted","Data":"789606d1fdf39be4ee5c93181865453cd4a76ddf0da3aa6bb22319ee3b1688c2"}
Sep 07 10:17:37.428970 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:37.428947    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fz2ql" event={"ID":"2774810d-622e-4fce-9911-1e9684c90ab3","Type":"ContainerStarted","Data":"854b73b6d5bc6c81077931de7f7f81535fb64016e000450abc0eaa76965ad5ad"}
Sep 07 10:17:38.536899 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:38.536841    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"]
Sep 07 10:17:38.537343 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:38.537118    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6" podUID="d096ebe3-efed-4ffb-9a95-792242cb09c5" containerName="discovery" containerID="cri-o://79f3f00b59b4c47ff241a028ed3dc09a2566138103be10e320e27ae230dc36ed" gracePeriod=30
Sep 07 10:17:39.330746 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.330717    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:39.427126 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.427087    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-kubeconfig\") pod \"d096ebe3-efed-4ffb-9a95-792242cb09c5\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") "
Sep 07 10:17:39.427306 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.427145    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d096ebe3-efed-4ffb-9a95-792242cb09c5-local-certs\") pod \"d096ebe3-efed-4ffb-9a95-792242cb09c5\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") "
Sep 07 10:17:39.427306 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.427269    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmjrb\" (UniqueName: \"kubernetes.io/projected/d096ebe3-efed-4ffb-9a95-792242cb09c5-kube-api-access-mmjrb\") pod \"d096ebe3-efed-4ffb-9a95-792242cb09c5\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") "
Sep 07 10:17:39.427416 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.427340    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-csr-dns-cert\") pod \"d096ebe3-efed-4ffb-9a95-792242cb09c5\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") "
Sep 07 10:17:39.427416 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.427369    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-token\") pod \"d096ebe3-efed-4ffb-9a95-792242cb09c5\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") "
Sep 07 10:17:39.427593 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.427572    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-cacerts\") pod \"d096ebe3-efed-4ffb-9a95-792242cb09c5\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") "
Sep 07 10:17:39.427699 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.427626    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-csr-ca-configmap\") pod \"d096ebe3-efed-4ffb-9a95-792242cb09c5\" (UID: \"d096ebe3-efed-4ffb-9a95-792242cb09c5\") "
Sep 07 10:17:39.428302 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.428248    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d096ebe3-efed-4ffb-9a95-792242cb09c5" (UID: "d096ebe3-efed-4ffb-9a95-792242cb09c5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:17:39.430081 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.430050    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d096ebe3-efed-4ffb-9a95-792242cb09c5-local-certs" (OuterVolumeSpecName: "local-certs") pod "d096ebe3-efed-4ffb-9a95-792242cb09c5" (UID: "d096ebe3-efed-4ffb-9a95-792242cb09c5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:17:39.430249 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.430222    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d096ebe3-efed-4ffb-9a95-792242cb09c5" (UID: "d096ebe3-efed-4ffb-9a95-792242cb09c5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:17:39.430392 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.430361    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d096ebe3-efed-4ffb-9a95-792242cb09c5-kube-api-access-mmjrb" (OuterVolumeSpecName: "kube-api-access-mmjrb") pod "d096ebe3-efed-4ffb-9a95-792242cb09c5" (UID: "d096ebe3-efed-4ffb-9a95-792242cb09c5"). InnerVolumeSpecName "kube-api-access-mmjrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:17:39.430494 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.430408    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d096ebe3-efed-4ffb-9a95-792242cb09c5" (UID: "d096ebe3-efed-4ffb-9a95-792242cb09c5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:17:39.430494 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.430480    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-cacerts" (OuterVolumeSpecName: "cacerts") pod "d096ebe3-efed-4ffb-9a95-792242cb09c5" (UID: "d096ebe3-efed-4ffb-9a95-792242cb09c5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:17:39.430687 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.430667    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "d096ebe3-efed-4ffb-9a95-792242cb09c5" (UID: "d096ebe3-efed-4ffb-9a95-792242cb09c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:17:39.441147 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.441115    2562 generic.go:356] "Generic (PLEG): container finished" podID="d096ebe3-efed-4ffb-9a95-792242cb09c5" containerID="79f3f00b59b4c47ff241a028ed3dc09a2566138103be10e320e27ae230dc36ed" exitCode=0
Sep 07 10:17:39.441284 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.441167    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"
Sep 07 10:17:39.441284 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.441199    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6" event={"ID":"d096ebe3-efed-4ffb-9a95-792242cb09c5","Type":"ContainerDied","Data":"79f3f00b59b4c47ff241a028ed3dc09a2566138103be10e320e27ae230dc36ed"}
Sep 07 10:17:39.441284 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.441268    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6" event={"ID":"d096ebe3-efed-4ffb-9a95-792242cb09c5","Type":"ContainerDied","Data":"0b87866981540372715512557c670c2a2497ca875935a869ab8eca502be3dbec"}
Sep 07 10:17:39.441432 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.441312    2562 scope.go:117] "RemoveContainer" containerID="79f3f00b59b4c47ff241a028ed3dc09a2566138103be10e320e27ae230dc36ed"
Sep 07 10:17:39.475065 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.475033    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"]
Sep 07 10:17:39.494283 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.494248    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kcvg6"]
Sep 07 10:17:39.528917 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.528879    2562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-cacerts\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:17:39.528917 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.528912    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-csr-ca-configmap\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:17:39.528917 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.528924    2562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-kubeconfig\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:17:39.529165 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.528934    2562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d096ebe3-efed-4ffb-9a95-792242cb09c5-local-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:17:39.529165 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.528942    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmjrb\" (UniqueName: \"kubernetes.io/projected/d096ebe3-efed-4ffb-9a95-792242cb09c5-kube-api-access-mmjrb\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:17:39.529165 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.528950    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-csr-dns-cert\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:17:39.529165 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.528958    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d096ebe3-efed-4ffb-9a95-792242cb09c5-istio-token\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:17:39.896682 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.896618    2562 scope.go:117] "RemoveContainer" containerID="79f3f00b59b4c47ff241a028ed3dc09a2566138103be10e320e27ae230dc36ed"
Sep 07 10:17:39.897070 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:17:39.897045    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f3f00b59b4c47ff241a028ed3dc09a2566138103be10e320e27ae230dc36ed\": container with ID starting with 79f3f00b59b4c47ff241a028ed3dc09a2566138103be10e320e27ae230dc36ed not found: ID does not exist" containerID="79f3f00b59b4c47ff241a028ed3dc09a2566138103be10e320e27ae230dc36ed"
Sep 07 10:17:39.897122 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.897081    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f3f00b59b4c47ff241a028ed3dc09a2566138103be10e320e27ae230dc36ed"} err="failed to get container status \"79f3f00b59b4c47ff241a028ed3dc09a2566138103be10e320e27ae230dc36ed\": rpc error: code = NotFound desc = could not find container \"79f3f00b59b4c47ff241a028ed3dc09a2566138103be10e320e27ae230dc36ed\": container with ID starting with 79f3f00b59b4c47ff241a028ed3dc09a2566138103be10e320e27ae230dc36ed not found: ID does not exist"
Sep 07 10:17:39.964042 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:39.964010    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d096ebe3-efed-4ffb-9a95-792242cb09c5" path="/var/lib/kubelet/pods/d096ebe3-efed-4ffb-9a95-792242cb09c5/volumes"
Sep 07 10:17:40.400092 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:40.400064    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-kjmb6"
Sep 07 10:17:40.448309 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:40.448268    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hww6q" event={"ID":"84ac334d-8bfc-40cd-81cd-0e12f7179acc","Type":"ContainerStarted","Data":"8a22cc59d50a7f86064f18ba591a35a50f4b73b9ee3f438d4496eb7b378bdd90"}
Sep 07 10:17:40.448497 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:40.448459    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hww6q"
Sep 07 10:17:40.449924 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:40.449898    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fz2ql" event={"ID":"2774810d-622e-4fce-9911-1e9684c90ab3","Type":"ContainerStarted","Data":"ff986ea63b88a41ee06a76db76b865fa7d48fdf80477e0620998a347b35fd560"}
Sep 07 10:17:40.450028 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:40.449952    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fz2ql"
Sep 07 10:17:40.467350 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:40.467297    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hww6q" podStartSLOduration=1.774989581 podStartE2EDuration="4.467283375s" totalImagesPullingTime="2.692293794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:17:40.465611705 +0000 UTC m=+362.963301634" watchObservedRunningTime="2026-09-07 10:17:40.467283375 +0000 UTC m=+362.964973284"
Sep 07 10:17:40.482480 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:17:40.482430    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fz2ql" podStartSLOduration=1.750712815 podStartE2EDuration="4.482417884s" totalImagesPullingTime="2.731705069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:17:40.480810532 +0000 UTC m=+362.978500468" watchObservedRunningTime="2026-09-07 10:17:40.482417884 +0000 UTC m=+362.980107793"
Sep 07 10:18:11.456288 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:11.456254    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hww6q"
Sep 07 10:18:11.459343 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:11.459320    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fz2ql"
Sep 07 10:18:30.565846 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.565811    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n"]
Sep 07 10:18:30.566215 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.566135    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d096ebe3-efed-4ffb-9a95-792242cb09c5" containerName="discovery"
Sep 07 10:18:30.566215 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.566147    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d096ebe3-efed-4ffb-9a95-792242cb09c5" containerName="discovery"
Sep 07 10:18:30.566215 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.566205    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d096ebe3-efed-4ffb-9a95-792242cb09c5" containerName="discovery"
Sep 07 10:18:30.570170 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.570135    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n"
Sep 07 10:18:30.573177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.573154    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m28xb\""
Sep 07 10:18:30.573311 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.573157    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 10:18:30.574025 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.574008    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 10:18:30.576532 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.576509    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n"]
Sep 07 10:18:30.616127 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.616093    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vpdm\" (UniqueName: \"kubernetes.io/projected/d29a398c-aa99-4486-bc21-5366b2c613fc-kube-api-access-5vpdm\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n\" (UID: \"d29a398c-aa99-4486-bc21-5366b2c613fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n"
Sep 07 10:18:30.616305 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.616163    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d29a398c-aa99-4486-bc21-5366b2c613fc-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n\" (UID: \"d29a398c-aa99-4486-bc21-5366b2c613fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n"
Sep 07 10:18:30.616305 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.616230    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d29a398c-aa99-4486-bc21-5366b2c613fc-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n\" (UID: \"d29a398c-aa99-4486-bc21-5366b2c613fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n"
Sep 07 10:18:30.663520 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.663492    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv"]
Sep 07 10:18:30.665938 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.665922    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv"
Sep 07 10:18:30.674599 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.674567    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv"]
Sep 07 10:18:30.717355 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.717320    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/853a27cb-a92e-4dd9-a24b-972c134eb3fd-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv\" (UID: \"853a27cb-a92e-4dd9-a24b-972c134eb3fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv"
Sep 07 10:18:30.717557 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.717373    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d29a398c-aa99-4486-bc21-5366b2c613fc-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n\" (UID: \"d29a398c-aa99-4486-bc21-5366b2c613fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n"
Sep 07 10:18:30.717557 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.717438    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vpdm\" (UniqueName: \"kubernetes.io/projected/d29a398c-aa99-4486-bc21-5366b2c613fc-kube-api-access-5vpdm\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n\" (UID: \"d29a398c-aa99-4486-bc21-5366b2c613fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n"
Sep 07 10:18:30.717557 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.717476    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrjg\" (UniqueName: \"kubernetes.io/projected/853a27cb-a92e-4dd9-a24b-972c134eb3fd-kube-api-access-dlrjg\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv\" (UID: \"853a27cb-a92e-4dd9-a24b-972c134eb3fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv"
Sep 07 10:18:30.717557 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.717536    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d29a398c-aa99-4486-bc21-5366b2c613fc-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n\" (UID: \"d29a398c-aa99-4486-bc21-5366b2c613fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n"
Sep 07 10:18:30.717720 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.717585    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/853a27cb-a92e-4dd9-a24b-972c134eb3fd-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv\" (UID: \"853a27cb-a92e-4dd9-a24b-972c134eb3fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv"
Sep 07 10:18:30.717720 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.717712    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d29a398c-aa99-4486-bc21-5366b2c613fc-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n\" (UID: \"d29a398c-aa99-4486-bc21-5366b2c613fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n"
Sep 07 10:18:30.717830 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.717814    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d29a398c-aa99-4486-bc21-5366b2c613fc-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n\" (UID: \"d29a398c-aa99-4486-bc21-5366b2c613fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n"
Sep 07 10:18:30.726510 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.726487    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vpdm\" (UniqueName: \"kubernetes.io/projected/d29a398c-aa99-4486-bc21-5366b2c613fc-kube-api-access-5vpdm\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n\" (UID: \"d29a398c-aa99-4486-bc21-5366b2c613fc\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n"
Sep 07 10:18:30.763538 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.763508    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd"]
Sep 07 10:18:30.766000 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.765983    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd"
Sep 07 10:18:30.775540 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.775513    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd"]
Sep 07 10:18:30.818799 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.818690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrjg\" (UniqueName: \"kubernetes.io/projected/853a27cb-a92e-4dd9-a24b-972c134eb3fd-kube-api-access-dlrjg\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv\" (UID: \"853a27cb-a92e-4dd9-a24b-972c134eb3fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv"
Sep 07 10:18:30.818799 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.818740    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72deda39-62a4-4010-876b-3b0fa7a73cc6-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd\" (UID: \"72deda39-62a4-4010-876b-3b0fa7a73cc6\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd"
Sep 07 10:18:30.818799 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.818791    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/853a27cb-a92e-4dd9-a24b-972c134eb3fd-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv\" (UID: \"853a27cb-a92e-4dd9-a24b-972c134eb3fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv"
Sep 07 10:18:30.819033 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.818933    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72deda39-62a4-4010-876b-3b0fa7a73cc6-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd\" (UID: \"72deda39-62a4-4010-876b-3b0fa7a73cc6\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd"
Sep 07 10:18:30.819033 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.818970    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/853a27cb-a92e-4dd9-a24b-972c134eb3fd-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv\" (UID: \"853a27cb-a92e-4dd9-a24b-972c134eb3fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv"
Sep 07 10:18:30.819033 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.818995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn5qd\" (UniqueName: \"kubernetes.io/projected/72deda39-62a4-4010-876b-3b0fa7a73cc6-kube-api-access-pn5qd\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd\" (UID: \"72deda39-62a4-4010-876b-3b0fa7a73cc6\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd"
Sep 07 10:18:30.819135 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.819125    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/853a27cb-a92e-4dd9-a24b-972c134eb3fd-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv\" (UID: \"853a27cb-a92e-4dd9-a24b-972c134eb3fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv"
Sep 07 10:18:30.819246 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.819231    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/853a27cb-a92e-4dd9-a24b-972c134eb3fd-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv\" (UID: \"853a27cb-a92e-4dd9-a24b-972c134eb3fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv"
Sep 07 10:18:30.828041 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.828018    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrjg\" (UniqueName: \"kubernetes.io/projected/853a27cb-a92e-4dd9-a24b-972c134eb3fd-kube-api-access-dlrjg\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv\" (UID: \"853a27cb-a92e-4dd9-a24b-972c134eb3fd\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv"
Sep 07 10:18:30.864969 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.864938    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7"]
Sep 07 10:18:30.867294 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.867275    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7"
Sep 07 10:18:30.876823 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.876795    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7"]
Sep 07 10:18:30.883072 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.883050    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n"
Sep 07 10:18:30.920342 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.920301    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72deda39-62a4-4010-876b-3b0fa7a73cc6-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd\" (UID: \"72deda39-62a4-4010-876b-3b0fa7a73cc6\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd"
Sep 07 10:18:30.920530 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.920359    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn5qd\" (UniqueName: \"kubernetes.io/projected/72deda39-62a4-4010-876b-3b0fa7a73cc6-kube-api-access-pn5qd\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd\" (UID: \"72deda39-62a4-4010-876b-3b0fa7a73cc6\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd"
Sep 07 10:18:30.920530 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.920433    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b24e8260-971a-414c-8293-228a5c1f57f3-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7\" (UID: \"b24e8260-971a-414c-8293-228a5c1f57f3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7"
Sep 07 10:18:30.920530 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.920461    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb44n\" (UniqueName: \"kubernetes.io/projected/b24e8260-971a-414c-8293-228a5c1f57f3-kube-api-access-cb44n\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7\" (UID: \"b24e8260-971a-414c-8293-228a5c1f57f3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7"
Sep 07 10:18:30.920530 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.920513    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72deda39-62a4-4010-876b-3b0fa7a73cc6-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd\" (UID: \"72deda39-62a4-4010-876b-3b0fa7a73cc6\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd"
Sep 07 10:18:30.920794 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.920574    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b24e8260-971a-414c-8293-228a5c1f57f3-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7\" (UID: \"b24e8260-971a-414c-8293-228a5c1f57f3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7"
Sep 07 10:18:30.920794 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.920710    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72deda39-62a4-4010-876b-3b0fa7a73cc6-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd\" (UID: \"72deda39-62a4-4010-876b-3b0fa7a73cc6\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd"
Sep 07 10:18:30.920937 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.920879    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72deda39-62a4-4010-876b-3b0fa7a73cc6-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd\" (UID: \"72deda39-62a4-4010-876b-3b0fa7a73cc6\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd"
Sep 07 10:18:30.929022 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.928997    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn5qd\" (UniqueName: \"kubernetes.io/projected/72deda39-62a4-4010-876b-3b0fa7a73cc6-kube-api-access-pn5qd\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd\" (UID: \"72deda39-62a4-4010-876b-3b0fa7a73cc6\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd"
Sep 07 10:18:30.975242 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:30.975210    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv"
Sep 07 10:18:31.022703 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.022162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b24e8260-971a-414c-8293-228a5c1f57f3-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7\" (UID: \"b24e8260-971a-414c-8293-228a5c1f57f3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7"
Sep 07 10:18:31.022703 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.022296    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b24e8260-971a-414c-8293-228a5c1f57f3-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7\" (UID: \"b24e8260-971a-414c-8293-228a5c1f57f3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7"
Sep 07 10:18:31.022703 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.022328    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb44n\" (UniqueName: \"kubernetes.io/projected/b24e8260-971a-414c-8293-228a5c1f57f3-kube-api-access-cb44n\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7\" (UID: \"b24e8260-971a-414c-8293-228a5c1f57f3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7"
Sep 07 10:18:31.023084 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.022742    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b24e8260-971a-414c-8293-228a5c1f57f3-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7\" (UID: \"b24e8260-971a-414c-8293-228a5c1f57f3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7"
Sep 07 10:18:31.023234 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.023211    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b24e8260-971a-414c-8293-228a5c1f57f3-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7\" (UID: \"b24e8260-971a-414c-8293-228a5c1f57f3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7"
Sep 07 10:18:31.031838 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.030891    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb44n\" (UniqueName: \"kubernetes.io/projected/b24e8260-971a-414c-8293-228a5c1f57f3-kube-api-access-cb44n\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7\" (UID: \"b24e8260-971a-414c-8293-228a5c1f57f3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7"
Sep 07 10:18:31.075392 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.075308    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd"
Sep 07 10:18:31.158078 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.158050    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n"]
Sep 07 10:18:31.159812 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:18:31.159785    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29a398c_aa99_4486_bc21_5366b2c613fc.slice/crio-18cea13b6672b1728984bd2cddfa74223be4b18382341f997b4c628122efcea7 WatchSource:0}: Error finding container 18cea13b6672b1728984bd2cddfa74223be4b18382341f997b4c628122efcea7: Status 404 returned error can't find the container with id 18cea13b6672b1728984bd2cddfa74223be4b18382341f997b4c628122efcea7
Sep 07 10:18:31.177238 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.177212    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7"
Sep 07 10:18:31.251365 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.251030    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv"]
Sep 07 10:18:31.252756 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:18:31.252721    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853a27cb_a92e_4dd9_a24b_972c134eb3fd.slice/crio-635d34f1f99e226d3dcd4796af7c1de5de81afa5bdb73a96ebf5cc2345444e78 WatchSource:0}: Error finding container 635d34f1f99e226d3dcd4796af7c1de5de81afa5bdb73a96ebf5cc2345444e78: Status 404 returned error can't find the container with id 635d34f1f99e226d3dcd4796af7c1de5de81afa5bdb73a96ebf5cc2345444e78
Sep 07 10:18:31.346224 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.346194    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd"]
Sep 07 10:18:31.347900 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:18:31.347879    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72deda39_62a4_4010_876b_3b0fa7a73cc6.slice/crio-0e1094e1207ea4a91a043180a1d7557b52d09a6126fe03c53daf6fad5c8b9e4e WatchSource:0}: Error finding container 0e1094e1207ea4a91a043180a1d7557b52d09a6126fe03c53daf6fad5c8b9e4e: Status 404 returned error can't find the container with id 0e1094e1207ea4a91a043180a1d7557b52d09a6126fe03c53daf6fad5c8b9e4e
Sep 07 10:18:31.445920 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.445895    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7"]
Sep 07 10:18:31.470585 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:18:31.470551    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24e8260_971a_414c_8293_228a5c1f57f3.slice/crio-45ba55f191334aa1c098fb4aa4cbe49f9939d2254916d5477b6d0982c90b655e WatchSource:0}: Error finding container 45ba55f191334aa1c098fb4aa4cbe49f9939d2254916d5477b6d0982c90b655e: Status 404 returned error can't find the container with id 45ba55f191334aa1c098fb4aa4cbe49f9939d2254916d5477b6d0982c90b655e
Sep 07 10:18:31.666118 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.666036    2562 generic.go:356] "Generic (PLEG): container finished" podID="b24e8260-971a-414c-8293-228a5c1f57f3" containerID="af147e950bfbc68828b0ecdf8eb95a43bc2d1801b2a25262cb5fcf4297b3c289" exitCode=0
Sep 07 10:18:31.666519 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.666117    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7" event={"ID":"b24e8260-971a-414c-8293-228a5c1f57f3","Type":"ContainerDied","Data":"af147e950bfbc68828b0ecdf8eb95a43bc2d1801b2a25262cb5fcf4297b3c289"}
Sep 07 10:18:31.666519 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.666156    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7" event={"ID":"b24e8260-971a-414c-8293-228a5c1f57f3","Type":"ContainerStarted","Data":"45ba55f191334aa1c098fb4aa4cbe49f9939d2254916d5477b6d0982c90b655e"}
Sep 07 10:18:31.667847 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.667821    2562 generic.go:356] "Generic (PLEG): container finished" podID="853a27cb-a92e-4dd9-a24b-972c134eb3fd" containerID="9c2c135d60ef65097a734c2f629462eaad3b0bd966354f4ff2eab8c9fff98f62" exitCode=0
Sep 07 10:18:31.667934 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.667874    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv" event={"ID":"853a27cb-a92e-4dd9-a24b-972c134eb3fd","Type":"ContainerDied","Data":"9c2c135d60ef65097a734c2f629462eaad3b0bd966354f4ff2eab8c9fff98f62"}
Sep 07 10:18:31.667934 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.667917    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv" event={"ID":"853a27cb-a92e-4dd9-a24b-972c134eb3fd","Type":"ContainerStarted","Data":"635d34f1f99e226d3dcd4796af7c1de5de81afa5bdb73a96ebf5cc2345444e78"}
Sep 07 10:18:31.669639 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.669611    2562 generic.go:356] "Generic (PLEG): container finished" podID="d29a398c-aa99-4486-bc21-5366b2c613fc" containerID="0d29ceb154f66768e6e0a7cd0b114da62eb2587bd1272fa8194888eea7c50add" exitCode=0
Sep 07 10:18:31.669730 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.669667    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n" event={"ID":"d29a398c-aa99-4486-bc21-5366b2c613fc","Type":"ContainerDied","Data":"0d29ceb154f66768e6e0a7cd0b114da62eb2587bd1272fa8194888eea7c50add"}
Sep 07 10:18:31.669730 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.669722    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n" event={"ID":"d29a398c-aa99-4486-bc21-5366b2c613fc","Type":"ContainerStarted","Data":"18cea13b6672b1728984bd2cddfa74223be4b18382341f997b4c628122efcea7"}
Sep 07 10:18:31.671498 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.671481    2562 generic.go:356] "Generic (PLEG): container finished" podID="72deda39-62a4-4010-876b-3b0fa7a73cc6" containerID="ba0dd02ebacac7f3907e6380787fb45120f8222184cbf49c78613b7b0219e3e5" exitCode=0
Sep 07 10:18:31.671585 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.671507    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd" event={"ID":"72deda39-62a4-4010-876b-3b0fa7a73cc6","Type":"ContainerDied","Data":"ba0dd02ebacac7f3907e6380787fb45120f8222184cbf49c78613b7b0219e3e5"}
Sep 07 10:18:31.671585 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:31.671532    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd" event={"ID":"72deda39-62a4-4010-876b-3b0fa7a73cc6","Type":"ContainerStarted","Data":"0e1094e1207ea4a91a043180a1d7557b52d09a6126fe03c53daf6fad5c8b9e4e"}
Sep 07 10:18:32.683728 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:32.683667    2562 generic.go:356] "Generic (PLEG): container finished" podID="b24e8260-971a-414c-8293-228a5c1f57f3" containerID="ce2bc423c0361a670f1c3878fe9df6fdbdf96fefb0f180d543392367936edf55" exitCode=0
Sep 07 10:18:32.684189 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:32.683782    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7" event={"ID":"b24e8260-971a-414c-8293-228a5c1f57f3","Type":"ContainerDied","Data":"ce2bc423c0361a670f1c3878fe9df6fdbdf96fefb0f180d543392367936edf55"}
Sep 07 10:18:32.685712 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:32.685689    2562 generic.go:356] "Generic (PLEG): container finished" podID="853a27cb-a92e-4dd9-a24b-972c134eb3fd" containerID="fbecd610d338c395eed55cb7aa26422aa76e22e4fd1a7c0d5945884320be2965" exitCode=0
Sep 07 10:18:32.685858 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:32.685748    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv" event={"ID":"853a27cb-a92e-4dd9-a24b-972c134eb3fd","Type":"ContainerDied","Data":"fbecd610d338c395eed55cb7aa26422aa76e22e4fd1a7c0d5945884320be2965"}
Sep 07 10:18:32.687612 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:32.687577    2562 generic.go:356] "Generic (PLEG): container finished" podID="d29a398c-aa99-4486-bc21-5366b2c613fc" containerID="8f9c816f89615cff357a0676f30f1a6889a7d4b47ea432ca2ada9e2206446631" exitCode=0
Sep 07 10:18:32.687735 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:32.687706    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n" event={"ID":"d29a398c-aa99-4486-bc21-5366b2c613fc","Type":"ContainerDied","Data":"8f9c816f89615cff357a0676f30f1a6889a7d4b47ea432ca2ada9e2206446631"}
Sep 07 10:18:33.694807 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:33.694757    2562 generic.go:356] "Generic (PLEG): container finished" podID="b24e8260-971a-414c-8293-228a5c1f57f3" containerID="87f1f8c161857e75941356a61be9b32cc1b5f1d2181e3f8c07355ddaa18404d7" exitCode=0
Sep 07 10:18:33.695216 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:33.694849    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7" event={"ID":"b24e8260-971a-414c-8293-228a5c1f57f3","Type":"ContainerDied","Data":"87f1f8c161857e75941356a61be9b32cc1b5f1d2181e3f8c07355ddaa18404d7"}
Sep 07 10:18:33.696806 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:33.696779    2562 generic.go:356] "Generic (PLEG): container finished" podID="853a27cb-a92e-4dd9-a24b-972c134eb3fd" containerID="63d4329253e1aaa4a4de0079b639697d9449da2d4f2c50ad061b5538f2b6bfff" exitCode=0
Sep 07 10:18:33.696894 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:33.696810    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv" event={"ID":"853a27cb-a92e-4dd9-a24b-972c134eb3fd","Type":"ContainerDied","Data":"63d4329253e1aaa4a4de0079b639697d9449da2d4f2c50ad061b5538f2b6bfff"}
Sep 07 10:18:33.698872 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:33.698850    2562 generic.go:356] "Generic (PLEG): container finished" podID="d29a398c-aa99-4486-bc21-5366b2c613fc" containerID="ddcbb268250810cb76da04bc7ae2d1a252f3101d5bd473bb7d3b5cad8ae0bedb" exitCode=0
Sep 07 10:18:33.698979 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:33.698926    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n" event={"ID":"d29a398c-aa99-4486-bc21-5366b2c613fc","Type":"ContainerDied","Data":"ddcbb268250810cb76da04bc7ae2d1a252f3101d5bd473bb7d3b5cad8ae0bedb"}
Sep 07 10:18:33.700736 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:33.700715    2562 generic.go:356] "Generic (PLEG): container finished" podID="72deda39-62a4-4010-876b-3b0fa7a73cc6" containerID="82b71ceefc86f502051152707c151400de21b1dc89772f1efc1abd0b0bd60d02" exitCode=0
Sep 07 10:18:33.700838 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:33.700779    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd" event={"ID":"72deda39-62a4-4010-876b-3b0fa7a73cc6","Type":"ContainerDied","Data":"82b71ceefc86f502051152707c151400de21b1dc89772f1efc1abd0b0bd60d02"}
Sep 07 10:18:34.707726 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.707689    2562 generic.go:356] "Generic (PLEG): container finished" podID="72deda39-62a4-4010-876b-3b0fa7a73cc6" containerID="81be314bf12a88b38c82c6ffb5027e2cc8b4e44c7a33fa66e5e16d57037d5abf" exitCode=0
Sep 07 10:18:34.708152 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.707797    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd" event={"ID":"72deda39-62a4-4010-876b-3b0fa7a73cc6","Type":"ContainerDied","Data":"81be314bf12a88b38c82c6ffb5027e2cc8b4e44c7a33fa66e5e16d57037d5abf"}
Sep 07 10:18:34.874345 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.874311    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n"
Sep 07 10:18:34.877003 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.876983    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7"
Sep 07 10:18:34.904590 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.904563    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv"
Sep 07 10:18:34.955017 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.954988    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b24e8260-971a-414c-8293-228a5c1f57f3-util\") pod \"b24e8260-971a-414c-8293-228a5c1f57f3\" (UID: \"b24e8260-971a-414c-8293-228a5c1f57f3\") "
Sep 07 10:18:34.955211 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.955037    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/853a27cb-a92e-4dd9-a24b-972c134eb3fd-util\") pod \"853a27cb-a92e-4dd9-a24b-972c134eb3fd\" (UID: \"853a27cb-a92e-4dd9-a24b-972c134eb3fd\") "
Sep 07 10:18:34.955211 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.955069    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vpdm\" (UniqueName: \"kubernetes.io/projected/d29a398c-aa99-4486-bc21-5366b2c613fc-kube-api-access-5vpdm\") pod \"d29a398c-aa99-4486-bc21-5366b2c613fc\" (UID: \"d29a398c-aa99-4486-bc21-5366b2c613fc\") "
Sep 07 10:18:34.955211 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.955103    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d29a398c-aa99-4486-bc21-5366b2c613fc-util\") pod \"d29a398c-aa99-4486-bc21-5366b2c613fc\" (UID: \"d29a398c-aa99-4486-bc21-5366b2c613fc\") "
Sep 07 10:18:34.955211 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.955139    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d29a398c-aa99-4486-bc21-5366b2c613fc-bundle\") pod \"d29a398c-aa99-4486-bc21-5366b2c613fc\" (UID: \"d29a398c-aa99-4486-bc21-5366b2c613fc\") "
Sep 07 10:18:34.955211 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.955173    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b24e8260-971a-414c-8293-228a5c1f57f3-bundle\") pod \"b24e8260-971a-414c-8293-228a5c1f57f3\" (UID: \"b24e8260-971a-414c-8293-228a5c1f57f3\") "
Sep 07 10:18:34.955211 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.955193    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb44n\" (UniqueName: \"kubernetes.io/projected/b24e8260-971a-414c-8293-228a5c1f57f3-kube-api-access-cb44n\") pod \"b24e8260-971a-414c-8293-228a5c1f57f3\" (UID: \"b24e8260-971a-414c-8293-228a5c1f57f3\") "
Sep 07 10:18:34.955486 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.955260    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlrjg\" (UniqueName: \"kubernetes.io/projected/853a27cb-a92e-4dd9-a24b-972c134eb3fd-kube-api-access-dlrjg\") pod \"853a27cb-a92e-4dd9-a24b-972c134eb3fd\" (UID: \"853a27cb-a92e-4dd9-a24b-972c134eb3fd\") "
Sep 07 10:18:34.955486 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.955285    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/853a27cb-a92e-4dd9-a24b-972c134eb3fd-bundle\") pod \"853a27cb-a92e-4dd9-a24b-972c134eb3fd\" (UID: \"853a27cb-a92e-4dd9-a24b-972c134eb3fd\") "
Sep 07 10:18:34.956053 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.955976    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24e8260-971a-414c-8293-228a5c1f57f3-bundle" (OuterVolumeSpecName: "bundle") pod "b24e8260-971a-414c-8293-228a5c1f57f3" (UID: "b24e8260-971a-414c-8293-228a5c1f57f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:18:34.956053 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.955991    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d29a398c-aa99-4486-bc21-5366b2c613fc-bundle" (OuterVolumeSpecName: "bundle") pod "d29a398c-aa99-4486-bc21-5366b2c613fc" (UID: "d29a398c-aa99-4486-bc21-5366b2c613fc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:18:34.956425 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.956377    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/853a27cb-a92e-4dd9-a24b-972c134eb3fd-bundle" (OuterVolumeSpecName: "bundle") pod "853a27cb-a92e-4dd9-a24b-972c134eb3fd" (UID: "853a27cb-a92e-4dd9-a24b-972c134eb3fd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:18:34.957575 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.957531    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24e8260-971a-414c-8293-228a5c1f57f3-kube-api-access-cb44n" (OuterVolumeSpecName: "kube-api-access-cb44n") pod "b24e8260-971a-414c-8293-228a5c1f57f3" (UID: "b24e8260-971a-414c-8293-228a5c1f57f3"). InnerVolumeSpecName "kube-api-access-cb44n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:18:34.957810 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.957733    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29a398c-aa99-4486-bc21-5366b2c613fc-kube-api-access-5vpdm" (OuterVolumeSpecName: "kube-api-access-5vpdm") pod "d29a398c-aa99-4486-bc21-5366b2c613fc" (UID: "d29a398c-aa99-4486-bc21-5366b2c613fc"). InnerVolumeSpecName "kube-api-access-5vpdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:18:34.958178 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.958156    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/853a27cb-a92e-4dd9-a24b-972c134eb3fd-kube-api-access-dlrjg" (OuterVolumeSpecName: "kube-api-access-dlrjg") pod "853a27cb-a92e-4dd9-a24b-972c134eb3fd" (UID: "853a27cb-a92e-4dd9-a24b-972c134eb3fd"). InnerVolumeSpecName "kube-api-access-dlrjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:18:34.961476 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.961456    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/853a27cb-a92e-4dd9-a24b-972c134eb3fd-util" (OuterVolumeSpecName: "util") pod "853a27cb-a92e-4dd9-a24b-972c134eb3fd" (UID: "853a27cb-a92e-4dd9-a24b-972c134eb3fd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:18:34.964256 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.964209    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24e8260-971a-414c-8293-228a5c1f57f3-util" (OuterVolumeSpecName: "util") pod "b24e8260-971a-414c-8293-228a5c1f57f3" (UID: "b24e8260-971a-414c-8293-228a5c1f57f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:18:34.964329 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:34.964238    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d29a398c-aa99-4486-bc21-5366b2c613fc-util" (OuterVolumeSpecName: "util") pod "d29a398c-aa99-4486-bc21-5366b2c613fc" (UID: "d29a398c-aa99-4486-bc21-5366b2c613fc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:18:35.056781 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.056721    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb44n\" (UniqueName: \"kubernetes.io/projected/b24e8260-971a-414c-8293-228a5c1f57f3-kube-api-access-cb44n\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:18:35.056781 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.056754    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlrjg\" (UniqueName: \"kubernetes.io/projected/853a27cb-a92e-4dd9-a24b-972c134eb3fd-kube-api-access-dlrjg\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:18:35.056781 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.056786    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/853a27cb-a92e-4dd9-a24b-972c134eb3fd-bundle\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:18:35.056781 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.056796    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b24e8260-971a-414c-8293-228a5c1f57f3-util\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:18:35.057043 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.056803    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/853a27cb-a92e-4dd9-a24b-972c134eb3fd-util\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:18:35.057043 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.056811    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vpdm\" (UniqueName: \"kubernetes.io/projected/d29a398c-aa99-4486-bc21-5366b2c613fc-kube-api-access-5vpdm\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:18:35.057043 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.056819    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d29a398c-aa99-4486-bc21-5366b2c613fc-util\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:18:35.057043 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.056826    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d29a398c-aa99-4486-bc21-5366b2c613fc-bundle\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:18:35.057043 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.056833    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b24e8260-971a-414c-8293-228a5c1f57f3-bundle\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:18:35.714816 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.714752    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7" event={"ID":"b24e8260-971a-414c-8293-228a5c1f57f3","Type":"ContainerDied","Data":"45ba55f191334aa1c098fb4aa4cbe49f9939d2254916d5477b6d0982c90b655e"}
Sep 07 10:18:35.714816 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.714797    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08vvn7"
Sep 07 10:18:35.715278 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.714863    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45ba55f191334aa1c098fb4aa4cbe49f9939d2254916d5477b6d0982c90b655e"
Sep 07 10:18:35.716636 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.716613    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv"
Sep 07 10:18:35.716800 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.716612    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053snkkv" event={"ID":"853a27cb-a92e-4dd9-a24b-972c134eb3fd","Type":"ContainerDied","Data":"635d34f1f99e226d3dcd4796af7c1de5de81afa5bdb73a96ebf5cc2345444e78"}
Sep 07 10:18:35.716800 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.716741    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="635d34f1f99e226d3dcd4796af7c1de5de81afa5bdb73a96ebf5cc2345444e78"
Sep 07 10:18:35.718617 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.718597    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n"
Sep 07 10:18:35.718744 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.718614    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqd85n" event={"ID":"d29a398c-aa99-4486-bc21-5366b2c613fc","Type":"ContainerDied","Data":"18cea13b6672b1728984bd2cddfa74223be4b18382341f997b4c628122efcea7"}
Sep 07 10:18:35.718744 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.718648    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18cea13b6672b1728984bd2cddfa74223be4b18382341f997b4c628122efcea7"
Sep 07 10:18:35.837201 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.837172    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd"
Sep 07 10:18:35.964203 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.964174    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn5qd\" (UniqueName: \"kubernetes.io/projected/72deda39-62a4-4010-876b-3b0fa7a73cc6-kube-api-access-pn5qd\") pod \"72deda39-62a4-4010-876b-3b0fa7a73cc6\" (UID: \"72deda39-62a4-4010-876b-3b0fa7a73cc6\") "
Sep 07 10:18:35.964379 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.964254    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72deda39-62a4-4010-876b-3b0fa7a73cc6-util\") pod \"72deda39-62a4-4010-876b-3b0fa7a73cc6\" (UID: \"72deda39-62a4-4010-876b-3b0fa7a73cc6\") "
Sep 07 10:18:35.964379 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.964312    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72deda39-62a4-4010-876b-3b0fa7a73cc6-bundle\") pod \"72deda39-62a4-4010-876b-3b0fa7a73cc6\" (UID: \"72deda39-62a4-4010-876b-3b0fa7a73cc6\") "
Sep 07 10:18:35.964896 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.964839    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72deda39-62a4-4010-876b-3b0fa7a73cc6-bundle" (OuterVolumeSpecName: "bundle") pod "72deda39-62a4-4010-876b-3b0fa7a73cc6" (UID: "72deda39-62a4-4010-876b-3b0fa7a73cc6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:18:35.966261 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.966239    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72deda39-62a4-4010-876b-3b0fa7a73cc6-kube-api-access-pn5qd" (OuterVolumeSpecName: "kube-api-access-pn5qd") pod "72deda39-62a4-4010-876b-3b0fa7a73cc6" (UID: "72deda39-62a4-4010-876b-3b0fa7a73cc6"). InnerVolumeSpecName "kube-api-access-pn5qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:18:35.969714 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:35.969688    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72deda39-62a4-4010-876b-3b0fa7a73cc6-util" (OuterVolumeSpecName: "util") pod "72deda39-62a4-4010-876b-3b0fa7a73cc6" (UID: "72deda39-62a4-4010-876b-3b0fa7a73cc6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:18:36.065893 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:36.065840    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn5qd\" (UniqueName: \"kubernetes.io/projected/72deda39-62a4-4010-876b-3b0fa7a73cc6-kube-api-access-pn5qd\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:18:36.065893 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:36.065882    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72deda39-62a4-4010-876b-3b0fa7a73cc6-util\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:18:36.065893 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:36.065893    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72deda39-62a4-4010-876b-3b0fa7a73cc6-bundle\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:18:36.725042 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:36.725016    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd"
Sep 07 10:18:36.725414 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:36.725010    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbtrwd" event={"ID":"72deda39-62a4-4010-876b-3b0fa7a73cc6","Type":"ContainerDied","Data":"0e1094e1207ea4a91a043180a1d7557b52d09a6126fe03c53daf6fad5c8b9e4e"}
Sep 07 10:18:36.725414 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:36.725154    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e1094e1207ea4a91a043180a1d7557b52d09a6126fe03c53daf6fad5c8b9e4e"
Sep 07 10:18:42.387343 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387307    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"]
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387584    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24e8260-971a-414c-8293-228a5c1f57f3" containerName="extract"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387594    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24e8260-971a-414c-8293-228a5c1f57f3" containerName="extract"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387607    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24e8260-971a-414c-8293-228a5c1f57f3" containerName="util"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387612    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24e8260-971a-414c-8293-228a5c1f57f3" containerName="util"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387618    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="853a27cb-a92e-4dd9-a24b-972c134eb3fd" containerName="pull"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387624    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="853a27cb-a92e-4dd9-a24b-972c134eb3fd" containerName="pull"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387630    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72deda39-62a4-4010-876b-3b0fa7a73cc6" containerName="pull"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387634    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="72deda39-62a4-4010-876b-3b0fa7a73cc6" containerName="pull"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387649    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72deda39-62a4-4010-876b-3b0fa7a73cc6" containerName="util"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387655    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="72deda39-62a4-4010-876b-3b0fa7a73cc6" containerName="util"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387661    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="853a27cb-a92e-4dd9-a24b-972c134eb3fd" containerName="extract"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387665    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="853a27cb-a92e-4dd9-a24b-972c134eb3fd" containerName="extract"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387672    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="853a27cb-a92e-4dd9-a24b-972c134eb3fd" containerName="util"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387696    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="853a27cb-a92e-4dd9-a24b-972c134eb3fd" containerName="util"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387702    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29a398c-aa99-4486-bc21-5366b2c613fc" containerName="extract"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387706    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29a398c-aa99-4486-bc21-5366b2c613fc" containerName="extract"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387712    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72deda39-62a4-4010-876b-3b0fa7a73cc6" containerName="extract"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387719    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="72deda39-62a4-4010-876b-3b0fa7a73cc6" containerName="extract"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387728    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24e8260-971a-414c-8293-228a5c1f57f3" containerName="pull"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387734    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24e8260-971a-414c-8293-228a5c1f57f3" containerName="pull"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387740    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29a398c-aa99-4486-bc21-5366b2c613fc" containerName="pull"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387745    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29a398c-aa99-4486-bc21-5366b2c613fc" containerName="pull"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387751    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29a398c-aa99-4486-bc21-5366b2c613fc" containerName="util"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387756    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29a398c-aa99-4486-bc21-5366b2c613fc" containerName="util"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387818    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29a398c-aa99-4486-bc21-5366b2c613fc" containerName="extract"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387825    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="853a27cb-a92e-4dd9-a24b-972c134eb3fd" containerName="extract"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387835    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b24e8260-971a-414c-8293-228a5c1f57f3" containerName="extract"
Sep 07 10:18:42.387853 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.387840    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="72deda39-62a4-4010-876b-3b0fa7a73cc6" containerName="extract"
Sep 07 10:18:42.391866 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.391846    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:18:42.394803 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.394728    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 07 10:18:42.395811 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.395793    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fsjn6\""
Sep 07 10:18:42.395941 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.395845    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 07 10:18:42.404563 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.404544    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"]
Sep 07 10:18:42.519141 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.519101    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmk27\" (UniqueName: \"kubernetes.io/projected/37a810e7-bc2a-4862-89de-d520d0dd5186-kube-api-access-rmk27\") pod \"limitador-operator-controller-manager-6c549b6d85-gzlrq\" (UID: \"37a810e7-bc2a-4862-89de-d520d0dd5186\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:18:42.619866 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.619821    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmk27\" (UniqueName: \"kubernetes.io/projected/37a810e7-bc2a-4862-89de-d520d0dd5186-kube-api-access-rmk27\") pod \"limitador-operator-controller-manager-6c549b6d85-gzlrq\" (UID: \"37a810e7-bc2a-4862-89de-d520d0dd5186\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:18:42.628832 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.628787    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmk27\" (UniqueName: \"kubernetes.io/projected/37a810e7-bc2a-4862-89de-d520d0dd5186-kube-api-access-rmk27\") pod \"limitador-operator-controller-manager-6c549b6d85-gzlrq\" (UID: \"37a810e7-bc2a-4862-89de-d520d0dd5186\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:18:42.701393 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.701365    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:18:42.976945 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:42.976916    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"]
Sep 07 10:18:42.978311 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:18:42.978274    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a810e7_bc2a_4862_89de_d520d0dd5186.slice/crio-55d8285d591f2d5e3ef59ca63d6636bff3e4cfcc8e79364a7a4c17b3e62ca632 WatchSource:0}: Error finding container 55d8285d591f2d5e3ef59ca63d6636bff3e4cfcc8e79364a7a4c17b3e62ca632: Status 404 returned error can't find the container with id 55d8285d591f2d5e3ef59ca63d6636bff3e4cfcc8e79364a7a4c17b3e62ca632
Sep 07 10:18:43.760010 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:43.759973    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq" event={"ID":"37a810e7-bc2a-4862-89de-d520d0dd5186","Type":"ContainerStarted","Data":"55d8285d591f2d5e3ef59ca63d6636bff3e4cfcc8e79364a7a4c17b3e62ca632"}
Sep 07 10:18:44.371551 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:44.371512    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-5pdhl"]
Sep 07 10:18:44.374957 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:44.374934    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5pdhl"
Sep 07 10:18:44.377678 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:44.377657    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 07 10:18:44.377822 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:44.377789    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-q9cdb\""
Sep 07 10:18:44.383025 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:44.382984    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-5pdhl"]
Sep 07 10:18:44.433487 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:44.433448    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwxvl\" (UniqueName: \"kubernetes.io/projected/7412841d-c20e-464a-b4c9-3fb5656954b1-kube-api-access-pwxvl\") pod \"dns-operator-controller-manager-5985564776-5pdhl\" (UID: \"7412841d-c20e-464a-b4c9-3fb5656954b1\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-5pdhl"
Sep 07 10:18:44.534441 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:44.534400    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwxvl\" (UniqueName: \"kubernetes.io/projected/7412841d-c20e-464a-b4c9-3fb5656954b1-kube-api-access-pwxvl\") pod \"dns-operator-controller-manager-5985564776-5pdhl\" (UID: \"7412841d-c20e-464a-b4c9-3fb5656954b1\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-5pdhl"
Sep 07 10:18:44.545695 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:44.545660    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwxvl\" (UniqueName: \"kubernetes.io/projected/7412841d-c20e-464a-b4c9-3fb5656954b1-kube-api-access-pwxvl\") pod \"dns-operator-controller-manager-5985564776-5pdhl\" (UID: \"7412841d-c20e-464a-b4c9-3fb5656954b1\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-5pdhl"
Sep 07 10:18:44.689490 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:44.689403    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5pdhl"
Sep 07 10:18:45.773303 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:45.773223    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq" event={"ID":"37a810e7-bc2a-4862-89de-d520d0dd5186","Type":"ContainerStarted","Data":"29be610bcacc51c41c8ed42a16c240e17b5e96af96311ff52b490fcd6cfd178b"}
Sep 07 10:18:45.773303 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:45.773295    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:18:45.792613 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:45.792588    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-5pdhl"]
Sep 07 10:18:45.794581 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:18:45.794553    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7412841d_c20e_464a_b4c9_3fb5656954b1.slice/crio-3d357c49bc052efade4829b035e6cc3bbd0c947f7e02aa086cb2c63cde27c3b0 WatchSource:0}: Error finding container 3d357c49bc052efade4829b035e6cc3bbd0c947f7e02aa086cb2c63cde27c3b0: Status 404 returned error can't find the container with id 3d357c49bc052efade4829b035e6cc3bbd0c947f7e02aa086cb2c63cde27c3b0
Sep 07 10:18:45.796515 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:45.796344    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq" podStartSLOduration=1.194286198 podStartE2EDuration="3.796325766s" totalImagesPullingTime="2.602039568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:18:45.795806476 +0000 UTC m=+428.293496404" watchObservedRunningTime="2026-09-07 10:18:45.796325766 +0000 UTC m=+428.294015676"
Sep 07 10:18:46.780445 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:46.780403    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5pdhl" event={"ID":"7412841d-c20e-464a-b4c9-3fb5656954b1","Type":"ContainerStarted","Data":"3d357c49bc052efade4829b035e6cc3bbd0c947f7e02aa086cb2c63cde27c3b0"}
Sep 07 10:18:46.983640 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:46.983600    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-qck72"]
Sep 07 10:18:46.987146 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:46.987123    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-qck72"
Sep 07 10:18:46.990096 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:46.990074    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-z54gm\""
Sep 07 10:18:47.001236 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:47.001213    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-qck72"]
Sep 07 10:18:47.056545 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:47.056441    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ghq2\" (UniqueName: \"kubernetes.io/projected/05332adc-2bbd-4592-8bb6-5c22a0183df6-kube-api-access-7ghq2\") pod \"authorino-operator-68d88948d8-qck72\" (UID: \"05332adc-2bbd-4592-8bb6-5c22a0183df6\") " pod="kuadrant-system/authorino-operator-68d88948d8-qck72"
Sep 07 10:18:47.157972 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:47.157926    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ghq2\" (UniqueName: \"kubernetes.io/projected/05332adc-2bbd-4592-8bb6-5c22a0183df6-kube-api-access-7ghq2\") pod \"authorino-operator-68d88948d8-qck72\" (UID: \"05332adc-2bbd-4592-8bb6-5c22a0183df6\") " pod="kuadrant-system/authorino-operator-68d88948d8-qck72"
Sep 07 10:18:47.173788 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:47.173746    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ghq2\" (UniqueName: \"kubernetes.io/projected/05332adc-2bbd-4592-8bb6-5c22a0183df6-kube-api-access-7ghq2\") pod \"authorino-operator-68d88948d8-qck72\" (UID: \"05332adc-2bbd-4592-8bb6-5c22a0183df6\") " pod="kuadrant-system/authorino-operator-68d88948d8-qck72"
Sep 07 10:18:47.299412 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:47.299385    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-qck72"
Sep 07 10:18:47.587717 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:47.587687    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-qck72"]
Sep 07 10:18:47.591290 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:18:47.591244    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05332adc_2bbd_4592_8bb6_5c22a0183df6.slice/crio-202b49f21df64a591b32072f12d82b6a18d65a154547b6c06e849c2982769504 WatchSource:0}: Error finding container 202b49f21df64a591b32072f12d82b6a18d65a154547b6c06e849c2982769504: Status 404 returned error can't find the container with id 202b49f21df64a591b32072f12d82b6a18d65a154547b6c06e849c2982769504
Sep 07 10:18:47.787164 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:47.787127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5pdhl" event={"ID":"7412841d-c20e-464a-b4c9-3fb5656954b1","Type":"ContainerStarted","Data":"3b9376799fb3b4e049f95c314238396c0d12dd862028bff384475c44b9534715"}
Sep 07 10:18:47.787603 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:47.787292    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5pdhl"
Sep 07 10:18:47.788515 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:47.788495    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-qck72" event={"ID":"05332adc-2bbd-4592-8bb6-5c22a0183df6","Type":"ContainerStarted","Data":"202b49f21df64a591b32072f12d82b6a18d65a154547b6c06e849c2982769504"}
Sep 07 10:18:47.808159 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:47.808111    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5pdhl" podStartSLOduration=2.3377914349999998 podStartE2EDuration="3.808096206s" totalImagesPullingTime="1.470304771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:18:47.807271429 +0000 UTC m=+430.304961338" watchObservedRunningTime="2026-09-07 10:18:47.808096206 +0000 UTC m=+430.305786114"
Sep 07 10:18:49.800775 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:49.800731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-qck72" event={"ID":"05332adc-2bbd-4592-8bb6-5c22a0183df6","Type":"ContainerStarted","Data":"49863f9bd952bbd87770458c798bd87b2b5c15b9689b231a3e7e2a0c8c10ddda"}
Sep 07 10:18:49.801155 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:49.800880    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-qck72"
Sep 07 10:18:49.820216 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:49.820162    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-qck72" podStartSLOduration=2.276193985 podStartE2EDuration="3.820146062s" totalImagesPullingTime="1.543952077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:18:49.817752683 +0000 UTC m=+432.315442610" watchObservedRunningTime="2026-09-07 10:18:49.820146062 +0000 UTC m=+432.317835970"
Sep 07 10:18:51.530580 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:51.530546    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"]
Sep 07 10:18:51.533912 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:51.533896    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:18:51.536818 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:51.536749    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-52674\""
Sep 07 10:18:51.547477 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:51.547456    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"]
Sep 07 10:18:51.592349 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:51.592312    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/accb6f78-2778-4d06-999c-07552b9d49db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-cz785\" (UID: \"accb6f78-2778-4d06-999c-07552b9d49db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:18:51.592525 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:51.592372    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97rnh\" (UniqueName: \"kubernetes.io/projected/accb6f78-2778-4d06-999c-07552b9d49db-kube-api-access-97rnh\") pod \"kuadrant-operator-controller-manager-58fc56757c-cz785\" (UID: \"accb6f78-2778-4d06-999c-07552b9d49db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:18:51.693321 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:51.693282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/accb6f78-2778-4d06-999c-07552b9d49db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-cz785\" (UID: \"accb6f78-2778-4d06-999c-07552b9d49db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:18:51.693513 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:51.693331    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97rnh\" (UniqueName: \"kubernetes.io/projected/accb6f78-2778-4d06-999c-07552b9d49db-kube-api-access-97rnh\") pod \"kuadrant-operator-controller-manager-58fc56757c-cz785\" (UID: \"accb6f78-2778-4d06-999c-07552b9d49db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:18:51.693732 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:51.693708    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/accb6f78-2778-4d06-999c-07552b9d49db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-cz785\" (UID: \"accb6f78-2778-4d06-999c-07552b9d49db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:18:51.704138 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:51.704114    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97rnh\" (UniqueName: \"kubernetes.io/projected/accb6f78-2778-4d06-999c-07552b9d49db-kube-api-access-97rnh\") pod \"kuadrant-operator-controller-manager-58fc56757c-cz785\" (UID: \"accb6f78-2778-4d06-999c-07552b9d49db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:18:51.843531 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:51.843445    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:18:52.118455 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:52.118378    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"]
Sep 07 10:18:52.123632 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:18:52.123601    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaccb6f78_2778_4d06_999c_07552b9d49db.slice/crio-efc527a5bd1339b5850788b8c5d53f65affab4918266087a5c3abaa3d53b2990 WatchSource:0}: Error finding container efc527a5bd1339b5850788b8c5d53f65affab4918266087a5c3abaa3d53b2990: Status 404 returned error can't find the container with id efc527a5bd1339b5850788b8c5d53f65affab4918266087a5c3abaa3d53b2990
Sep 07 10:18:52.818329 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:52.818289    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785" event={"ID":"accb6f78-2778-4d06-999c-07552b9d49db","Type":"ContainerStarted","Data":"efc527a5bd1339b5850788b8c5d53f65affab4918266087a5c3abaa3d53b2990"}
Sep 07 10:18:56.783396 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:56.783365    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:18:56.843105 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:56.843065    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785" event={"ID":"accb6f78-2778-4d06-999c-07552b9d49db","Type":"ContainerStarted","Data":"8c529a289f5a3274d11b57d88032ce8d4d2d37067267fb6fb68c09ec65a00eb6"}
Sep 07 10:18:56.843267 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:56.843230    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:18:56.862950 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:56.862895    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785" podStartSLOduration=2.022089283 podStartE2EDuration="5.862880355s" totalImagesPullingTime="3.840791072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:18:56.861508489 +0000 UTC m=+439.359198398" watchObservedRunningTime="2026-09-07 10:18:56.862880355 +0000 UTC m=+439.360570264"
Sep 07 10:18:58.797031 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:18:58.796995    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-5pdhl"
Sep 07 10:19:00.808418 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:00.808341    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-qck72"
Sep 07 10:19:07.849587 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:07.849548    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:19:08.808222 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:08.808187    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"]
Sep 07 10:19:08.812754 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:08.812734    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"
Sep 07 10:19:08.828826 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:08.828792    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"]
Sep 07 10:19:08.941977 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:08.941940    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4b15387-99f9-4535-acce-afe2b8d8eb9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-rs5x2\" (UID: \"e4b15387-99f9-4535-acce-afe2b8d8eb9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"
Sep 07 10:19:08.942350 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:08.941985    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf7kx\" (UniqueName: \"kubernetes.io/projected/e4b15387-99f9-4535-acce-afe2b8d8eb9f-kube-api-access-wf7kx\") pod \"kuadrant-operator-controller-manager-96db98c47-rs5x2\" (UID: \"e4b15387-99f9-4535-acce-afe2b8d8eb9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"
Sep 07 10:19:09.043452 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:09.043395    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4b15387-99f9-4535-acce-afe2b8d8eb9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-rs5x2\" (UID: \"e4b15387-99f9-4535-acce-afe2b8d8eb9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"
Sep 07 10:19:09.043629 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:09.043476    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf7kx\" (UniqueName: \"kubernetes.io/projected/e4b15387-99f9-4535-acce-afe2b8d8eb9f-kube-api-access-wf7kx\") pod \"kuadrant-operator-controller-manager-96db98c47-rs5x2\" (UID: \"e4b15387-99f9-4535-acce-afe2b8d8eb9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"
Sep 07 10:19:09.043917 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:09.043894    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4b15387-99f9-4535-acce-afe2b8d8eb9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-rs5x2\" (UID: \"e4b15387-99f9-4535-acce-afe2b8d8eb9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"
Sep 07 10:19:09.057515 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:09.057490    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf7kx\" (UniqueName: \"kubernetes.io/projected/e4b15387-99f9-4535-acce-afe2b8d8eb9f-kube-api-access-wf7kx\") pod \"kuadrant-operator-controller-manager-96db98c47-rs5x2\" (UID: \"e4b15387-99f9-4535-acce-afe2b8d8eb9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"
Sep 07 10:19:09.123566 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:09.123477    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"
Sep 07 10:19:09.404025 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:09.403996    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"]
Sep 07 10:19:09.405600 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:19:09.405567    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b15387_99f9_4535_acce_afe2b8d8eb9f.slice/crio-6de09e4ec82127c67ae3fe206fe5411cbd8f65f8bd32aa763a94a8ce46c0d886 WatchSource:0}: Error finding container 6de09e4ec82127c67ae3fe206fe5411cbd8f65f8bd32aa763a94a8ce46c0d886: Status 404 returned error can't find the container with id 6de09e4ec82127c67ae3fe206fe5411cbd8f65f8bd32aa763a94a8ce46c0d886
Sep 07 10:19:09.910118 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:09.910085    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2" event={"ID":"e4b15387-99f9-4535-acce-afe2b8d8eb9f","Type":"ContainerStarted","Data":"78e4280fe9eac585f2e72a5d1796e64d3e93c087c2e004607fcc793c0d52886b"}
Sep 07 10:19:09.910306 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:09.910138    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2" event={"ID":"e4b15387-99f9-4535-acce-afe2b8d8eb9f","Type":"ContainerStarted","Data":"6de09e4ec82127c67ae3fe206fe5411cbd8f65f8bd32aa763a94a8ce46c0d886"}
Sep 07 10:19:09.910306 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:09.910193    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"
Sep 07 10:19:09.931867 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:09.931789    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2" podStartSLOduration=1.931754061 podStartE2EDuration="1.931754061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:19:09.930453768 +0000 UTC m=+452.428143688" watchObservedRunningTime="2026-09-07 10:19:09.931754061 +0000 UTC m=+452.429443971"
Sep 07 10:19:09.976714 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:09.976677    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"]
Sep 07 10:19:09.977108 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:09.976885    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785" podUID="accb6f78-2778-4d06-999c-07552b9d49db" containerName="manager" containerID="cri-o://8c529a289f5a3274d11b57d88032ce8d4d2d37067267fb6fb68c09ec65a00eb6" gracePeriod=2
Sep 07 10:19:09.982224 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:09.982195    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"]
Sep 07 10:19:09.991609 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:09.991575    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"]
Sep 07 10:19:10.002597 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.002566    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"]
Sep 07 10:19:10.004598 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.004570    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"]
Sep 07 10:19:10.004922 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.004907    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="accb6f78-2778-4d06-999c-07552b9d49db" containerName="manager"
Sep 07 10:19:10.004922 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.004922    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="accb6f78-2778-4d06-999c-07552b9d49db" containerName="manager"
Sep 07 10:19:10.005033 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.004937    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b15387-99f9-4535-acce-afe2b8d8eb9f" containerName="manager"
Sep 07 10:19:10.005033 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.004943    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b15387-99f9-4535-acce-afe2b8d8eb9f" containerName="manager"
Sep 07 10:19:10.005033 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.004990    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b15387-99f9-4535-acce-afe2b8d8eb9f" containerName="manager"
Sep 07 10:19:10.005033 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.005013    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="accb6f78-2778-4d06-999c-07552b9d49db" containerName="manager"
Sep 07 10:19:10.008498 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.008459    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"
Sep 07 10:19:10.010777 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.010725    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cz785\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="accb6f78-2778-4d06-999c-07552b9d49db" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:19:10.013891 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.013869    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"]
Sep 07 10:19:10.014065 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.014042    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq" podUID="37a810e7-bc2a-4862-89de-d520d0dd5186" containerName="manager" containerID="cri-o://29be610bcacc51c41c8ed42a16c240e17b5e96af96311ff52b490fcd6cfd178b" gracePeriod=2
Sep 07 10:19:10.018215 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.018191    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"]
Sep 07 10:19:10.021356 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.021336    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"
Sep 07 10:19:10.025064 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.025038    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"]
Sep 07 10:19:10.026808 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.026780    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"]
Sep 07 10:19:10.041930 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.041889    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-gzlrq\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="37a810e7-bc2a-4862-89de-d520d0dd5186" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:19:10.044492 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.044466    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"]
Sep 07 10:19:10.048220 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.048193    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wvfb9"]
Sep 07 10:19:10.048537 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.048523    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37a810e7-bc2a-4862-89de-d520d0dd5186" containerName="manager"
Sep 07 10:19:10.048537 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.048539    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="37a810e7-bc2a-4862-89de-d520d0dd5186" containerName="manager"
Sep 07 10:19:10.048685 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.048639    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="37a810e7-bc2a-4862-89de-d520d0dd5186" containerName="manager"
Sep 07 10:19:10.051878 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.051856    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wvfb9"
Sep 07 10:19:10.063490 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.063460    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wvfb9"]
Sep 07 10:19:10.066716 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.066656    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cz785\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="accb6f78-2778-4d06-999c-07552b9d49db" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:19:10.068876 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.068850    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cz785\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="accb6f78-2778-4d06-999c-07552b9d49db" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:19:10.092024 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.091985    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-gzlrq\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="37a810e7-bc2a-4862-89de-d520d0dd5186" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:19:10.153333 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.153297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-26mrg\" (UID: \"dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"
Sep 07 10:19:10.153523 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.153426    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh6d5\" (UniqueName: \"kubernetes.io/projected/3f3ea3d6-640f-41b7-ad46-ed333f74b4af-kube-api-access-lh6d5\") pod \"kuadrant-operator-controller-manager-96db98c47-5p424\" (UID: \"3f3ea3d6-640f-41b7-ad46-ed333f74b4af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"
Sep 07 10:19:10.153523 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.153489    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f3ea3d6-640f-41b7-ad46-ed333f74b4af-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-5p424\" (UID: \"3f3ea3d6-640f-41b7-ad46-ed333f74b4af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"
Sep 07 10:19:10.153649 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.153529    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzfnx\" (UniqueName: \"kubernetes.io/projected/f4b88551-5dee-416b-bbf7-a514dc9ee2f1-kube-api-access-gzfnx\") pod \"limitador-operator-controller-manager-6c549b6d85-wvfb9\" (UID: \"f4b88551-5dee-416b-bbf7-a514dc9ee2f1\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wvfb9"
Sep 07 10:19:10.153649 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.153560    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sdst\" (UniqueName: \"kubernetes.io/projected/dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e-kube-api-access-8sdst\") pod \"kuadrant-operator-controller-manager-58fc56757c-26mrg\" (UID: \"dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"
Sep 07 10:19:10.254927 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.254895    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh6d5\" (UniqueName: \"kubernetes.io/projected/3f3ea3d6-640f-41b7-ad46-ed333f74b4af-kube-api-access-lh6d5\") pod \"kuadrant-operator-controller-manager-96db98c47-5p424\" (UID: \"3f3ea3d6-640f-41b7-ad46-ed333f74b4af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"
Sep 07 10:19:10.255135 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.254943    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f3ea3d6-640f-41b7-ad46-ed333f74b4af-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-5p424\" (UID: \"3f3ea3d6-640f-41b7-ad46-ed333f74b4af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"
Sep 07 10:19:10.255135 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.254969    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzfnx\" (UniqueName: \"kubernetes.io/projected/f4b88551-5dee-416b-bbf7-a514dc9ee2f1-kube-api-access-gzfnx\") pod \"limitador-operator-controller-manager-6c549b6d85-wvfb9\" (UID: \"f4b88551-5dee-416b-bbf7-a514dc9ee2f1\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wvfb9"
Sep 07 10:19:10.255135 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.254987    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sdst\" (UniqueName: \"kubernetes.io/projected/dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e-kube-api-access-8sdst\") pod \"kuadrant-operator-controller-manager-58fc56757c-26mrg\" (UID: \"dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"
Sep 07 10:19:10.255135 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.255015    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-26mrg\" (UID: \"dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"
Sep 07 10:19:10.255485 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.255388    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-26mrg\" (UID: \"dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"
Sep 07 10:19:10.255485 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.255389    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f3ea3d6-640f-41b7-ad46-ed333f74b4af-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-5p424\" (UID: \"3f3ea3d6-640f-41b7-ad46-ed333f74b4af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"
Sep 07 10:19:10.260658 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.260632    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:19:10.262786 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.262723    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-gzlrq\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="37a810e7-bc2a-4862-89de-d520d0dd5186" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:19:10.263734 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.263712    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:19:10.264142 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.264123    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh6d5\" (UniqueName: \"kubernetes.io/projected/3f3ea3d6-640f-41b7-ad46-ed333f74b4af-kube-api-access-lh6d5\") pod \"kuadrant-operator-controller-manager-96db98c47-5p424\" (UID: \"3f3ea3d6-640f-41b7-ad46-ed333f74b4af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"
Sep 07 10:19:10.264387 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.264357    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzfnx\" (UniqueName: \"kubernetes.io/projected/f4b88551-5dee-416b-bbf7-a514dc9ee2f1-kube-api-access-gzfnx\") pod \"limitador-operator-controller-manager-6c549b6d85-wvfb9\" (UID: \"f4b88551-5dee-416b-bbf7-a514dc9ee2f1\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wvfb9"
Sep 07 10:19:10.264677 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.264658    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sdst\" (UniqueName: \"kubernetes.io/projected/dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e-kube-api-access-8sdst\") pod \"kuadrant-operator-controller-manager-58fc56757c-26mrg\" (UID: \"dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"
Sep 07 10:19:10.264715 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.264667    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cz785\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="accb6f78-2778-4d06-999c-07552b9d49db" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:19:10.266700 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.266681    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cz785\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="accb6f78-2778-4d06-999c-07552b9d49db" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:19:10.268508 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.268492    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-gzlrq\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="37a810e7-bc2a-4862-89de-d520d0dd5186" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:19:10.355946 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.355905    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmk27\" (UniqueName: \"kubernetes.io/projected/37a810e7-bc2a-4862-89de-d520d0dd5186-kube-api-access-rmk27\") pod \"37a810e7-bc2a-4862-89de-d520d0dd5186\" (UID: \"37a810e7-bc2a-4862-89de-d520d0dd5186\") "
Sep 07 10:19:10.356097 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.355964    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97rnh\" (UniqueName: \"kubernetes.io/projected/accb6f78-2778-4d06-999c-07552b9d49db-kube-api-access-97rnh\") pod \"accb6f78-2778-4d06-999c-07552b9d49db\" (UID: \"accb6f78-2778-4d06-999c-07552b9d49db\") "
Sep 07 10:19:10.356097 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.356013    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/accb6f78-2778-4d06-999c-07552b9d49db-extensions-socket-volume\") pod \"accb6f78-2778-4d06-999c-07552b9d49db\" (UID: \"accb6f78-2778-4d06-999c-07552b9d49db\") "
Sep 07 10:19:10.356550 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.356515    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accb6f78-2778-4d06-999c-07552b9d49db-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "accb6f78-2778-4d06-999c-07552b9d49db" (UID: "accb6f78-2778-4d06-999c-07552b9d49db"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:19:10.358264 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.358234    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/accb6f78-2778-4d06-999c-07552b9d49db-kube-api-access-97rnh" (OuterVolumeSpecName: "kube-api-access-97rnh") pod "accb6f78-2778-4d06-999c-07552b9d49db" (UID: "accb6f78-2778-4d06-999c-07552b9d49db"). InnerVolumeSpecName "kube-api-access-97rnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:19:10.358264 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.358241    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a810e7-bc2a-4862-89de-d520d0dd5186-kube-api-access-rmk27" (OuterVolumeSpecName: "kube-api-access-rmk27") pod "37a810e7-bc2a-4862-89de-d520d0dd5186" (UID: "37a810e7-bc2a-4862-89de-d520d0dd5186"). InnerVolumeSpecName "kube-api-access-rmk27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:19:10.383549 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.383506    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"
Sep 07 10:19:10.391877 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.391848    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"
Sep 07 10:19:10.398841 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.398818    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wvfb9"
Sep 07 10:19:10.456781 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.456731    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97rnh\" (UniqueName: \"kubernetes.io/projected/accb6f78-2778-4d06-999c-07552b9d49db-kube-api-access-97rnh\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:19:10.456932 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.456788    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/accb6f78-2778-4d06-999c-07552b9d49db-extensions-socket-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:19:10.456932 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.456804    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmk27\" (UniqueName: \"kubernetes.io/projected/37a810e7-bc2a-4862-89de-d520d0dd5186-kube-api-access-rmk27\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:19:10.665547 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.665505    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"]
Sep 07 10:19:10.667918 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:19:10.667890    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbbf5e4b_cad1_4e12_8873_6e5f26c1d38e.slice/crio-510afbfb45bf60e41e1e762840af1509ae161c7277c8198fb814e4d41dd33dae WatchSource:0}: Error finding container 510afbfb45bf60e41e1e762840af1509ae161c7277c8198fb814e4d41dd33dae: Status 404 returned error can't find the container with id 510afbfb45bf60e41e1e762840af1509ae161c7277c8198fb814e4d41dd33dae
Sep 07 10:19:10.680547 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.680504    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"]
Sep 07 10:19:10.682238 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:19:10.682204    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3ea3d6_640f_41b7_ad46_ed333f74b4af.slice/crio-35ba2cc48049da2880e389cdd763e4a666700e02ac1e5dc94215fae86b4f9f33 WatchSource:0}: Error finding container 35ba2cc48049da2880e389cdd763e4a666700e02ac1e5dc94215fae86b4f9f33: Status 404 returned error can't find the container with id 35ba2cc48049da2880e389cdd763e4a666700e02ac1e5dc94215fae86b4f9f33
Sep 07 10:19:10.695561 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.695512    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wvfb9"]
Sep 07 10:19:10.709929 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:19:10.709900    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b88551_5dee_416b_bbf7_a514dc9ee2f1.slice/crio-0a91e62fd1ae435bd43d041d9c786ff4266060762b5ce39d7a55bf372d694758 WatchSource:0}: Error finding container 0a91e62fd1ae435bd43d041d9c786ff4266060762b5ce39d7a55bf372d694758: Status 404 returned error can't find the container with id 0a91e62fd1ae435bd43d041d9c786ff4266060762b5ce39d7a55bf372d694758
Sep 07 10:19:10.916858 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.916824    2562 generic.go:356] "Generic (PLEG): container finished" podID="37a810e7-bc2a-4862-89de-d520d0dd5186" containerID="29be610bcacc51c41c8ed42a16c240e17b5e96af96311ff52b490fcd6cfd178b" exitCode=0
Sep 07 10:19:10.917048 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.916886    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:19:10.917048 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.916914    2562 scope.go:117] "RemoveContainer" containerID="29be610bcacc51c41c8ed42a16c240e17b5e96af96311ff52b490fcd6cfd178b"
Sep 07 10:19:10.918960 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.918931    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424" event={"ID":"3f3ea3d6-640f-41b7-ad46-ed333f74b4af","Type":"ContainerStarted","Data":"c130d5d561600cf1f1f82c1580b6b464373e5d05ee301ad74ec5676e3d43909e"}
Sep 07 10:19:10.919085 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.918977    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424" event={"ID":"3f3ea3d6-640f-41b7-ad46-ed333f74b4af","Type":"ContainerStarted","Data":"35ba2cc48049da2880e389cdd763e4a666700e02ac1e5dc94215fae86b4f9f33"}
Sep 07 10:19:10.919085 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.919042    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"
Sep 07 10:19:10.920585 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.920561    2562 generic.go:356] "Generic (PLEG): container finished" podID="accb6f78-2778-4d06-999c-07552b9d49db" containerID="8c529a289f5a3274d11b57d88032ce8d4d2d37067267fb6fb68c09ec65a00eb6" exitCode=0
Sep 07 10:19:10.920681 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.920600    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:19:10.921980 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.921951    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cz785\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="accb6f78-2778-4d06-999c-07552b9d49db" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:19:10.922661 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.922637    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wvfb9" event={"ID":"f4b88551-5dee-416b-bbf7-a514dc9ee2f1","Type":"ContainerStarted","Data":"f2c25fd682637bf284840d4c474ad073b3ef0063b8eb7670d300a31062d7a7bb"}
Sep 07 10:19:10.922783 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.922684    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wvfb9" event={"ID":"f4b88551-5dee-416b-bbf7-a514dc9ee2f1","Type":"ContainerStarted","Data":"0a91e62fd1ae435bd43d041d9c786ff4266060762b5ce39d7a55bf372d694758"}
Sep 07 10:19:10.922783 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.922736    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wvfb9"
Sep 07 10:19:10.924658 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.924635    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg" event={"ID":"dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e","Type":"ContainerStarted","Data":"1e647831a9d3b41e4903cfffec339a9ee65f2f68c61c3e1ad4cddeb9cbaded09"}
Sep 07 10:19:10.924748 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.924675    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg" event={"ID":"dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e","Type":"ContainerStarted","Data":"510afbfb45bf60e41e1e762840af1509ae161c7277c8198fb814e4d41dd33dae"}
Sep 07 10:19:10.924831 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.924807    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2" container="manager"
Sep 07 10:19:10.924953 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.924921    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-gzlrq\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="37a810e7-bc2a-4862-89de-d520d0dd5186" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:19:10.927343 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.927314    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cz785\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="accb6f78-2778-4d06-999c-07552b9d49db" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:19:10.930856 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.928876    2562 scope.go:117] "RemoveContainer" containerID="29be610bcacc51c41c8ed42a16c240e17b5e96af96311ff52b490fcd6cfd178b"
Sep 07 10:19:10.931054 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.930921    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29be610bcacc51c41c8ed42a16c240e17b5e96af96311ff52b490fcd6cfd178b\": container with ID starting with 29be610bcacc51c41c8ed42a16c240e17b5e96af96311ff52b490fcd6cfd178b not found: ID does not exist" containerID="29be610bcacc51c41c8ed42a16c240e17b5e96af96311ff52b490fcd6cfd178b"
Sep 07 10:19:10.931054 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.930951    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29be610bcacc51c41c8ed42a16c240e17b5e96af96311ff52b490fcd6cfd178b"} err="failed to get container status \"29be610bcacc51c41c8ed42a16c240e17b5e96af96311ff52b490fcd6cfd178b\": rpc error: code = NotFound desc = could not find container \"29be610bcacc51c41c8ed42a16c240e17b5e96af96311ff52b490fcd6cfd178b\": container with ID starting with 29be610bcacc51c41c8ed42a16c240e17b5e96af96311ff52b490fcd6cfd178b not found: ID does not exist"
Sep 07 10:19:10.931054 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.930973    2562 scope.go:117] "RemoveContainer" containerID="8c529a289f5a3274d11b57d88032ce8d4d2d37067267fb6fb68c09ec65a00eb6"
Sep 07 10:19:10.941246 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.941223    2562 scope.go:117] "RemoveContainer" containerID="8c529a289f5a3274d11b57d88032ce8d4d2d37067267fb6fb68c09ec65a00eb6"
Sep 07 10:19:10.941598 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.941572    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c529a289f5a3274d11b57d88032ce8d4d2d37067267fb6fb68c09ec65a00eb6\": container with ID starting with 8c529a289f5a3274d11b57d88032ce8d4d2d37067267fb6fb68c09ec65a00eb6 not found: ID does not exist" containerID="8c529a289f5a3274d11b57d88032ce8d4d2d37067267fb6fb68c09ec65a00eb6"
Sep 07 10:19:10.941690 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.941607    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c529a289f5a3274d11b57d88032ce8d4d2d37067267fb6fb68c09ec65a00eb6"} err="failed to get container status \"8c529a289f5a3274d11b57d88032ce8d4d2d37067267fb6fb68c09ec65a00eb6\": rpc error: code = NotFound desc = could not find container \"8c529a289f5a3274d11b57d88032ce8d4d2d37067267fb6fb68c09ec65a00eb6\": container with ID starting with 8c529a289f5a3274d11b57d88032ce8d4d2d37067267fb6fb68c09ec65a00eb6 not found: ID does not exist"
Sep 07 10:19:10.964904 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.964847    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wvfb9" podStartSLOduration=0.964826131 podStartE2EDuration="964.826131ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:19:10.961846433 +0000 UTC m=+453.459536363" watchObservedRunningTime="2026-09-07 10:19:10.964826131 +0000 UTC m=+453.462516039"
Sep 07 10:19:10.994547 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:10.994496    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg" podStartSLOduration=1.994479983 podStartE2EDuration="1.994479983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:19:10.992920313 +0000 UTC m=+453.490610261" watchObservedRunningTime="2026-09-07 10:19:10.994479983 +0000 UTC m=+453.492169892"
Sep 07 10:19:10.995367 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.995341    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-gzlrq\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="37a810e7-bc2a-4862-89de-d520d0dd5186" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:19:10.997733 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:10.997709    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-rs5x2\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="e4b15387-99f9-4535-acce-afe2b8d8eb9f" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"
Sep 07 10:19:11.031385 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:11.031331    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424" podStartSLOduration=2.03131221 podStartE2EDuration="2.03131221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:19:11.029380999 +0000 UTC m=+453.527070957" watchObservedRunningTime="2026-09-07 10:19:11.03131221 +0000 UTC m=+453.529002119"
Sep 07 10:19:11.032007 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:11.031977    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-gzlrq\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="37a810e7-bc2a-4862-89de-d520d0dd5186" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:19:11.034739 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:11.034704    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-rs5x2\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="e4b15387-99f9-4535-acce-afe2b8d8eb9f" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"
Sep 07 10:19:11.037065 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:11.037019    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cz785\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="accb6f78-2778-4d06-999c-07552b9d49db" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:19:11.933322 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:11.933283    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2" podUID="e4b15387-99f9-4535-acce-afe2b8d8eb9f" containerName="manager" containerID="cri-o://78e4280fe9eac585f2e72a5d1796e64d3e93c087c2e004607fcc793c0d52886b" gracePeriod=2
Sep 07 10:19:11.933897 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:11.933810    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"
Sep 07 10:19:11.936043 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:11.936008    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-gzlrq\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="37a810e7-bc2a-4862-89de-d520d0dd5186" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-gzlrq"
Sep 07 10:19:11.937941 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:11.937916    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-rs5x2\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="e4b15387-99f9-4535-acce-afe2b8d8eb9f" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"
Sep 07 10:19:11.940034 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:11.940001    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cz785\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="accb6f78-2778-4d06-999c-07552b9d49db" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cz785"
Sep 07 10:19:11.964795 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:11.964756    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37a810e7-bc2a-4862-89de-d520d0dd5186" path="/var/lib/kubelet/pods/37a810e7-bc2a-4862-89de-d520d0dd5186/volumes"
Sep 07 10:19:11.965204 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:11.965185    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="accb6f78-2778-4d06-999c-07552b9d49db" path="/var/lib/kubelet/pods/accb6f78-2778-4d06-999c-07552b9d49db/volumes"
Sep 07 10:19:12.166780 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:12.166743    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"
Sep 07 10:19:12.272335 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:12.272299    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4b15387-99f9-4535-acce-afe2b8d8eb9f-extensions-socket-volume\") pod \"e4b15387-99f9-4535-acce-afe2b8d8eb9f\" (UID: \"e4b15387-99f9-4535-acce-afe2b8d8eb9f\") "
Sep 07 10:19:12.272502 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:12.272371    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf7kx\" (UniqueName: \"kubernetes.io/projected/e4b15387-99f9-4535-acce-afe2b8d8eb9f-kube-api-access-wf7kx\") pod \"e4b15387-99f9-4535-acce-afe2b8d8eb9f\" (UID: \"e4b15387-99f9-4535-acce-afe2b8d8eb9f\") "
Sep 07 10:19:12.272724 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:12.272694    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b15387-99f9-4535-acce-afe2b8d8eb9f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e4b15387-99f9-4535-acce-afe2b8d8eb9f" (UID: "e4b15387-99f9-4535-acce-afe2b8d8eb9f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:19:12.274367 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:12.274344    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b15387-99f9-4535-acce-afe2b8d8eb9f-kube-api-access-wf7kx" (OuterVolumeSpecName: "kube-api-access-wf7kx") pod "e4b15387-99f9-4535-acce-afe2b8d8eb9f" (UID: "e4b15387-99f9-4535-acce-afe2b8d8eb9f"). InnerVolumeSpecName "kube-api-access-wf7kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:19:12.373536 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:12.373491    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4b15387-99f9-4535-acce-afe2b8d8eb9f-extensions-socket-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:19:12.373536 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:12.373525    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf7kx\" (UniqueName: \"kubernetes.io/projected/e4b15387-99f9-4535-acce-afe2b8d8eb9f-kube-api-access-wf7kx\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:19:12.939849 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:12.939808    2562 generic.go:356] "Generic (PLEG): container finished" podID="e4b15387-99f9-4535-acce-afe2b8d8eb9f" containerID="78e4280fe9eac585f2e72a5d1796e64d3e93c087c2e004607fcc793c0d52886b" exitCode=0
Sep 07 10:19:12.940058 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:12.939859    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"
Sep 07 10:19:12.940058 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:12.939909    2562 scope.go:117] "RemoveContainer" containerID="78e4280fe9eac585f2e72a5d1796e64d3e93c087c2e004607fcc793c0d52886b"
Sep 07 10:19:12.948622 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:12.948602    2562 scope.go:117] "RemoveContainer" containerID="78e4280fe9eac585f2e72a5d1796e64d3e93c087c2e004607fcc793c0d52886b"
Sep 07 10:19:12.949002 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:12.948982    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e4280fe9eac585f2e72a5d1796e64d3e93c087c2e004607fcc793c0d52886b\": container with ID starting with 78e4280fe9eac585f2e72a5d1796e64d3e93c087c2e004607fcc793c0d52886b not found: ID does not exist" containerID="78e4280fe9eac585f2e72a5d1796e64d3e93c087c2e004607fcc793c0d52886b"
Sep 07 10:19:12.949072 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:12.949010    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e4280fe9eac585f2e72a5d1796e64d3e93c087c2e004607fcc793c0d52886b"} err="failed to get container status \"78e4280fe9eac585f2e72a5d1796e64d3e93c087c2e004607fcc793c0d52886b\": rpc error: code = NotFound desc = could not find container \"78e4280fe9eac585f2e72a5d1796e64d3e93c087c2e004607fcc793c0d52886b\": container with ID starting with 78e4280fe9eac585f2e72a5d1796e64d3e93c087c2e004607fcc793c0d52886b not found: ID does not exist"
Sep 07 10:19:12.950895 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:12.950870    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-rs5x2\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="e4b15387-99f9-4535-acce-afe2b8d8eb9f" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-rs5x2"
Sep 07 10:19:13.963606 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:13.963570    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4b15387-99f9-4535-acce-afe2b8d8eb9f" path="/var/lib/kubelet/pods/e4b15387-99f9-4535-acce-afe2b8d8eb9f/volumes"
Sep 07 10:19:21.936113 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:21.936078    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"
Sep 07 10:19:21.936577 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:21.936153    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-wvfb9"
Sep 07 10:19:22.081187 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.081149    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"]
Sep 07 10:19:22.081438 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.081393    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg" podUID="dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e" containerName="manager" containerID="cri-o://1e647831a9d3b41e4903cfffec339a9ee65f2f68c61c3e1ad4cddeb9cbaded09" gracePeriod=10
Sep 07 10:19:22.083427 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.083400    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"
Sep 07 10:19:22.323122 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.323096    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"
Sep 07 10:19:22.356032 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.355996    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sdst\" (UniqueName: \"kubernetes.io/projected/dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e-kube-api-access-8sdst\") pod \"dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e\" (UID: \"dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e\") "
Sep 07 10:19:22.358114 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.358084    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e-kube-api-access-8sdst" (OuterVolumeSpecName: "kube-api-access-8sdst") pod "dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e" (UID: "dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e"). InnerVolumeSpecName "kube-api-access-8sdst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:19:22.456733 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.456649    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e-extensions-socket-volume\") pod \"dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e\" (UID: \"dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e\") "
Sep 07 10:19:22.456926 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.456907    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sdst\" (UniqueName: \"kubernetes.io/projected/dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e-kube-api-access-8sdst\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:19:22.457049 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.457032    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e" (UID: "dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:19:22.528655 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.528617    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"]
Sep 07 10:19:22.528983 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.528970    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e" containerName="manager"
Sep 07 10:19:22.529038 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.528984    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e" containerName="manager"
Sep 07 10:19:22.529078 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.529040    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e" containerName="manager"
Sep 07 10:19:22.531032 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.531016    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"
Sep 07 10:19:22.546712 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.546683    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"]
Sep 07 10:19:22.557284 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.557252    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37bc40d6-5bc7-4b15-8341-6eb834084e4d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-gnptw\" (UID: \"37bc40d6-5bc7-4b15-8341-6eb834084e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"
Sep 07 10:19:22.557442 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.557321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmwh\" (UniqueName: \"kubernetes.io/projected/37bc40d6-5bc7-4b15-8341-6eb834084e4d-kube-api-access-xnmwh\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-gnptw\" (UID: \"37bc40d6-5bc7-4b15-8341-6eb834084e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"
Sep 07 10:19:22.557442 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.557352    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e-extensions-socket-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:19:22.658070 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.658028    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37bc40d6-5bc7-4b15-8341-6eb834084e4d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-gnptw\" (UID: \"37bc40d6-5bc7-4b15-8341-6eb834084e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"
Sep 07 10:19:22.658238 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.658112    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmwh\" (UniqueName: \"kubernetes.io/projected/37bc40d6-5bc7-4b15-8341-6eb834084e4d-kube-api-access-xnmwh\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-gnptw\" (UID: \"37bc40d6-5bc7-4b15-8341-6eb834084e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"
Sep 07 10:19:22.658411 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.658390    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37bc40d6-5bc7-4b15-8341-6eb834084e4d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-gnptw\" (UID: \"37bc40d6-5bc7-4b15-8341-6eb834084e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"
Sep 07 10:19:22.672610 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.672574    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmwh\" (UniqueName: \"kubernetes.io/projected/37bc40d6-5bc7-4b15-8341-6eb834084e4d-kube-api-access-xnmwh\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-gnptw\" (UID: \"37bc40d6-5bc7-4b15-8341-6eb834084e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"
Sep 07 10:19:22.841357 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.841271    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"
Sep 07 10:19:22.999424 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.999387    2562 generic.go:356] "Generic (PLEG): container finished" podID="dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e" containerID="1e647831a9d3b41e4903cfffec339a9ee65f2f68c61c3e1ad4cddeb9cbaded09" exitCode=0
Sep 07 10:19:22.999842 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.999431    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"
Sep 07 10:19:22.999842 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.999459    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg" event={"ID":"dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e","Type":"ContainerDied","Data":"1e647831a9d3b41e4903cfffec339a9ee65f2f68c61c3e1ad4cddeb9cbaded09"}
Sep 07 10:19:22.999842 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.999501    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg" event={"ID":"dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e","Type":"ContainerDied","Data":"510afbfb45bf60e41e1e762840af1509ae161c7277c8198fb814e4d41dd33dae"}
Sep 07 10:19:22.999842 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:22.999522    2562 scope.go:117] "RemoveContainer" containerID="1e647831a9d3b41e4903cfffec339a9ee65f2f68c61c3e1ad4cddeb9cbaded09"
Sep 07 10:19:23.008892 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:23.008873    2562 scope.go:117] "RemoveContainer" containerID="1e647831a9d3b41e4903cfffec339a9ee65f2f68c61c3e1ad4cddeb9cbaded09"
Sep 07 10:19:23.009193 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:23.009176    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e647831a9d3b41e4903cfffec339a9ee65f2f68c61c3e1ad4cddeb9cbaded09\": container with ID starting with 1e647831a9d3b41e4903cfffec339a9ee65f2f68c61c3e1ad4cddeb9cbaded09 not found: ID does not exist" containerID="1e647831a9d3b41e4903cfffec339a9ee65f2f68c61c3e1ad4cddeb9cbaded09"
Sep 07 10:19:23.009246 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:23.009200    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e647831a9d3b41e4903cfffec339a9ee65f2f68c61c3e1ad4cddeb9cbaded09"} err="failed to get container status \"1e647831a9d3b41e4903cfffec339a9ee65f2f68c61c3e1ad4cddeb9cbaded09\": rpc error: code = NotFound desc = could not find container \"1e647831a9d3b41e4903cfffec339a9ee65f2f68c61c3e1ad4cddeb9cbaded09\": container with ID starting with 1e647831a9d3b41e4903cfffec339a9ee65f2f68c61c3e1ad4cddeb9cbaded09 not found: ID does not exist"
Sep 07 10:19:23.032511 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:23.032475    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"]
Sep 07 10:19:23.047878 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:23.047841    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-26mrg"]
Sep 07 10:19:23.133775 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:23.133737    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"]
Sep 07 10:19:23.135629 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:19:23.135600    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37bc40d6_5bc7_4b15_8341_6eb834084e4d.slice/crio-c6add9ba6b6c2f4c8781e817ad63a558640963627b1ab9bdcdaedaa60ff0995a WatchSource:0}: Error finding container c6add9ba6b6c2f4c8781e817ad63a558640963627b1ab9bdcdaedaa60ff0995a: Status 404 returned error can't find the container with id c6add9ba6b6c2f4c8781e817ad63a558640963627b1ab9bdcdaedaa60ff0995a
Sep 07 10:19:23.966782 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:23.965645    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e" path="/var/lib/kubelet/pods/dbbf5e4b-cad1-4e12-8873-6e5f26c1d38e/volumes"
Sep 07 10:19:24.006884 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:24.006843    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw" event={"ID":"37bc40d6-5bc7-4b15-8341-6eb834084e4d","Type":"ContainerStarted","Data":"9897999afb695526be589d5da21b9c93e4c8e40e9e550b59c0ebe417d0962726"}
Sep 07 10:19:24.007273 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:24.006895    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw" event={"ID":"37bc40d6-5bc7-4b15-8341-6eb834084e4d","Type":"ContainerStarted","Data":"c6add9ba6b6c2f4c8781e817ad63a558640963627b1ab9bdcdaedaa60ff0995a"}
Sep 07 10:19:24.007273 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:24.006998    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"
Sep 07 10:19:24.028844 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:24.028781    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw" podStartSLOduration=2.028742481 podStartE2EDuration="2.028742481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:19:24.027221956 +0000 UTC m=+466.524911884" watchObservedRunningTime="2026-09-07 10:19:24.028742481 +0000 UTC m=+466.526432391"
Sep 07 10:19:35.014614 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:35.014581    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"
Sep 07 10:19:35.069643 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:35.069612    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"]
Sep 07 10:19:35.069919 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:35.069894    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424" podUID="3f3ea3d6-640f-41b7-ad46-ed333f74b4af" containerName="manager" containerID="cri-o://c130d5d561600cf1f1f82c1580b6b464373e5d05ee301ad74ec5676e3d43909e" gracePeriod=10
Sep 07 10:19:35.304824 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:35.304798    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"
Sep 07 10:19:35.350404 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:35.350369    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh6d5\" (UniqueName: \"kubernetes.io/projected/3f3ea3d6-640f-41b7-ad46-ed333f74b4af-kube-api-access-lh6d5\") pod \"3f3ea3d6-640f-41b7-ad46-ed333f74b4af\" (UID: \"3f3ea3d6-640f-41b7-ad46-ed333f74b4af\") "
Sep 07 10:19:35.350559 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:35.350421    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f3ea3d6-640f-41b7-ad46-ed333f74b4af-extensions-socket-volume\") pod \"3f3ea3d6-640f-41b7-ad46-ed333f74b4af\" (UID: \"3f3ea3d6-640f-41b7-ad46-ed333f74b4af\") "
Sep 07 10:19:35.350791 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:35.350756    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3ea3d6-640f-41b7-ad46-ed333f74b4af-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3f3ea3d6-640f-41b7-ad46-ed333f74b4af" (UID: "3f3ea3d6-640f-41b7-ad46-ed333f74b4af"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:19:35.352435 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:35.352409    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3ea3d6-640f-41b7-ad46-ed333f74b4af-kube-api-access-lh6d5" (OuterVolumeSpecName: "kube-api-access-lh6d5") pod "3f3ea3d6-640f-41b7-ad46-ed333f74b4af" (UID: "3f3ea3d6-640f-41b7-ad46-ed333f74b4af"). InnerVolumeSpecName "kube-api-access-lh6d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:19:35.451699 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:35.451657    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh6d5\" (UniqueName: \"kubernetes.io/projected/3f3ea3d6-640f-41b7-ad46-ed333f74b4af-kube-api-access-lh6d5\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:19:35.451699 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:35.451691    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f3ea3d6-640f-41b7-ad46-ed333f74b4af-extensions-socket-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:19:36.072737 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:36.072696    2562 generic.go:356] "Generic (PLEG): container finished" podID="3f3ea3d6-640f-41b7-ad46-ed333f74b4af" containerID="c130d5d561600cf1f1f82c1580b6b464373e5d05ee301ad74ec5676e3d43909e" exitCode=0
Sep 07 10:19:36.073146 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:36.072756    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"
Sep 07 10:19:36.073146 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:36.072787    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424" event={"ID":"3f3ea3d6-640f-41b7-ad46-ed333f74b4af","Type":"ContainerDied","Data":"c130d5d561600cf1f1f82c1580b6b464373e5d05ee301ad74ec5676e3d43909e"}
Sep 07 10:19:36.073146 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:36.072840    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424" event={"ID":"3f3ea3d6-640f-41b7-ad46-ed333f74b4af","Type":"ContainerDied","Data":"35ba2cc48049da2880e389cdd763e4a666700e02ac1e5dc94215fae86b4f9f33"}
Sep 07 10:19:36.073146 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:36.072862    2562 scope.go:117] "RemoveContainer" containerID="c130d5d561600cf1f1f82c1580b6b464373e5d05ee301ad74ec5676e3d43909e"
Sep 07 10:19:36.081341 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:36.081321    2562 scope.go:117] "RemoveContainer" containerID="c130d5d561600cf1f1f82c1580b6b464373e5d05ee301ad74ec5676e3d43909e"
Sep 07 10:19:36.081614 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:36.081593    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c130d5d561600cf1f1f82c1580b6b464373e5d05ee301ad74ec5676e3d43909e\": container with ID starting with c130d5d561600cf1f1f82c1580b6b464373e5d05ee301ad74ec5676e3d43909e not found: ID does not exist" containerID="c130d5d561600cf1f1f82c1580b6b464373e5d05ee301ad74ec5676e3d43909e"
Sep 07 10:19:36.081658 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:36.081621    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c130d5d561600cf1f1f82c1580b6b464373e5d05ee301ad74ec5676e3d43909e"} err="failed to get container status \"c130d5d561600cf1f1f82c1580b6b464373e5d05ee301ad74ec5676e3d43909e\": rpc error: code = NotFound desc = could not find container \"c130d5d561600cf1f1f82c1580b6b464373e5d05ee301ad74ec5676e3d43909e\": container with ID starting with c130d5d561600cf1f1f82c1580b6b464373e5d05ee301ad74ec5676e3d43909e not found: ID does not exist"
Sep 07 10:19:36.091412 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:36.091385    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"]
Sep 07 10:19:36.094972 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:36.094950    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-5p424"]
Sep 07 10:19:37.964421 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:37.964393    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f3ea3d6-640f-41b7-ad46-ed333f74b4af" path="/var/lib/kubelet/pods/3f3ea3d6-640f-41b7-ad46-ed333f74b4af/volumes"
Sep 07 10:19:52.867435 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:52.867397    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pclz7"]
Sep 07 10:19:52.867935 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:52.867742    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3ea3d6-640f-41b7-ad46-ed333f74b4af" containerName="manager"
Sep 07 10:19:52.867935 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:52.867755    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3ea3d6-640f-41b7-ad46-ed333f74b4af" containerName="manager"
Sep 07 10:19:52.867935 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:52.867854    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3ea3d6-640f-41b7-ad46-ed333f74b4af" containerName="manager"
Sep 07 10:19:52.872442 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:52.872422    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7"
Sep 07 10:19:52.875010 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:52.874987    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 07 10:19:52.875010 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:52.875001    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5pfgz\""
Sep 07 10:19:52.879152 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:52.879127    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pclz7"]
Sep 07 10:19:52.964109 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:52.964068    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pclz7"]
Sep 07 10:19:52.994909 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:52.994876    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zctwh\" (UniqueName: \"kubernetes.io/projected/4f0548a9-64e2-450f-8491-30412e1badcc-kube-api-access-zctwh\") pod \"limitador-limitador-859c4d96d6-pclz7\" (UID: \"4f0548a9-64e2-450f-8491-30412e1badcc\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7"
Sep 07 10:19:52.995057 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:52.994943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4f0548a9-64e2-450f-8491-30412e1badcc-config-file\") pod \"limitador-limitador-859c4d96d6-pclz7\" (UID: \"4f0548a9-64e2-450f-8491-30412e1badcc\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7"
Sep 07 10:19:53.096313 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:53.096278    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zctwh\" (UniqueName: \"kubernetes.io/projected/4f0548a9-64e2-450f-8491-30412e1badcc-kube-api-access-zctwh\") pod \"limitador-limitador-859c4d96d6-pclz7\" (UID: \"4f0548a9-64e2-450f-8491-30412e1badcc\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7"
Sep 07 10:19:53.096503 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:53.096332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4f0548a9-64e2-450f-8491-30412e1badcc-config-file\") pod \"limitador-limitador-859c4d96d6-pclz7\" (UID: \"4f0548a9-64e2-450f-8491-30412e1badcc\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7"
Sep 07 10:19:53.096918 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:53.096901    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4f0548a9-64e2-450f-8491-30412e1badcc-config-file\") pod \"limitador-limitador-859c4d96d6-pclz7\" (UID: \"4f0548a9-64e2-450f-8491-30412e1badcc\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7"
Sep 07 10:19:53.103549 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:53.103523    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zctwh\" (UniqueName: \"kubernetes.io/projected/4f0548a9-64e2-450f-8491-30412e1badcc-kube-api-access-zctwh\") pod \"limitador-limitador-859c4d96d6-pclz7\" (UID: \"4f0548a9-64e2-450f-8491-30412e1badcc\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7"
Sep 07 10:19:53.183948 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:53.183867    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7"
Sep 07 10:19:53.455532 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:53.455495    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pclz7"]
Sep 07 10:19:53.456818 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:19:53.456791    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0548a9_64e2_450f_8491_30412e1badcc.slice/crio-9e5277e28b8eb4bbe8ed10464bc7b140d77bbdbe5add1de2f24f5f35ed22a31e WatchSource:0}: Error finding container 9e5277e28b8eb4bbe8ed10464bc7b140d77bbdbe5add1de2f24f5f35ed22a31e: Status 404 returned error can't find the container with id 9e5277e28b8eb4bbe8ed10464bc7b140d77bbdbe5add1de2f24f5f35ed22a31e
Sep 07 10:19:53.657022 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:53.656982    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-r975n"]
Sep 07 10:19:53.661865 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:53.661842    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-r975n"
Sep 07 10:19:53.664094 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:53.664073    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cbrkq\""
Sep 07 10:19:53.666849 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:53.666824    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-r975n"]
Sep 07 10:19:53.702320 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:53.702286    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fw48\" (UniqueName: \"kubernetes.io/projected/920968c6-c05b-432c-ad96-b29a2c7c06d6-kube-api-access-5fw48\") pod \"authorino-b54df94b4-r975n\" (UID: \"920968c6-c05b-432c-ad96-b29a2c7c06d6\") " pod="kuadrant-system/authorino-b54df94b4-r975n"
Sep 07 10:19:53.803465 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:53.803380    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fw48\" (UniqueName: \"kubernetes.io/projected/920968c6-c05b-432c-ad96-b29a2c7c06d6-kube-api-access-5fw48\") pod \"authorino-b54df94b4-r975n\" (UID: \"920968c6-c05b-432c-ad96-b29a2c7c06d6\") " pod="kuadrant-system/authorino-b54df94b4-r975n"
Sep 07 10:19:53.811888 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:53.811855    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fw48\" (UniqueName: \"kubernetes.io/projected/920968c6-c05b-432c-ad96-b29a2c7c06d6-kube-api-access-5fw48\") pod \"authorino-b54df94b4-r975n\" (UID: \"920968c6-c05b-432c-ad96-b29a2c7c06d6\") " pod="kuadrant-system/authorino-b54df94b4-r975n"
Sep 07 10:19:53.972040 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:53.972002    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-r975n"
Sep 07 10:19:54.163812 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:54.163704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7" event={"ID":"4f0548a9-64e2-450f-8491-30412e1badcc","Type":"ContainerStarted","Data":"9e5277e28b8eb4bbe8ed10464bc7b140d77bbdbe5add1de2f24f5f35ed22a31e"}
Sep 07 10:19:54.244052 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:54.244023    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-r975n"]
Sep 07 10:19:54.246404 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:19:54.246374    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod920968c6_c05b_432c_ad96_b29a2c7c06d6.slice/crio-3034b4eab92c4ee5a0cb89cc632bb2478cea71825de9934da3a92e9c22278567 WatchSource:0}: Error finding container 3034b4eab92c4ee5a0cb89cc632bb2478cea71825de9934da3a92e9c22278567: Status 404 returned error can't find the container with id 3034b4eab92c4ee5a0cb89cc632bb2478cea71825de9934da3a92e9c22278567
Sep 07 10:19:55.173053 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:55.173006    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7" event={"ID":"4f0548a9-64e2-450f-8491-30412e1badcc","Type":"ContainerStarted","Data":"99675408b861b8b827d8d1f9c8f1e36f150cdb19822b1bb01b78cc7bc468aa2e"}
Sep 07 10:19:55.173524 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:55.173166    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7"
Sep 07 10:19:55.174736 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:55.174705    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-r975n" event={"ID":"920968c6-c05b-432c-ad96-b29a2c7c06d6","Type":"ContainerStarted","Data":"3034b4eab92c4ee5a0cb89cc632bb2478cea71825de9934da3a92e9c22278567"}
Sep 07 10:19:55.191590 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:55.191277    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7" podStartSLOduration=2.038842484 podStartE2EDuration="3.191219798s" totalImagesPullingTime="1.152377314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:19:55.189227371 +0000 UTC m=+497.686917306" watchObservedRunningTime="2026-09-07 10:19:55.191219798 +0000 UTC m=+497.688909707"
Sep 07 10:19:57.150814 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:57.150716    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-r975n"]
Sep 07 10:19:57.187464 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:57.187424    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-r975n" event={"ID":"920968c6-c05b-432c-ad96-b29a2c7c06d6","Type":"ContainerStarted","Data":"1e152740952b921d2cf343028b65e03f8e92da3ba57e5e11b010545f96946d78"}
Sep 07 10:19:57.202638 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:57.202585    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-r975n" podStartSLOduration=1.575181503 podStartE2EDuration="4.202569212s" totalImagesPullingTime="2.627387709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:19:57.20197739 +0000 UTC m=+499.699667304" watchObservedRunningTime="2026-09-07 10:19:57.202569212 +0000 UTC m=+499.700259121"
Sep 07 10:19:58.192058 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:58.191994    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-r975n" podUID="920968c6-c05b-432c-ad96-b29a2c7c06d6" containerName="authorino" containerID="cri-o://1e152740952b921d2cf343028b65e03f8e92da3ba57e5e11b010545f96946d78" gracePeriod=30
Sep 07 10:19:58.439170 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:58.439147    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-r975n"
Sep 07 10:19:58.553318 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:58.553280    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fw48\" (UniqueName: \"kubernetes.io/projected/920968c6-c05b-432c-ad96-b29a2c7c06d6-kube-api-access-5fw48\") pod \"920968c6-c05b-432c-ad96-b29a2c7c06d6\" (UID: \"920968c6-c05b-432c-ad96-b29a2c7c06d6\") "
Sep 07 10:19:58.555306 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:58.555279    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920968c6-c05b-432c-ad96-b29a2c7c06d6-kube-api-access-5fw48" (OuterVolumeSpecName: "kube-api-access-5fw48") pod "920968c6-c05b-432c-ad96-b29a2c7c06d6" (UID: "920968c6-c05b-432c-ad96-b29a2c7c06d6"). InnerVolumeSpecName "kube-api-access-5fw48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:19:58.653837 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:58.653800    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fw48\" (UniqueName: \"kubernetes.io/projected/920968c6-c05b-432c-ad96-b29a2c7c06d6-kube-api-access-5fw48\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:19:59.198919 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:59.198887    2562 generic.go:356] "Generic (PLEG): container finished" podID="920968c6-c05b-432c-ad96-b29a2c7c06d6" containerID="1e152740952b921d2cf343028b65e03f8e92da3ba57e5e11b010545f96946d78" exitCode=0
Sep 07 10:19:59.199364 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:59.198932    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-r975n" event={"ID":"920968c6-c05b-432c-ad96-b29a2c7c06d6","Type":"ContainerDied","Data":"1e152740952b921d2cf343028b65e03f8e92da3ba57e5e11b010545f96946d78"}
Sep 07 10:19:59.199364 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:59.198936    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-r975n"
Sep 07 10:19:59.199364 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:59.198959    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-r975n" event={"ID":"920968c6-c05b-432c-ad96-b29a2c7c06d6","Type":"ContainerDied","Data":"3034b4eab92c4ee5a0cb89cc632bb2478cea71825de9934da3a92e9c22278567"}
Sep 07 10:19:59.199364 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:59.198976    2562 scope.go:117] "RemoveContainer" containerID="1e152740952b921d2cf343028b65e03f8e92da3ba57e5e11b010545f96946d78"
Sep 07 10:19:59.208471 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:59.208446    2562 scope.go:117] "RemoveContainer" containerID="1e152740952b921d2cf343028b65e03f8e92da3ba57e5e11b010545f96946d78"
Sep 07 10:19:59.208739 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:19:59.208721    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e152740952b921d2cf343028b65e03f8e92da3ba57e5e11b010545f96946d78\": container with ID starting with 1e152740952b921d2cf343028b65e03f8e92da3ba57e5e11b010545f96946d78 not found: ID does not exist" containerID="1e152740952b921d2cf343028b65e03f8e92da3ba57e5e11b010545f96946d78"
Sep 07 10:19:59.208817 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:59.208748    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e152740952b921d2cf343028b65e03f8e92da3ba57e5e11b010545f96946d78"} err="failed to get container status \"1e152740952b921d2cf343028b65e03f8e92da3ba57e5e11b010545f96946d78\": rpc error: code = NotFound desc = could not find container \"1e152740952b921d2cf343028b65e03f8e92da3ba57e5e11b010545f96946d78\": container with ID starting with 1e152740952b921d2cf343028b65e03f8e92da3ba57e5e11b010545f96946d78 not found: ID does not exist"
Sep 07 10:19:59.219246 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:59.219216    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-r975n"]
Sep 07 10:19:59.224679 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:59.224656    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-r975n"]
Sep 07 10:19:59.964463 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:19:59.964430    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="920968c6-c05b-432c-ad96-b29a2c7c06d6" path="/var/lib/kubelet/pods/920968c6-c05b-432c-ad96-b29a2c7c06d6/volumes"
Sep 07 10:20:06.181326 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:06.181296    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7"
Sep 07 10:20:08.160469 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:08.160432    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pclz7"]
Sep 07 10:20:08.160952 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:08.160674    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7" podUID="4f0548a9-64e2-450f-8491-30412e1badcc" containerName="limitador" containerID="cri-o://99675408b861b8b827d8d1f9c8f1e36f150cdb19822b1bb01b78cc7bc468aa2e" gracePeriod=30
Sep 07 10:20:08.706751 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:08.706727    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7"
Sep 07 10:20:08.838623 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:08.838532    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4f0548a9-64e2-450f-8491-30412e1badcc-config-file\") pod \"4f0548a9-64e2-450f-8491-30412e1badcc\" (UID: \"4f0548a9-64e2-450f-8491-30412e1badcc\") "
Sep 07 10:20:08.838799 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:08.838667    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zctwh\" (UniqueName: \"kubernetes.io/projected/4f0548a9-64e2-450f-8491-30412e1badcc-kube-api-access-zctwh\") pod \"4f0548a9-64e2-450f-8491-30412e1badcc\" (UID: \"4f0548a9-64e2-450f-8491-30412e1badcc\") "
Sep 07 10:20:08.838971 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:08.838944    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0548a9-64e2-450f-8491-30412e1badcc-config-file" (OuterVolumeSpecName: "config-file") pod "4f0548a9-64e2-450f-8491-30412e1badcc" (UID: "4f0548a9-64e2-450f-8491-30412e1badcc"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:20:08.840695 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:08.840670    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0548a9-64e2-450f-8491-30412e1badcc-kube-api-access-zctwh" (OuterVolumeSpecName: "kube-api-access-zctwh") pod "4f0548a9-64e2-450f-8491-30412e1badcc" (UID: "4f0548a9-64e2-450f-8491-30412e1badcc"). InnerVolumeSpecName "kube-api-access-zctwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:20:08.939351 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:08.939322    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zctwh\" (UniqueName: \"kubernetes.io/projected/4f0548a9-64e2-450f-8491-30412e1badcc-kube-api-access-zctwh\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:20:08.939351 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:08.939350    2562 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4f0548a9-64e2-450f-8491-30412e1badcc-config-file\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:20:09.254564 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:09.254526    2562 generic.go:356] "Generic (PLEG): container finished" podID="4f0548a9-64e2-450f-8491-30412e1badcc" containerID="99675408b861b8b827d8d1f9c8f1e36f150cdb19822b1bb01b78cc7bc468aa2e" exitCode=0
Sep 07 10:20:09.255059 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:09.254586    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7"
Sep 07 10:20:09.255059 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:09.254603    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7" event={"ID":"4f0548a9-64e2-450f-8491-30412e1badcc","Type":"ContainerDied","Data":"99675408b861b8b827d8d1f9c8f1e36f150cdb19822b1bb01b78cc7bc468aa2e"}
Sep 07 10:20:09.255059 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:09.254660    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-pclz7" event={"ID":"4f0548a9-64e2-450f-8491-30412e1badcc","Type":"ContainerDied","Data":"9e5277e28b8eb4bbe8ed10464bc7b140d77bbdbe5add1de2f24f5f35ed22a31e"}
Sep 07 10:20:09.255059 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:09.254686    2562 scope.go:117] "RemoveContainer" containerID="99675408b861b8b827d8d1f9c8f1e36f150cdb19822b1bb01b78cc7bc468aa2e"
Sep 07 10:20:09.264010 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:09.263985    2562 scope.go:117] "RemoveContainer" containerID="99675408b861b8b827d8d1f9c8f1e36f150cdb19822b1bb01b78cc7bc468aa2e"
Sep 07 10:20:09.264302 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:20:09.264279    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99675408b861b8b827d8d1f9c8f1e36f150cdb19822b1bb01b78cc7bc468aa2e\": container with ID starting with 99675408b861b8b827d8d1f9c8f1e36f150cdb19822b1bb01b78cc7bc468aa2e not found: ID does not exist" containerID="99675408b861b8b827d8d1f9c8f1e36f150cdb19822b1bb01b78cc7bc468aa2e"
Sep 07 10:20:09.264374 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:09.264313    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99675408b861b8b827d8d1f9c8f1e36f150cdb19822b1bb01b78cc7bc468aa2e"} err="failed to get container status \"99675408b861b8b827d8d1f9c8f1e36f150cdb19822b1bb01b78cc7bc468aa2e\": rpc error: code = NotFound desc = could not find container \"99675408b861b8b827d8d1f9c8f1e36f150cdb19822b1bb01b78cc7bc468aa2e\": container with ID starting with 99675408b861b8b827d8d1f9c8f1e36f150cdb19822b1bb01b78cc7bc468aa2e not found: ID does not exist"
Sep 07 10:20:09.275123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:09.275091    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pclz7"]
Sep 07 10:20:09.279484 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:09.279458    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pclz7"]
Sep 07 10:20:09.963855 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:09.963822    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f0548a9-64e2-450f-8491-30412e1badcc" path="/var/lib/kubelet/pods/4f0548a9-64e2-450f-8491-30412e1badcc/volumes"
Sep 07 10:20:13.078776 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.078731    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-46pxc"]
Sep 07 10:20:13.079145 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.079084    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f0548a9-64e2-450f-8491-30412e1badcc" containerName="limitador"
Sep 07 10:20:13.079145 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.079096    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f0548a9-64e2-450f-8491-30412e1badcc" containerName="limitador"
Sep 07 10:20:13.079145 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.079104    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920968c6-c05b-432c-ad96-b29a2c7c06d6" containerName="authorino"
Sep 07 10:20:13.079145 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.079110    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="920968c6-c05b-432c-ad96-b29a2c7c06d6" containerName="authorino"
Sep 07 10:20:13.079264 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.079158    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f0548a9-64e2-450f-8491-30412e1badcc" containerName="limitador"
Sep 07 10:20:13.079264 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.079166    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="920968c6-c05b-432c-ad96-b29a2c7c06d6" containerName="authorino"
Sep 07 10:20:13.083338 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.083319    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-46pxc"
Sep 07 10:20:13.085784 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.085749    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 07 10:20:13.085896 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.085806    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 10:20:13.086729 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.086703    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 10:20:13.086877 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.086712    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fnwtt\""
Sep 07 10:20:13.089405 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.089385    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-46pxc"]
Sep 07 10:20:13.176588 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.176554    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/657ea806-9abd-4930-bb33-12f76eebf5b2-data\") pod \"postgres-69fb4594cf-46pxc\" (UID: \"657ea806-9abd-4930-bb33-12f76eebf5b2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-46pxc"
Sep 07 10:20:13.176588 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.176592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc5lb\" (UniqueName: \"kubernetes.io/projected/657ea806-9abd-4930-bb33-12f76eebf5b2-kube-api-access-rc5lb\") pod \"postgres-69fb4594cf-46pxc\" (UID: \"657ea806-9abd-4930-bb33-12f76eebf5b2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-46pxc"
Sep 07 10:20:13.277389 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.277355    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/657ea806-9abd-4930-bb33-12f76eebf5b2-data\") pod \"postgres-69fb4594cf-46pxc\" (UID: \"657ea806-9abd-4930-bb33-12f76eebf5b2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-46pxc"
Sep 07 10:20:13.277389 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.277392    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc5lb\" (UniqueName: \"kubernetes.io/projected/657ea806-9abd-4930-bb33-12f76eebf5b2-kube-api-access-rc5lb\") pod \"postgres-69fb4594cf-46pxc\" (UID: \"657ea806-9abd-4930-bb33-12f76eebf5b2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-46pxc"
Sep 07 10:20:13.277754 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.277734    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/657ea806-9abd-4930-bb33-12f76eebf5b2-data\") pod \"postgres-69fb4594cf-46pxc\" (UID: \"657ea806-9abd-4930-bb33-12f76eebf5b2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-46pxc"
Sep 07 10:20:13.285104 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.285081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc5lb\" (UniqueName: \"kubernetes.io/projected/657ea806-9abd-4930-bb33-12f76eebf5b2-kube-api-access-rc5lb\") pod \"postgres-69fb4594cf-46pxc\" (UID: \"657ea806-9abd-4930-bb33-12f76eebf5b2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-46pxc"
Sep 07 10:20:13.395059 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.394964    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-46pxc"
Sep 07 10:20:13.664393 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:13.664164    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-46pxc"]
Sep 07 10:20:13.666566 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:20:13.666537    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657ea806_9abd_4930_bb33_12f76eebf5b2.slice/crio-8f8cdcde5413c91017157b22cb5607413c07a7a1ccbe78255d35daf227a5263a WatchSource:0}: Error finding container 8f8cdcde5413c91017157b22cb5607413c07a7a1ccbe78255d35daf227a5263a: Status 404 returned error can't find the container with id 8f8cdcde5413c91017157b22cb5607413c07a7a1ccbe78255d35daf227a5263a
Sep 07 10:20:14.282090 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:14.282051    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-46pxc" event={"ID":"657ea806-9abd-4930-bb33-12f76eebf5b2","Type":"ContainerStarted","Data":"8f8cdcde5413c91017157b22cb5607413c07a7a1ccbe78255d35daf227a5263a"}
Sep 07 10:20:19.308630 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:19.308589    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-46pxc" event={"ID":"657ea806-9abd-4930-bb33-12f76eebf5b2","Type":"ContainerStarted","Data":"f5a9e9e4a28d239141a58b58b50378d16f1d01dc60fa27deda1728af3b3e4199"}
Sep 07 10:20:19.309009 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:19.308722    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-46pxc"
Sep 07 10:20:19.323841 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:19.323794    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-46pxc" podStartSLOduration=1.600798393 podStartE2EDuration="6.323778088s" totalImagesPullingTime="4.722979695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:19.322785395 +0000 UTC m=+521.820475322" watchObservedRunningTime="2026-09-07 10:20:19.323778088 +0000 UTC m=+521.821468355"
Sep 07 10:20:25.342968 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:25.342934    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-46pxc"
Sep 07 10:20:25.932496 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:25.932460    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-hdqfc"]
Sep 07 10:20:25.937587 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:25.937563    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-hdqfc"
Sep 07 10:20:25.939951 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:25.939926    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cbrkq\""
Sep 07 10:20:25.942945 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:25.942923    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-hdqfc"]
Sep 07 10:20:25.980089 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:25.980054    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptxsm\" (UniqueName: \"kubernetes.io/projected/11696321-273d-4f6c-8403-24d34dea73e5-kube-api-access-ptxsm\") pod \"authorino-5dc7ccb5f-hdqfc\" (UID: \"11696321-273d-4f6c-8403-24d34dea73e5\") " pod="kuadrant-system/authorino-5dc7ccb5f-hdqfc"
Sep 07 10:20:25.995425 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:25.995395    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-hdqfc"]
Sep 07 10:20:25.995635 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:20:25.995614    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ptxsm], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-hdqfc" podUID="11696321-273d-4f6c-8403-24d34dea73e5"
Sep 07 10:20:26.021916 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.021880    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-n22p9"]
Sep 07 10:20:26.025200 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.025173    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-n22p9"
Sep 07 10:20:26.027456 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.027424    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 07 10:20:26.032945 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.032921    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-n22p9"]
Sep 07 10:20:26.080474 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.080429    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c6f6d114-fc8a-4269-85dd-73ba49bb019e-tls-cert\") pod \"authorino-687bd45585-n22p9\" (UID: \"c6f6d114-fc8a-4269-85dd-73ba49bb019e\") " pod="kuadrant-system/authorino-687bd45585-n22p9"
Sep 07 10:20:26.080653 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.080484    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptxsm\" (UniqueName: \"kubernetes.io/projected/11696321-273d-4f6c-8403-24d34dea73e5-kube-api-access-ptxsm\") pod \"authorino-5dc7ccb5f-hdqfc\" (UID: \"11696321-273d-4f6c-8403-24d34dea73e5\") " pod="kuadrant-system/authorino-5dc7ccb5f-hdqfc"
Sep 07 10:20:26.080653 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.080541    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhjzt\" (UniqueName: \"kubernetes.io/projected/c6f6d114-fc8a-4269-85dd-73ba49bb019e-kube-api-access-fhjzt\") pod \"authorino-687bd45585-n22p9\" (UID: \"c6f6d114-fc8a-4269-85dd-73ba49bb019e\") " pod="kuadrant-system/authorino-687bd45585-n22p9"
Sep 07 10:20:26.088552 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.088519    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptxsm\" (UniqueName: \"kubernetes.io/projected/11696321-273d-4f6c-8403-24d34dea73e5-kube-api-access-ptxsm\") pod \"authorino-5dc7ccb5f-hdqfc\" (UID: \"11696321-273d-4f6c-8403-24d34dea73e5\") " pod="kuadrant-system/authorino-5dc7ccb5f-hdqfc"
Sep 07 10:20:26.181138 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.181100    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c6f6d114-fc8a-4269-85dd-73ba49bb019e-tls-cert\") pod \"authorino-687bd45585-n22p9\" (UID: \"c6f6d114-fc8a-4269-85dd-73ba49bb019e\") " pod="kuadrant-system/authorino-687bd45585-n22p9"
Sep 07 10:20:26.181340 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.181163    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhjzt\" (UniqueName: \"kubernetes.io/projected/c6f6d114-fc8a-4269-85dd-73ba49bb019e-kube-api-access-fhjzt\") pod \"authorino-687bd45585-n22p9\" (UID: \"c6f6d114-fc8a-4269-85dd-73ba49bb019e\") " pod="kuadrant-system/authorino-687bd45585-n22p9"
Sep 07 10:20:26.183185 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.183122    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-n22p9"]
Sep 07 10:20:26.183367 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:20:26.183345    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fhjzt tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-n22p9" podUID="c6f6d114-fc8a-4269-85dd-73ba49bb019e"
Sep 07 10:20:26.183735 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.183705    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c6f6d114-fc8a-4269-85dd-73ba49bb019e-tls-cert\") pod \"authorino-687bd45585-n22p9\" (UID: \"c6f6d114-fc8a-4269-85dd-73ba49bb019e\") " pod="kuadrant-system/authorino-687bd45585-n22p9"
Sep 07 10:20:26.195396 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.195368    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhjzt\" (UniqueName: \"kubernetes.io/projected/c6f6d114-fc8a-4269-85dd-73ba49bb019e-kube-api-access-fhjzt\") pod \"authorino-687bd45585-n22p9\" (UID: \"c6f6d114-fc8a-4269-85dd-73ba49bb019e\") " pod="kuadrant-system/authorino-687bd45585-n22p9"
Sep 07 10:20:26.342121 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.342086    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-n22p9"
Sep 07 10:20:26.342308 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.342086    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-hdqfc"
Sep 07 10:20:26.349718 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.349696    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-n22p9"
Sep 07 10:20:26.352465 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.352448    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-hdqfc"
Sep 07 10:20:26.382250 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.382215    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptxsm\" (UniqueName: \"kubernetes.io/projected/11696321-273d-4f6c-8403-24d34dea73e5-kube-api-access-ptxsm\") pod \"11696321-273d-4f6c-8403-24d34dea73e5\" (UID: \"11696321-273d-4f6c-8403-24d34dea73e5\") "
Sep 07 10:20:26.382250 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.382249    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c6f6d114-fc8a-4269-85dd-73ba49bb019e-tls-cert\") pod \"c6f6d114-fc8a-4269-85dd-73ba49bb019e\" (UID: \"c6f6d114-fc8a-4269-85dd-73ba49bb019e\") "
Sep 07 10:20:26.384359 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.384328    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11696321-273d-4f6c-8403-24d34dea73e5-kube-api-access-ptxsm" (OuterVolumeSpecName: "kube-api-access-ptxsm") pod "11696321-273d-4f6c-8403-24d34dea73e5" (UID: "11696321-273d-4f6c-8403-24d34dea73e5"). InnerVolumeSpecName "kube-api-access-ptxsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:20:26.384359 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.384332    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f6d114-fc8a-4269-85dd-73ba49bb019e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c6f6d114-fc8a-4269-85dd-73ba49bb019e" (UID: "c6f6d114-fc8a-4269-85dd-73ba49bb019e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:20:26.483258 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.483224    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhjzt\" (UniqueName: \"kubernetes.io/projected/c6f6d114-fc8a-4269-85dd-73ba49bb019e-kube-api-access-fhjzt\") pod \"c6f6d114-fc8a-4269-85dd-73ba49bb019e\" (UID: \"c6f6d114-fc8a-4269-85dd-73ba49bb019e\") "
Sep 07 10:20:26.483440 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.483398    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptxsm\" (UniqueName: \"kubernetes.io/projected/11696321-273d-4f6c-8403-24d34dea73e5-kube-api-access-ptxsm\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:20:26.483440 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.483415    2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c6f6d114-fc8a-4269-85dd-73ba49bb019e-tls-cert\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:20:26.485276 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.485239    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f6d114-fc8a-4269-85dd-73ba49bb019e-kube-api-access-fhjzt" (OuterVolumeSpecName: "kube-api-access-fhjzt") pod "c6f6d114-fc8a-4269-85dd-73ba49bb019e" (UID: "c6f6d114-fc8a-4269-85dd-73ba49bb019e"). InnerVolumeSpecName "kube-api-access-fhjzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:20:26.583859 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:26.583819    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhjzt\" (UniqueName: \"kubernetes.io/projected/c6f6d114-fc8a-4269-85dd-73ba49bb019e-kube-api-access-fhjzt\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:20:27.347509 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:27.347477    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-hdqfc"
Sep 07 10:20:27.347704 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:27.347478    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-n22p9"
Sep 07 10:20:27.381085 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:27.381046    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-hdqfc"]
Sep 07 10:20:27.384858 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:27.384829    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-hdqfc"]
Sep 07 10:20:27.406508 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:27.406455    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-n22p9"]
Sep 07 10:20:27.408241 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:27.408217    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-n22p9"]
Sep 07 10:20:27.969402 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:27.969362    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11696321-273d-4f6c-8403-24d34dea73e5" path="/var/lib/kubelet/pods/11696321-273d-4f6c-8403-24d34dea73e5/volumes"
Sep 07 10:20:27.969620 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:27.969608    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6f6d114-fc8a-4269-85dd-73ba49bb019e" path="/var/lib/kubelet/pods/c6f6d114-fc8a-4269-85dd-73ba49bb019e/volumes"
Sep 07 10:20:33.364449 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.364366    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6845bf5f8f-k8qbw"]
Sep 07 10:20:33.368002 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.367974    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6845bf5f8f-k8qbw"
Sep 07 10:20:33.370394 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.370365    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 07 10:20:33.370609 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.370576    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lh264\""
Sep 07 10:20:33.371346 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.371316    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 07 10:20:33.371455 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.371396    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 07 10:20:33.376525 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.376490    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6845bf5f8f-k8qbw"]
Sep 07 10:20:33.431872 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.431830    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qdqx\" (UniqueName: \"kubernetes.io/projected/31c373a1-bb08-458c-8096-b961df47e89a-kube-api-access-4qdqx\") pod \"maas-controller-6845bf5f8f-k8qbw\" (UID: \"31c373a1-bb08-458c-8096-b961df47e89a\") " pod="opendatahub/maas-controller-6845bf5f8f-k8qbw"
Sep 07 10:20:33.432072 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.431901    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c373a1-bb08-458c-8096-b961df47e89a-metrics-certs\") pod \"maas-controller-6845bf5f8f-k8qbw\" (UID: \"31c373a1-bb08-458c-8096-b961df47e89a\") " pod="opendatahub/maas-controller-6845bf5f8f-k8qbw"
Sep 07 10:20:33.432072 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.431954    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31c373a1-bb08-458c-8096-b961df47e89a-webhook-cert\") pod \"maas-controller-6845bf5f8f-k8qbw\" (UID: \"31c373a1-bb08-458c-8096-b961df47e89a\") " pod="opendatahub/maas-controller-6845bf5f8f-k8qbw"
Sep 07 10:20:33.532378 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.532334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31c373a1-bb08-458c-8096-b961df47e89a-webhook-cert\") pod \"maas-controller-6845bf5f8f-k8qbw\" (UID: \"31c373a1-bb08-458c-8096-b961df47e89a\") " pod="opendatahub/maas-controller-6845bf5f8f-k8qbw"
Sep 07 10:20:33.532378 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.532385    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qdqx\" (UniqueName: \"kubernetes.io/projected/31c373a1-bb08-458c-8096-b961df47e89a-kube-api-access-4qdqx\") pod \"maas-controller-6845bf5f8f-k8qbw\" (UID: \"31c373a1-bb08-458c-8096-b961df47e89a\") " pod="opendatahub/maas-controller-6845bf5f8f-k8qbw"
Sep 07 10:20:33.532596 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.532443    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c373a1-bb08-458c-8096-b961df47e89a-metrics-certs\") pod \"maas-controller-6845bf5f8f-k8qbw\" (UID: \"31c373a1-bb08-458c-8096-b961df47e89a\") " pod="opendatahub/maas-controller-6845bf5f8f-k8qbw"
Sep 07 10:20:33.532596 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:20:33.532510    2562 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 07 10:20:33.532596 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:20:33.532592    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31c373a1-bb08-458c-8096-b961df47e89a-webhook-cert podName:31c373a1-bb08-458c-8096-b961df47e89a nodeName:}" failed. No retries permitted until 2026-09-07 10:20:34.032573419 +0000 UTC m=+536.530263311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/31c373a1-bb08-458c-8096-b961df47e89a-webhook-cert") pod "maas-controller-6845bf5f8f-k8qbw" (UID: "31c373a1-bb08-458c-8096-b961df47e89a") : secret "maas-controller-webhook-cert" not found
Sep 07 10:20:33.535027 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.534996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c373a1-bb08-458c-8096-b961df47e89a-metrics-certs\") pod \"maas-controller-6845bf5f8f-k8qbw\" (UID: \"31c373a1-bb08-458c-8096-b961df47e89a\") " pod="opendatahub/maas-controller-6845bf5f8f-k8qbw"
Sep 07 10:20:33.539002 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.538974    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6899cfd859-v7wgq"]
Sep 07 10:20:33.542468 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.542448    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899cfd859-v7wgq"
Sep 07 10:20:33.547000 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.546974    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qdqx\" (UniqueName: \"kubernetes.io/projected/31c373a1-bb08-458c-8096-b961df47e89a-kube-api-access-4qdqx\") pod \"maas-controller-6845bf5f8f-k8qbw\" (UID: \"31c373a1-bb08-458c-8096-b961df47e89a\") " pod="opendatahub/maas-controller-6845bf5f8f-k8qbw"
Sep 07 10:20:33.552903 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.552876    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6899cfd859-v7wgq"]
Sep 07 10:20:33.633506 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.633421    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/278ec7ee-2beb-4a50-8961-b10b5c150d5b-webhook-cert\") pod \"maas-controller-6899cfd859-v7wgq\" (UID: \"278ec7ee-2beb-4a50-8961-b10b5c150d5b\") " pod="opendatahub/maas-controller-6899cfd859-v7wgq"
Sep 07 10:20:33.633506 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.633462    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278ec7ee-2beb-4a50-8961-b10b5c150d5b-metrics-certs\") pod \"maas-controller-6899cfd859-v7wgq\" (UID: \"278ec7ee-2beb-4a50-8961-b10b5c150d5b\") " pod="opendatahub/maas-controller-6899cfd859-v7wgq"
Sep 07 10:20:33.633683 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.633512    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4kjx\" (UniqueName: \"kubernetes.io/projected/278ec7ee-2beb-4a50-8961-b10b5c150d5b-kube-api-access-v4kjx\") pod \"maas-controller-6899cfd859-v7wgq\" (UID: \"278ec7ee-2beb-4a50-8961-b10b5c150d5b\") " pod="opendatahub/maas-controller-6899cfd859-v7wgq"
Sep 07 10:20:33.734451 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.734407    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/278ec7ee-2beb-4a50-8961-b10b5c150d5b-webhook-cert\") pod \"maas-controller-6899cfd859-v7wgq\" (UID: \"278ec7ee-2beb-4a50-8961-b10b5c150d5b\") " pod="opendatahub/maas-controller-6899cfd859-v7wgq"
Sep 07 10:20:33.734451 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.734453    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278ec7ee-2beb-4a50-8961-b10b5c150d5b-metrics-certs\") pod \"maas-controller-6899cfd859-v7wgq\" (UID: \"278ec7ee-2beb-4a50-8961-b10b5c150d5b\") " pod="opendatahub/maas-controller-6899cfd859-v7wgq"
Sep 07 10:20:33.734712 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.734491    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4kjx\" (UniqueName: \"kubernetes.io/projected/278ec7ee-2beb-4a50-8961-b10b5c150d5b-kube-api-access-v4kjx\") pod \"maas-controller-6899cfd859-v7wgq\" (UID: \"278ec7ee-2beb-4a50-8961-b10b5c150d5b\") " pod="opendatahub/maas-controller-6899cfd859-v7wgq"
Sep 07 10:20:33.736961 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.736937    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/278ec7ee-2beb-4a50-8961-b10b5c150d5b-webhook-cert\") pod \"maas-controller-6899cfd859-v7wgq\" (UID: \"278ec7ee-2beb-4a50-8961-b10b5c150d5b\") " pod="opendatahub/maas-controller-6899cfd859-v7wgq"
Sep 07 10:20:33.737085 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.736966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278ec7ee-2beb-4a50-8961-b10b5c150d5b-metrics-certs\") pod \"maas-controller-6899cfd859-v7wgq\" (UID: \"278ec7ee-2beb-4a50-8961-b10b5c150d5b\") " pod="opendatahub/maas-controller-6899cfd859-v7wgq"
Sep 07 10:20:33.742665 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.742630    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4kjx\" (UniqueName: \"kubernetes.io/projected/278ec7ee-2beb-4a50-8961-b10b5c150d5b-kube-api-access-v4kjx\") pod \"maas-controller-6899cfd859-v7wgq\" (UID: \"278ec7ee-2beb-4a50-8961-b10b5c150d5b\") " pod="opendatahub/maas-controller-6899cfd859-v7wgq"
Sep 07 10:20:33.861280 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:33.861237    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899cfd859-v7wgq"
Sep 07 10:20:34.036325 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:34.036279    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31c373a1-bb08-458c-8096-b961df47e89a-webhook-cert\") pod \"maas-controller-6845bf5f8f-k8qbw\" (UID: \"31c373a1-bb08-458c-8096-b961df47e89a\") " pod="opendatahub/maas-controller-6845bf5f8f-k8qbw"
Sep 07 10:20:34.038658 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:34.038626    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31c373a1-bb08-458c-8096-b961df47e89a-webhook-cert\") pod \"maas-controller-6845bf5f8f-k8qbw\" (UID: \"31c373a1-bb08-458c-8096-b961df47e89a\") " pod="opendatahub/maas-controller-6845bf5f8f-k8qbw"
Sep 07 10:20:34.132403 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:20:34.132368    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278ec7ee_2beb_4a50_8961_b10b5c150d5b.slice/crio-0c58363a3b6c9e62fb6c8cf984d5856509a2ebf21aa50214f4d12126deba0c5c WatchSource:0}: Error finding container 0c58363a3b6c9e62fb6c8cf984d5856509a2ebf21aa50214f4d12126deba0c5c: Status 404 returned error can't find the container with id 0c58363a3b6c9e62fb6c8cf984d5856509a2ebf21aa50214f4d12126deba0c5c
Sep 07 10:20:34.142400 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:34.142375    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6899cfd859-v7wgq"]
Sep 07 10:20:34.280496 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:34.280459    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6845bf5f8f-k8qbw"
Sep 07 10:20:34.386722 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:34.386689    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899cfd859-v7wgq" event={"ID":"278ec7ee-2beb-4a50-8961-b10b5c150d5b","Type":"ContainerStarted","Data":"0c58363a3b6c9e62fb6c8cf984d5856509a2ebf21aa50214f4d12126deba0c5c"}
Sep 07 10:20:34.551980 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:34.551952    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6845bf5f8f-k8qbw"]
Sep 07 10:20:34.553417 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:20:34.553383    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c373a1_bb08_458c_8096_b961df47e89a.slice/crio-fabd8217ae30a3dac7898f840ec8daef4c8bee67199a1e10c1779fd71a779a35 WatchSource:0}: Error finding container fabd8217ae30a3dac7898f840ec8daef4c8bee67199a1e10c1779fd71a779a35: Status 404 returned error can't find the container with id fabd8217ae30a3dac7898f840ec8daef4c8bee67199a1e10c1779fd71a779a35
Sep 07 10:20:35.398579 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:35.398474    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6845bf5f8f-k8qbw" event={"ID":"31c373a1-bb08-458c-8096-b961df47e89a","Type":"ContainerStarted","Data":"fabd8217ae30a3dac7898f840ec8daef4c8bee67199a1e10c1779fd71a779a35"}
Sep 07 10:20:37.411312 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:37.411269    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899cfd859-v7wgq" event={"ID":"278ec7ee-2beb-4a50-8961-b10b5c150d5b","Type":"ContainerStarted","Data":"eb2fa5d8121d8b01e30220e75542f9e4868ead1abe8076f124b443d30d812455"}
Sep 07 10:20:37.411738 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:37.411432    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6899cfd859-v7wgq"
Sep 07 10:20:37.412827 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:37.412807    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6845bf5f8f-k8qbw" event={"ID":"31c373a1-bb08-458c-8096-b961df47e89a","Type":"ContainerStarted","Data":"1fa2edb5dff2ef50951e029d0b27607700a724a37580c474b0d410030fef5df7"}
Sep 07 10:20:37.412970 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:37.412956    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6845bf5f8f-k8qbw"
Sep 07 10:20:37.430102 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:37.430052    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6899cfd859-v7wgq" podStartSLOduration=1.869658922 podStartE2EDuration="4.430038523s" totalImagesPullingTime="2.560379601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:37.428586013 +0000 UTC m=+539.926275942" watchObservedRunningTime="2026-09-07 10:20:37.430038523 +0000 UTC m=+539.927728431"
Sep 07 10:20:37.446869 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:37.446821    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6845bf5f8f-k8qbw" podStartSLOduration=2.296124136 podStartE2EDuration="4.446806232s" totalImagesPullingTime="2.150682096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:37.444681212 +0000 UTC m=+539.942371169" watchObservedRunningTime="2026-09-07 10:20:37.446806232 +0000 UTC m=+539.944496142"
Sep 07 10:20:48.419911 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:48.419881    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6899cfd859-v7wgq"
Sep 07 10:20:48.420396 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:48.419948    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6845bf5f8f-k8qbw"
Sep 07 10:20:48.480328 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:48.480294    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6845bf5f8f-k8qbw"]
Sep 07 10:20:48.480538 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:48.480514    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6845bf5f8f-k8qbw" podUID="31c373a1-bb08-458c-8096-b961df47e89a" containerName="manager" containerID="cri-o://1fa2edb5dff2ef50951e029d0b27607700a724a37580c474b0d410030fef5df7" gracePeriod=30
Sep 07 10:20:48.728021 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:48.727993    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6845bf5f8f-k8qbw"
Sep 07 10:20:48.751804 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:48.751753    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31c373a1-bb08-458c-8096-b961df47e89a-webhook-cert\") pod \"31c373a1-bb08-458c-8096-b961df47e89a\" (UID: \"31c373a1-bb08-458c-8096-b961df47e89a\") "
Sep 07 10:20:48.751976 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:48.751851    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c373a1-bb08-458c-8096-b961df47e89a-metrics-certs\") pod \"31c373a1-bb08-458c-8096-b961df47e89a\" (UID: \"31c373a1-bb08-458c-8096-b961df47e89a\") "
Sep 07 10:20:48.751976 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:48.751891    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qdqx\" (UniqueName: \"kubernetes.io/projected/31c373a1-bb08-458c-8096-b961df47e89a-kube-api-access-4qdqx\") pod \"31c373a1-bb08-458c-8096-b961df47e89a\" (UID: \"31c373a1-bb08-458c-8096-b961df47e89a\") "
Sep 07 10:20:48.753982 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:48.753943    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c373a1-bb08-458c-8096-b961df47e89a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "31c373a1-bb08-458c-8096-b961df47e89a" (UID: "31c373a1-bb08-458c-8096-b961df47e89a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:20:48.753982 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:48.753971    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c373a1-bb08-458c-8096-b961df47e89a-kube-api-access-4qdqx" (OuterVolumeSpecName: "kube-api-access-4qdqx") pod "31c373a1-bb08-458c-8096-b961df47e89a" (UID: "31c373a1-bb08-458c-8096-b961df47e89a"). InnerVolumeSpecName "kube-api-access-4qdqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:20:48.754133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:48.754017    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c373a1-bb08-458c-8096-b961df47e89a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "31c373a1-bb08-458c-8096-b961df47e89a" (UID: "31c373a1-bb08-458c-8096-b961df47e89a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:20:48.853054 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:48.853019    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qdqx\" (UniqueName: \"kubernetes.io/projected/31c373a1-bb08-458c-8096-b961df47e89a-kube-api-access-4qdqx\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:20:48.853054 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:48.853047    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31c373a1-bb08-458c-8096-b961df47e89a-webhook-cert\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:20:48.853054 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:48.853056    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31c373a1-bb08-458c-8096-b961df47e89a-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:20:49.478948 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:49.478906    2562 generic.go:356] "Generic (PLEG): container finished" podID="31c373a1-bb08-458c-8096-b961df47e89a" containerID="1fa2edb5dff2ef50951e029d0b27607700a724a37580c474b0d410030fef5df7" exitCode=0
Sep 07 10:20:49.479344 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:49.478967    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6845bf5f8f-k8qbw"
Sep 07 10:20:49.479344 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:49.478991    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6845bf5f8f-k8qbw" event={"ID":"31c373a1-bb08-458c-8096-b961df47e89a","Type":"ContainerDied","Data":"1fa2edb5dff2ef50951e029d0b27607700a724a37580c474b0d410030fef5df7"}
Sep 07 10:20:49.479344 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:49.479065    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6845bf5f8f-k8qbw" event={"ID":"31c373a1-bb08-458c-8096-b961df47e89a","Type":"ContainerDied","Data":"fabd8217ae30a3dac7898f840ec8daef4c8bee67199a1e10c1779fd71a779a35"}
Sep 07 10:20:49.479344 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:49.479092    2562 scope.go:117] "RemoveContainer" containerID="1fa2edb5dff2ef50951e029d0b27607700a724a37580c474b0d410030fef5df7"
Sep 07 10:20:49.488489 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:49.488319    2562 scope.go:117] "RemoveContainer" containerID="1fa2edb5dff2ef50951e029d0b27607700a724a37580c474b0d410030fef5df7"
Sep 07 10:20:49.488643 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:20:49.488624    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa2edb5dff2ef50951e029d0b27607700a724a37580c474b0d410030fef5df7\": container with ID starting with 1fa2edb5dff2ef50951e029d0b27607700a724a37580c474b0d410030fef5df7 not found: ID does not exist" containerID="1fa2edb5dff2ef50951e029d0b27607700a724a37580c474b0d410030fef5df7"
Sep 07 10:20:49.488689 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:49.488651    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa2edb5dff2ef50951e029d0b27607700a724a37580c474b0d410030fef5df7"} err="failed to get container status \"1fa2edb5dff2ef50951e029d0b27607700a724a37580c474b0d410030fef5df7\": rpc error: code = NotFound desc = could not find container \"1fa2edb5dff2ef50951e029d0b27607700a724a37580c474b0d410030fef5df7\": container with ID starting with 1fa2edb5dff2ef50951e029d0b27607700a724a37580c474b0d410030fef5df7 not found: ID does not exist"
Sep 07 10:20:49.499518 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:49.499488    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6845bf5f8f-k8qbw"]
Sep 07 10:20:49.503131 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:49.503107    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6845bf5f8f-k8qbw"]
Sep 07 10:20:49.964337 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:20:49.964305    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31c373a1-bb08-458c-8096-b961df47e89a" path="/var/lib/kubelet/pods/31c373a1-bb08-458c-8096-b961df47e89a/volumes"
Sep 07 10:21:11.764636 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.764595    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"]
Sep 07 10:21:11.765133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.764958    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c373a1-bb08-458c-8096-b961df47e89a" containerName="manager"
Sep 07 10:21:11.765133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.764971    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c373a1-bb08-458c-8096-b961df47e89a" containerName="manager"
Sep 07 10:21:11.765133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.765039    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="31c373a1-bb08-458c-8096-b961df47e89a" containerName="manager"
Sep 07 10:21:11.772026 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.772006    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:21:11.774553 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.774532    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 07 10:21:11.775376 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.775358    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2ldcm\""
Sep 07 10:21:11.775496 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.775357    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 10:21:11.779940 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.779411    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"]
Sep 07 10:21:11.848111 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.848074    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c60791-4193-4a28-bf52-bb5cbd452f07-metrics-certs\") pod \"maas-api-7596b5dfb4-2hklc\" (UID: \"00c60791-4193-4a28-bf52-bb5cbd452f07\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:21:11.848111 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.848114    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c60791-4193-4a28-bf52-bb5cbd452f07-maas-api-tls\") pod \"maas-api-7596b5dfb4-2hklc\" (UID: \"00c60791-4193-4a28-bf52-bb5cbd452f07\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:21:11.848332 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.848132    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzmbz\" (UniqueName: \"kubernetes.io/projected/00c60791-4193-4a28-bf52-bb5cbd452f07-kube-api-access-hzmbz\") pod \"maas-api-7596b5dfb4-2hklc\" (UID: \"00c60791-4193-4a28-bf52-bb5cbd452f07\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:21:11.848332 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.848213    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c60791-4193-4a28-bf52-bb5cbd452f07-openshift-service-ca\") pod \"maas-api-7596b5dfb4-2hklc\" (UID: \"00c60791-4193-4a28-bf52-bb5cbd452f07\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:21:11.949014 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.948966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c60791-4193-4a28-bf52-bb5cbd452f07-metrics-certs\") pod \"maas-api-7596b5dfb4-2hklc\" (UID: \"00c60791-4193-4a28-bf52-bb5cbd452f07\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:21:11.949175 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.949024    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c60791-4193-4a28-bf52-bb5cbd452f07-maas-api-tls\") pod \"maas-api-7596b5dfb4-2hklc\" (UID: \"00c60791-4193-4a28-bf52-bb5cbd452f07\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:21:11.949175 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.949050    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzmbz\" (UniqueName: \"kubernetes.io/projected/00c60791-4193-4a28-bf52-bb5cbd452f07-kube-api-access-hzmbz\") pod \"maas-api-7596b5dfb4-2hklc\" (UID: \"00c60791-4193-4a28-bf52-bb5cbd452f07\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:21:11.949175 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.949096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c60791-4193-4a28-bf52-bb5cbd452f07-openshift-service-ca\") pod \"maas-api-7596b5dfb4-2hklc\" (UID: \"00c60791-4193-4a28-bf52-bb5cbd452f07\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:21:11.949813 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.949750    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c60791-4193-4a28-bf52-bb5cbd452f07-openshift-service-ca\") pod \"maas-api-7596b5dfb4-2hklc\" (UID: \"00c60791-4193-4a28-bf52-bb5cbd452f07\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:21:11.951507 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.951487    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c60791-4193-4a28-bf52-bb5cbd452f07-maas-api-tls\") pod \"maas-api-7596b5dfb4-2hklc\" (UID: \"00c60791-4193-4a28-bf52-bb5cbd452f07\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:21:11.951572 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.951529    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c60791-4193-4a28-bf52-bb5cbd452f07-metrics-certs\") pod \"maas-api-7596b5dfb4-2hklc\" (UID: \"00c60791-4193-4a28-bf52-bb5cbd452f07\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:21:11.957650 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:11.957623    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzmbz\" (UniqueName: \"kubernetes.io/projected/00c60791-4193-4a28-bf52-bb5cbd452f07-kube-api-access-hzmbz\") pod \"maas-api-7596b5dfb4-2hklc\" (UID: \"00c60791-4193-4a28-bf52-bb5cbd452f07\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:21:12.083326 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:12.083248    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:21:12.343812 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:12.343691    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-btccx"]
Sep 07 10:21:12.347526 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:12.347510    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btccx"
Sep 07 10:21:12.350211 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:12.350187    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 07 10:21:12.358877 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:12.358853    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-btccx"]
Sep 07 10:21:12.363958 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:12.363937    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"]
Sep 07 10:21:12.366690 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:21:12.366666    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c60791_4193_4a28_bf52_bb5cbd452f07.slice/crio-4f73d886d572935829cafedcedfc58c03a24800c01735bfe82f8aab02c669344 WatchSource:0}: Error finding container 4f73d886d572935829cafedcedfc58c03a24800c01735bfe82f8aab02c669344: Status 404 returned error can't find the container with id 4f73d886d572935829cafedcedfc58c03a24800c01735bfe82f8aab02c669344
Sep 07 10:21:12.453023 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:12.452991    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83ac846-f741-4fd9-bf3c-d0733fc9a4c2-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-btccx\" (UID: \"c83ac846-f741-4fd9-bf3c-d0733fc9a4c2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btccx"
Sep 07 10:21:12.453180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:12.453046    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj64t\" (UniqueName: \"kubernetes.io/projected/c83ac846-f741-4fd9-bf3c-d0733fc9a4c2-kube-api-access-qj64t\") pod \"payload-pre-processing-59fbc47bf8-btccx\" (UID: \"c83ac846-f741-4fd9-bf3c-d0733fc9a4c2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btccx"
Sep 07 10:21:12.554200 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:12.554158    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj64t\" (UniqueName: \"kubernetes.io/projected/c83ac846-f741-4fd9-bf3c-d0733fc9a4c2-kube-api-access-qj64t\") pod \"payload-pre-processing-59fbc47bf8-btccx\" (UID: \"c83ac846-f741-4fd9-bf3c-d0733fc9a4c2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btccx"
Sep 07 10:21:12.554399 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:12.554266    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83ac846-f741-4fd9-bf3c-d0733fc9a4c2-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-btccx\" (UID: \"c83ac846-f741-4fd9-bf3c-d0733fc9a4c2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btccx"
Sep 07 10:21:12.554983 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:12.554962    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83ac846-f741-4fd9-bf3c-d0733fc9a4c2-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-btccx\" (UID: \"c83ac846-f741-4fd9-bf3c-d0733fc9a4c2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btccx"
Sep 07 10:21:12.565162 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:12.565135    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj64t\" (UniqueName: \"kubernetes.io/projected/c83ac846-f741-4fd9-bf3c-d0733fc9a4c2-kube-api-access-qj64t\") pod \"payload-pre-processing-59fbc47bf8-btccx\" (UID: \"c83ac846-f741-4fd9-bf3c-d0733fc9a4c2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btccx"
Sep 07 10:21:12.598978 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:12.598902    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc" event={"ID":"00c60791-4193-4a28-bf52-bb5cbd452f07","Type":"ContainerStarted","Data":"4f73d886d572935829cafedcedfc58c03a24800c01735bfe82f8aab02c669344"}
Sep 07 10:21:12.658997 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:12.658966    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btccx"
Sep 07 10:21:12.930378 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:12.930350    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-btccx"]
Sep 07 10:21:12.932492 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:21:12.932466    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83ac846_f741_4fd9_bf3c_d0733fc9a4c2.slice/crio-18b9df3350ca3fe9a427290e0d8ad9e36a8a12c755a3cbb835498528ed6a09fc WatchSource:0}: Error finding container 18b9df3350ca3fe9a427290e0d8ad9e36a8a12c755a3cbb835498528ed6a09fc: Status 404 returned error can't find the container with id 18b9df3350ca3fe9a427290e0d8ad9e36a8a12c755a3cbb835498528ed6a09fc
Sep 07 10:21:13.609108 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:13.609050    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btccx" event={"ID":"c83ac846-f741-4fd9-bf3c-d0733fc9a4c2","Type":"ContainerStarted","Data":"18b9df3350ca3fe9a427290e0d8ad9e36a8a12c755a3cbb835498528ed6a09fc"}
Sep 07 10:21:14.615984 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:14.615952    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc" event={"ID":"00c60791-4193-4a28-bf52-bb5cbd452f07","Type":"ContainerStarted","Data":"3a69d45789d91d8eddc9547062df7fe19c2121606fec5a7cc2c71bc17ed48bf4"}
Sep 07 10:21:14.616361 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:14.616017    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:21:14.633385 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:14.633332    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc" podStartSLOduration=1.8234595759999999 podStartE2EDuration="3.633284598s" totalImagesPullingTime="1.809825022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:14.632547678 +0000 UTC m=+577.130237590" watchObservedRunningTime="2026-09-07 10:21:14.633284598 +0000 UTC m=+577.130974505"
Sep 07 10:21:15.622242 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:15.622202    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btccx" event={"ID":"c83ac846-f741-4fd9-bf3c-d0733fc9a4c2","Type":"ContainerStarted","Data":"8b9aa8a77b85f99126078423df3436c384c6f0d366514780e1aa6610a8d31b91"}
Sep 07 10:21:15.638548 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:15.638494    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btccx" podStartSLOduration=1.973252003 podStartE2EDuration="3.638477404s" totalImagesPullingTime="1.665225401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:15.636536658 +0000 UTC m=+578.134226577" watchObservedRunningTime="2026-09-07 10:21:15.638477404 +0000 UTC m=+578.136167311"
Sep 07 10:21:16.627473 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:16.627440    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btccx"
Sep 07 10:21:20.628135 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:20.628103    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:21:27.635265 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:21:27.635230    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btccx"
Sep 07 10:22:11.888063 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:11.888026    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"]
Sep 07 10:22:11.891524 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:11.891503    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:11.894194 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:11.894158    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 07 10:22:11.894194 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:11.894161    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 07 10:22:11.894364 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:11.894161    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-87q54\""
Sep 07 10:22:11.894364 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:11.894211    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 07 10:22:11.900788 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:11.900742    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"]
Sep 07 10:22:11.973676 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:11.973639    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d7052df-af1d-475e-9f29-151723309ba5-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:11.973676 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:11.973675    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d7052df-af1d-475e-9f29-151723309ba5-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:11.973903 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:11.973713    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d7052df-af1d-475e-9f29-151723309ba5-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:11.973903 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:11.973823    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d7052df-af1d-475e-9f29-151723309ba5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:11.973903 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:11.973859    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d7052df-af1d-475e-9f29-151723309ba5-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:11.973903 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:11.973885    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txdkz\" (UniqueName: \"kubernetes.io/projected/4d7052df-af1d-475e-9f29-151723309ba5-kube-api-access-txdkz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:12.074469 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:12.074430    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d7052df-af1d-475e-9f29-151723309ba5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:12.074469 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:12.074475    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d7052df-af1d-475e-9f29-151723309ba5-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:12.074706 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:12.074501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txdkz\" (UniqueName: \"kubernetes.io/projected/4d7052df-af1d-475e-9f29-151723309ba5-kube-api-access-txdkz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:12.074706 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:12.074562    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d7052df-af1d-475e-9f29-151723309ba5-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:12.074706 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:12.074584    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d7052df-af1d-475e-9f29-151723309ba5-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:12.074706 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:12.074621    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d7052df-af1d-475e-9f29-151723309ba5-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:12.074949 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:12.074915    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d7052df-af1d-475e-9f29-151723309ba5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:12.074994 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:12.074959    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d7052df-af1d-475e-9f29-151723309ba5-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:12.074994 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:12.074975    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d7052df-af1d-475e-9f29-151723309ba5-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:12.076969 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:12.076930    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d7052df-af1d-475e-9f29-151723309ba5-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:12.077144 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:12.077123    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d7052df-af1d-475e-9f29-151723309ba5-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:12.082391 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:12.082369    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txdkz\" (UniqueName: \"kubernetes.io/projected/4d7052df-af1d-475e-9f29-151723309ba5-kube-api-access-txdkz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d\" (UID: \"4d7052df-af1d-475e-9f29-151723309ba5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:12.203094 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:12.203060    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:12.476691 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:12.476662    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"]
Sep 07 10:22:12.478476 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:22:12.478445    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7052df_af1d_475e_9f29_151723309ba5.slice/crio-ebcc8749efcf7d5c9235c8e198e1f66b95af881a2ba69a2c9b8ca75600526c00 WatchSource:0}: Error finding container ebcc8749efcf7d5c9235c8e198e1f66b95af881a2ba69a2c9b8ca75600526c00: Status 404 returned error can't find the container with id ebcc8749efcf7d5c9235c8e198e1f66b95af881a2ba69a2c9b8ca75600526c00
Sep 07 10:22:12.479702 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:12.479683    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 10:22:12.922732 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:12.922647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d" event={"ID":"4d7052df-af1d-475e-9f29-151723309ba5","Type":"ContainerStarted","Data":"ebcc8749efcf7d5c9235c8e198e1f66b95af881a2ba69a2c9b8ca75600526c00"}
Sep 07 10:22:13.930398 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:13.930364    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d" event={"ID":"4d7052df-af1d-475e-9f29-151723309ba5","Type":"ContainerStarted","Data":"8a0e819132d7695d07f407f3d3b8027331a75ec8231f3663bc6a59dc8a0b9907"}
Sep 07 10:22:13.951871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:13.951814    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d" podStartSLOduration=1.98703524 podStartE2EDuration="2.951791821s" totalImagesPullingTime="964.756581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:22:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:22:13.951124301 +0000 UTC m=+636.448814221" watchObservedRunningTime="2026-09-07 10:22:13.951791821 +0000 UTC m=+636.449481752"
Sep 07 10:22:22.203426 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:22.203380    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:22.203860 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:22.203468    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:22.215897 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:22.215868    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:22.989643 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:22.989611    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d"
Sep 07 10:22:40.388392 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.388347    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"]
Sep 07 10:22:40.391006 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.390980    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.393467 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.393444    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 07 10:22:40.402965 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.402940    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"]
Sep 07 10:22:40.526340 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.526303    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.526340 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.526353    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.526554 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.526372    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.526554 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.526452    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.526554 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.526492    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.526554 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.526540    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcvtm\" (UniqueName: \"kubernetes.io/projected/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-kube-api-access-mcvtm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.626959 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.626915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.627146 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.626971    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.627146 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.627020    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcvtm\" (UniqueName: \"kubernetes.io/projected/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-kube-api-access-mcvtm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.627146 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.627063    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.627146 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.627103    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.627146 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.627126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.627407 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.627381    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.627491 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.627427    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.627554 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.627541    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.629454 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.629427    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.629640 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.629620    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.634877 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.634848    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcvtm\" (UniqueName: \"kubernetes.io/projected/ceeff70b-0f4d-4088-ae6a-89d03d9c385f-kube-api-access-mcvtm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c\" (UID: \"ceeff70b-0f4d-4088-ae6a-89d03d9c385f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.702825 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.702756    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:40.973921 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:40.973868    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"]
Sep 07 10:22:40.976088 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:22:40.976055    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceeff70b_0f4d_4088_ae6a_89d03d9c385f.slice/crio-f4b32816a54b34e1005c1f31b4dc5f4cecb02e85a3e03f0a63b2f91f652952cc WatchSource:0}: Error finding container f4b32816a54b34e1005c1f31b4dc5f4cecb02e85a3e03f0a63b2f91f652952cc: Status 404 returned error can't find the container with id f4b32816a54b34e1005c1f31b4dc5f4cecb02e85a3e03f0a63b2f91f652952cc
Sep 07 10:22:41.080496 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:41.080462    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c" event={"ID":"ceeff70b-0f4d-4088-ae6a-89d03d9c385f","Type":"ContainerStarted","Data":"f4b32816a54b34e1005c1f31b4dc5f4cecb02e85a3e03f0a63b2f91f652952cc"}
Sep 07 10:22:42.088187 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:42.088148    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c" event={"ID":"ceeff70b-0f4d-4088-ae6a-89d03d9c385f","Type":"ContainerStarted","Data":"fa8188a431a8bb7d679d7e77eb5818d5a1cb7ded83b96db13480d96a8661e081"}
Sep 07 10:22:42.105885 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:42.105831    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c" podStartSLOduration=1.902297428 podStartE2EDuration="2.105813769s" totalImagesPullingTime="203.516341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:22:42.104152264 +0000 UTC m=+664.601842201" watchObservedRunningTime="2026-09-07 10:22:42.105813769 +0000 UTC m=+664.603503677"
Sep 07 10:22:50.703065 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:50.703018    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:50.703532 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:50.703097    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:50.715504 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:50.715475    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:22:51.149442 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:22:51.149359    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c"
Sep 07 10:23:10.002334 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.002296    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"]
Sep 07 10:23:10.004965 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.004949    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.007577 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.007556    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 07 10:23:10.013704 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.013680    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"]
Sep 07 10:23:10.080935 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.080892    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d654b656-f1c9-4a80-9c62-63df56dac3dd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.081093 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.080965    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq2zq\" (UniqueName: \"kubernetes.io/projected/d654b656-f1c9-4a80-9c62-63df56dac3dd-kube-api-access-lq2zq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.081093 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.081025    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d654b656-f1c9-4a80-9c62-63df56dac3dd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.081093 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.081042    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d654b656-f1c9-4a80-9c62-63df56dac3dd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.081093 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.081075    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d654b656-f1c9-4a80-9c62-63df56dac3dd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.081234 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.081109    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d654b656-f1c9-4a80-9c62-63df56dac3dd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.182449 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.182407    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d654b656-f1c9-4a80-9c62-63df56dac3dd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.182449 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.182454    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d654b656-f1c9-4a80-9c62-63df56dac3dd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.182697 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.182478    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d654b656-f1c9-4a80-9c62-63df56dac3dd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.182697 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.182506    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d654b656-f1c9-4a80-9c62-63df56dac3dd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.182697 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.182581    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d654b656-f1c9-4a80-9c62-63df56dac3dd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.182697 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.182628    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq2zq\" (UniqueName: \"kubernetes.io/projected/d654b656-f1c9-4a80-9c62-63df56dac3dd-kube-api-access-lq2zq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.183100 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.183071    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d654b656-f1c9-4a80-9c62-63df56dac3dd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.183227 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.183100    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d654b656-f1c9-4a80-9c62-63df56dac3dd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.183227 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.183126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d654b656-f1c9-4a80-9c62-63df56dac3dd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.185187 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.185163    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d654b656-f1c9-4a80-9c62-63df56dac3dd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.185303 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.185282    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d654b656-f1c9-4a80-9c62-63df56dac3dd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.190807 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.190754    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq2zq\" (UniqueName: \"kubernetes.io/projected/d654b656-f1c9-4a80-9c62-63df56dac3dd-kube-api-access-lq2zq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vws5s\" (UID: \"d654b656-f1c9-4a80-9c62-63df56dac3dd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.314860 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.314752    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:10.588424 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:10.588387    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"]
Sep 07 10:23:10.590038 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:23:10.590009    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd654b656_f1c9_4a80_9c62_63df56dac3dd.slice/crio-592e8902153da1f3a8d87e701816cdf27b238678d041c29822cfdbbb07b2856e WatchSource:0}: Error finding container 592e8902153da1f3a8d87e701816cdf27b238678d041c29822cfdbbb07b2856e: Status 404 returned error can't find the container with id 592e8902153da1f3a8d87e701816cdf27b238678d041c29822cfdbbb07b2856e
Sep 07 10:23:11.249575 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:11.249537    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s" event={"ID":"d654b656-f1c9-4a80-9c62-63df56dac3dd","Type":"ContainerStarted","Data":"bc6db3476c3a3b121612bebcb7d35bb5727dea32dc4c6fda9493a48ae9eb3e54"}
Sep 07 10:23:11.249575 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:11.249582    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s" event={"ID":"d654b656-f1c9-4a80-9c62-63df56dac3dd","Type":"ContainerStarted","Data":"592e8902153da1f3a8d87e701816cdf27b238678d041c29822cfdbbb07b2856e"}
Sep 07 10:23:11.271024 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:11.270968    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s" podStartSLOduration=2.080225321 podStartE2EDuration="2.270951484s" totalImagesPullingTime="190.726163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:23:11.268791476 +0000 UTC m=+693.766481385" watchObservedRunningTime="2026-09-07 10:23:11.270951484 +0000 UTC m=+693.768641392"
Sep 07 10:23:20.314980 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:20.314932    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:20.315396 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:20.315009    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:20.327747 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:20.327716    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:23:21.312523 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:23:21.312493    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vws5s"
Sep 07 10:24:19.649729 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.649697    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"]
Sep 07 10:24:19.652375 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.652356    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:24:19.661060 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.661040    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-05ecbf-serving-cert\""
Sep 07 10:24:19.679921 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.679878    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"]
Sep 07 10:24:19.683247 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.683217    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bckr\" (UniqueName: \"kubernetes.io/projected/d8c5359b-0c75-4a64-92d4-c73d21bd2058-kube-api-access-2bckr\") pod \"maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt\" (UID: \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:24:19.683382 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.683271    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c5359b-0c75-4a64-92d4-c73d21bd2058-metrics-certs\") pod \"maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt\" (UID: \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:24:19.683382 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.683312    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c5359b-0c75-4a64-92d4-c73d21bd2058-openshift-service-ca\") pod \"maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt\" (UID: \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:24:19.683489 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.683410    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c5359b-0c75-4a64-92d4-c73d21bd2058-maas-api-tls\") pod \"maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt\" (UID: \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:24:19.783993 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.783948    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bckr\" (UniqueName: \"kubernetes.io/projected/d8c5359b-0c75-4a64-92d4-c73d21bd2058-kube-api-access-2bckr\") pod \"maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt\" (UID: \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:24:19.783993 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.783998    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c5359b-0c75-4a64-92d4-c73d21bd2058-metrics-certs\") pod \"maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt\" (UID: \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:24:19.784208 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.784030    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c5359b-0c75-4a64-92d4-c73d21bd2058-openshift-service-ca\") pod \"maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt\" (UID: \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:24:19.784208 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.784081    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c5359b-0c75-4a64-92d4-c73d21bd2058-maas-api-tls\") pod \"maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt\" (UID: \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:24:19.784676 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.784653    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c5359b-0c75-4a64-92d4-c73d21bd2058-openshift-service-ca\") pod \"maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt\" (UID: \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:24:19.786605 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.786582    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c5359b-0c75-4a64-92d4-c73d21bd2058-metrics-certs\") pod \"maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt\" (UID: \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:24:19.786605 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.786599    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c5359b-0c75-4a64-92d4-c73d21bd2058-maas-api-tls\") pod \"maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt\" (UID: \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:24:19.811001 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.810970    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bckr\" (UniqueName: \"kubernetes.io/projected/d8c5359b-0c75-4a64-92d4-c73d21bd2058-kube-api-access-2bckr\") pod \"maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt\" (UID: \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:24:19.961936 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:19.961907    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:24:20.238003 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.237967    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"]
Sep 07 10:24:20.239315 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:24:20.239286    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c5359b_0c75_4a64_92d4_c73d21bd2058.slice/crio-50a6003b06d755577df0d9cd4408edfb1e397b9cd9f6249391eb49f90dcd4522 WatchSource:0}: Error finding container 50a6003b06d755577df0d9cd4408edfb1e397b9cd9f6249391eb49f90dcd4522: Status 404 returned error can't find the container with id 50a6003b06d755577df0d9cd4408edfb1e397b9cd9f6249391eb49f90dcd4522
Sep 07 10:24:20.516313 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.516228    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"]
Sep 07 10:24:20.519610 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.519590    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"
Sep 07 10:24:20.522608 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.522583    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-05ecbf-dockercfg-hklw9\""
Sep 07 10:24:20.522738 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.522582    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-05ecbf\""
Sep 07 10:24:20.539045 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.539018    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"]
Sep 07 10:24:20.551081 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.551053    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"]
Sep 07 10:24:20.553707 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.553689    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"
Sep 07 10:24:20.566858 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.566827    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"]
Sep 07 10:24:20.591648 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.591619    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv7zz\" (UniqueName: \"kubernetes.io/projected/5243336b-9ae7-469c-a2ef-5941f675876c-kube-api-access-fv7zz\") pod \"payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg\" (UID: \"5243336b-9ae7-469c-a2ef-5941f675876c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"
Sep 07 10:24:20.591873 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.591663    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5243336b-9ae7-469c-a2ef-5941f675876c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg\" (UID: \"5243336b-9ae7-469c-a2ef-5941f675876c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"
Sep 07 10:24:20.591873 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.591802    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98cd82-6fd9-4640-9cb4-619e4588e92f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8\" (UID: \"1b98cd82-6fd9-4640-9cb4-619e4588e92f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"
Sep 07 10:24:20.591873 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.591834    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bxgq\" (UniqueName: \"kubernetes.io/projected/1b98cd82-6fd9-4640-9cb4-619e4588e92f-kube-api-access-7bxgq\") pod \"payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8\" (UID: \"1b98cd82-6fd9-4640-9cb4-619e4588e92f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"
Sep 07 10:24:20.617633 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.617596    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt" event={"ID":"d8c5359b-0c75-4a64-92d4-c73d21bd2058","Type":"ContainerStarted","Data":"50a6003b06d755577df0d9cd4408edfb1e397b9cd9f6249391eb49f90dcd4522"}
Sep 07 10:24:20.692988 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.692952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv7zz\" (UniqueName: \"kubernetes.io/projected/5243336b-9ae7-469c-a2ef-5941f675876c-kube-api-access-fv7zz\") pod \"payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg\" (UID: \"5243336b-9ae7-469c-a2ef-5941f675876c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"
Sep 07 10:24:20.692988 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.693000    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5243336b-9ae7-469c-a2ef-5941f675876c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg\" (UID: \"5243336b-9ae7-469c-a2ef-5941f675876c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"
Sep 07 10:24:20.693472 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.693064    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98cd82-6fd9-4640-9cb4-619e4588e92f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8\" (UID: \"1b98cd82-6fd9-4640-9cb4-619e4588e92f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"
Sep 07 10:24:20.693472 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.693094    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bxgq\" (UniqueName: \"kubernetes.io/projected/1b98cd82-6fd9-4640-9cb4-619e4588e92f-kube-api-access-7bxgq\") pod \"payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8\" (UID: \"1b98cd82-6fd9-4640-9cb4-619e4588e92f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"
Sep 07 10:24:20.693828 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.693810    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98cd82-6fd9-4640-9cb4-619e4588e92f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8\" (UID: \"1b98cd82-6fd9-4640-9cb4-619e4588e92f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"
Sep 07 10:24:20.693965 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.693941    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5243336b-9ae7-469c-a2ef-5941f675876c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg\" (UID: \"5243336b-9ae7-469c-a2ef-5941f675876c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"
Sep 07 10:24:20.701652 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.701627    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bxgq\" (UniqueName: \"kubernetes.io/projected/1b98cd82-6fd9-4640-9cb4-619e4588e92f-kube-api-access-7bxgq\") pod \"payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8\" (UID: \"1b98cd82-6fd9-4640-9cb4-619e4588e92f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"
Sep 07 10:24:20.701827 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.701632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv7zz\" (UniqueName: \"kubernetes.io/projected/5243336b-9ae7-469c-a2ef-5941f675876c-kube-api-access-fv7zz\") pod \"payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg\" (UID: \"5243336b-9ae7-469c-a2ef-5941f675876c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"
Sep 07 10:24:20.830790 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.830740    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"
Sep 07 10:24:20.864230 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:20.864085    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"
Sep 07 10:24:21.112476 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:21.112446    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"]
Sep 07 10:24:21.113467 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:24:21.113442    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b98cd82_6fd9_4640_9cb4_619e4588e92f.slice/crio-1ef17b1a52fe185b8d4c776171e63abfd131ded0646d4ff7676e6795d9c0bb51 WatchSource:0}: Error finding container 1ef17b1a52fe185b8d4c776171e63abfd131ded0646d4ff7676e6795d9c0bb51: Status 404 returned error can't find the container with id 1ef17b1a52fe185b8d4c776171e63abfd131ded0646d4ff7676e6795d9c0bb51
Sep 07 10:24:21.147439 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:21.147414    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"]
Sep 07 10:24:21.150026 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:24:21.149834    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5243336b_9ae7_469c_a2ef_5941f675876c.slice/crio-e418de3fb3ca2e3002f928c6b02ac941a17973fbab29fae5f4ecf44f3176aa93 WatchSource:0}: Error finding container e418de3fb3ca2e3002f928c6b02ac941a17973fbab29fae5f4ecf44f3176aa93: Status 404 returned error can't find the container with id e418de3fb3ca2e3002f928c6b02ac941a17973fbab29fae5f4ecf44f3176aa93
Sep 07 10:24:21.625113 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:21.625073    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8" event={"ID":"1b98cd82-6fd9-4640-9cb4-619e4588e92f","Type":"ContainerStarted","Data":"5ded69d26bc4c58d54f4805d0c9ac45ef243d35e602b9a4e17221f956bdc64fc"}
Sep 07 10:24:21.625320 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:21.625137    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8" event={"ID":"1b98cd82-6fd9-4640-9cb4-619e4588e92f","Type":"ContainerStarted","Data":"1ef17b1a52fe185b8d4c776171e63abfd131ded0646d4ff7676e6795d9c0bb51"}
Sep 07 10:24:21.625320 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:21.625266    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"
Sep 07 10:24:21.626656 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:21.626625    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt" event={"ID":"d8c5359b-0c75-4a64-92d4-c73d21bd2058","Type":"ContainerStarted","Data":"7f4c69fba2758846cc61d79c7e76bebd2b5233a0d7f8f3d4a73311a18631791f"}
Sep 07 10:24:21.626797 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:21.626674    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:24:21.628164 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:21.628144    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg" event={"ID":"5243336b-9ae7-469c-a2ef-5941f675876c","Type":"ContainerStarted","Data":"d2072f9d18a17f92ee1f1c348301d29b66cca6d478b8151de4640eba8976b6c1"}
Sep 07 10:24:21.628241 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:21.628178    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg" event={"ID":"5243336b-9ae7-469c-a2ef-5941f675876c","Type":"ContainerStarted","Data":"e418de3fb3ca2e3002f928c6b02ac941a17973fbab29fae5f4ecf44f3176aa93"}
Sep 07 10:24:21.628241 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:21.628208    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"
Sep 07 10:24:21.643214 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:21.643170    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8" podStartSLOduration=1.6431553110000001 podStartE2EDuration="1.643155311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:21.64035721 +0000 UTC m=+764.138047173" watchObservedRunningTime="2026-09-07 10:24:21.643155311 +0000 UTC m=+764.140845219"
Sep 07 10:24:21.656984 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:21.656934    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg" podStartSLOduration=1.656906668 podStartE2EDuration="1.656906668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:21.656512813 +0000 UTC m=+764.154202723" watchObservedRunningTime="2026-09-07 10:24:21.656906668 +0000 UTC m=+764.154596578"
Sep 07 10:24:21.678351 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:21.678301    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt" podStartSLOduration=2.116587868 podStartE2EDuration="2.678282384s" totalImagesPullingTime="561.694516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:21.675877316 +0000 UTC m=+764.173567225" watchObservedRunningTime="2026-09-07 10:24:21.678282384 +0000 UTC m=+764.175972291"
Sep 07 10:24:26.983775 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:26.983728    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"]
Sep 07 10:24:26.986701 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:26.986672    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:26.989047 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:26.989025    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8ec6b6b5-serving-cert\""
Sep 07 10:24:26.998353 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:26.998320    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"]
Sep 07 10:24:27.052147 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.052107    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d183562-1f0f-4cd3-a648-ebe7c58ff476-metrics-certs\") pod \"maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:27.052147 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.052155    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlxx\" (UniqueName: \"kubernetes.io/projected/6d183562-1f0f-4cd3-a648-ebe7c58ff476-kube-api-access-lxlxx\") pod \"maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:27.052382 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.052273    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d183562-1f0f-4cd3-a648-ebe7c58ff476-maas-api-tls\") pod \"maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:27.052382 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.052298    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d183562-1f0f-4cd3-a648-ebe7c58ff476-openshift-service-ca\") pod \"maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:27.153228 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.153190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d183562-1f0f-4cd3-a648-ebe7c58ff476-maas-api-tls\") pod \"maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:27.153228 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.153231    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d183562-1f0f-4cd3-a648-ebe7c58ff476-openshift-service-ca\") pod \"maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:27.153482 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.153272    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d183562-1f0f-4cd3-a648-ebe7c58ff476-metrics-certs\") pod \"maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:27.153482 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.153303    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlxx\" (UniqueName: \"kubernetes.io/projected/6d183562-1f0f-4cd3-a648-ebe7c58ff476-kube-api-access-lxlxx\") pod \"maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:27.153482 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:24:27.153343    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-serving-cert: secret "maas-api-e2e-ait-8ec6b6b5-serving-cert" not found
Sep 07 10:24:27.153482 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:24:27.153422    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d183562-1f0f-4cd3-a648-ebe7c58ff476-maas-api-tls podName:6d183562-1f0f-4cd3-a648-ebe7c58ff476 nodeName:}" failed. No retries permitted until 2026-09-07 10:24:27.653404467 +0000 UTC m=+770.151094353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d183562-1f0f-4cd3-a648-ebe7c58ff476-maas-api-tls") pod "maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m" (UID: "6d183562-1f0f-4cd3-a648-ebe7c58ff476") : secret "maas-api-e2e-ait-8ec6b6b5-serving-cert" not found
Sep 07 10:24:27.154054 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.154023    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d183562-1f0f-4cd3-a648-ebe7c58ff476-openshift-service-ca\") pod \"maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:27.155949 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.155928    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d183562-1f0f-4cd3-a648-ebe7c58ff476-metrics-certs\") pod \"maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:27.162435 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.162410    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlxx\" (UniqueName: \"kubernetes.io/projected/6d183562-1f0f-4cd3-a648-ebe7c58ff476-kube-api-access-lxlxx\") pod \"maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:27.641155 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.640697    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:24:27.657612 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.657574    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d183562-1f0f-4cd3-a648-ebe7c58ff476-maas-api-tls\") pod \"maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:27.660615 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.660583    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d183562-1f0f-4cd3-a648-ebe7c58ff476-maas-api-tls\") pod \"maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:27.746397 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.746364    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"]
Sep 07 10:24:27.749743 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.749721    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"
Sep 07 10:24:27.752301 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.752280    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8ec6b6b5-dockercfg-slnzg\""
Sep 07 10:24:27.752412 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.752356    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8ec6b6b5\""
Sep 07 10:24:27.760875 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.760849    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"]
Sep 07 10:24:27.863131 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.863097    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7666e1ef-91e5-400b-bfff-21101775b442-plugins-config-volume\") pod \"payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl\" (UID: \"7666e1ef-91e5-400b-bfff-21101775b442\") " pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"
Sep 07 10:24:27.863291 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.863164    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqchg\" (UniqueName: \"kubernetes.io/projected/7666e1ef-91e5-400b-bfff-21101775b442-kube-api-access-qqchg\") pod \"payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl\" (UID: \"7666e1ef-91e5-400b-bfff-21101775b442\") " pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"
Sep 07 10:24:27.901154 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.901069    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:27.964441 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.964278    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7666e1ef-91e5-400b-bfff-21101775b442-plugins-config-volume\") pod \"payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl\" (UID: \"7666e1ef-91e5-400b-bfff-21101775b442\") " pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"
Sep 07 10:24:27.964441 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.964399    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqchg\" (UniqueName: \"kubernetes.io/projected/7666e1ef-91e5-400b-bfff-21101775b442-kube-api-access-qqchg\") pod \"payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl\" (UID: \"7666e1ef-91e5-400b-bfff-21101775b442\") " pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"
Sep 07 10:24:27.965734 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.965589    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7666e1ef-91e5-400b-bfff-21101775b442-plugins-config-volume\") pod \"payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl\" (UID: \"7666e1ef-91e5-400b-bfff-21101775b442\") " pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"
Sep 07 10:24:27.973253 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:27.973206    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqchg\" (UniqueName: \"kubernetes.io/projected/7666e1ef-91e5-400b-bfff-21101775b442-kube-api-access-qqchg\") pod \"payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl\" (UID: \"7666e1ef-91e5-400b-bfff-21101775b442\") " pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"
Sep 07 10:24:28.063555 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:28.063523    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"
Sep 07 10:24:28.179055 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:28.179029    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"]
Sep 07 10:24:28.180614 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:24:28.180587    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d183562_1f0f_4cd3_a648_ebe7c58ff476.slice/crio-10631fd73e230bc47ea15104510bb65e0788775f7822de9b364ad2496f0035e6 WatchSource:0}: Error finding container 10631fd73e230bc47ea15104510bb65e0788775f7822de9b364ad2496f0035e6: Status 404 returned error can't find the container with id 10631fd73e230bc47ea15104510bb65e0788775f7822de9b364ad2496f0035e6
Sep 07 10:24:28.341641 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:28.341610    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"]
Sep 07 10:24:28.343259 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:24:28.343229    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7666e1ef_91e5_400b_bfff_21101775b442.slice/crio-3dc10c96901ed06ec02910d70a8f576183f30b2133313bb20814399d3870c2d3 WatchSource:0}: Error finding container 3dc10c96901ed06ec02910d70a8f576183f30b2133313bb20814399d3870c2d3: Status 404 returned error can't find the container with id 3dc10c96901ed06ec02910d70a8f576183f30b2133313bb20814399d3870c2d3
Sep 07 10:24:28.681323 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:28.681284    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m" event={"ID":"6d183562-1f0f-4cd3-a648-ebe7c58ff476","Type":"ContainerStarted","Data":"ef8f0512e17d9c59eae8027927a6c4627d8886f79a78a41924128929c15f4547"}
Sep 07 10:24:28.681508 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:28.681375    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m" event={"ID":"6d183562-1f0f-4cd3-a648-ebe7c58ff476","Type":"ContainerStarted","Data":"10631fd73e230bc47ea15104510bb65e0788775f7822de9b364ad2496f0035e6"}
Sep 07 10:24:28.681508 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:28.681405    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:28.683069 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:28.683035    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl" event={"ID":"7666e1ef-91e5-400b-bfff-21101775b442","Type":"ContainerStarted","Data":"a5728f73889ed148e77c63d1bb7999e54bff5a96b23f82d01392084ce6135cd2"}
Sep 07 10:24:28.683197 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:28.683088    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl" event={"ID":"7666e1ef-91e5-400b-bfff-21101775b442","Type":"ContainerStarted","Data":"3dc10c96901ed06ec02910d70a8f576183f30b2133313bb20814399d3870c2d3"}
Sep 07 10:24:28.683197 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:28.683116    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"
Sep 07 10:24:28.702543 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:28.702500    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m" podStartSLOduration=2.367500399 podStartE2EDuration="2.702487405s" totalImagesPullingTime="334.987006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:28.70037665 +0000 UTC m=+771.198066580" watchObservedRunningTime="2026-09-07 10:24:28.702487405 +0000 UTC m=+771.200177313"
Sep 07 10:24:28.725377 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:28.725324    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl" podStartSLOduration=1.725308315 podStartE2EDuration="1.725308315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:28.723942042 +0000 UTC m=+771.221631988" watchObservedRunningTime="2026-09-07 10:24:28.725308315 +0000 UTC m=+771.222998223"
Sep 07 10:24:32.635109 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:32.635075    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"
Sep 07 10:24:32.635541 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:32.635484    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"
Sep 07 10:24:34.693047 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:34.693017    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:24:39.690311 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:39.690275    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"
Sep 07 10:24:49.692256 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:49.692218    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"]
Sep 07 10:24:49.695205 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:49.695187    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"
Sep 07 10:24:49.698072 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:49.698048    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f17c1f\""
Sep 07 10:24:49.698168 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:49.698048    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f17c1f-dockercfg-xmck7\""
Sep 07 10:24:49.705270 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:49.705243    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"]
Sep 07 10:24:49.764233 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:49.764202    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4596f968-3a01-444a-ab86-94d696af4337-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll\" (UID: \"4596f968-3a01-444a-ab86-94d696af4337\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"
Sep 07 10:24:49.764412 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:49.764312    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzjkb\" (UniqueName: \"kubernetes.io/projected/4596f968-3a01-444a-ab86-94d696af4337-kube-api-access-qzjkb\") pod \"payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll\" (UID: \"4596f968-3a01-444a-ab86-94d696af4337\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"
Sep 07 10:24:49.864813 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:49.864738    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzjkb\" (UniqueName: \"kubernetes.io/projected/4596f968-3a01-444a-ab86-94d696af4337-kube-api-access-qzjkb\") pod \"payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll\" (UID: \"4596f968-3a01-444a-ab86-94d696af4337\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"
Sep 07 10:24:49.864987 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:49.864875    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4596f968-3a01-444a-ab86-94d696af4337-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll\" (UID: \"4596f968-3a01-444a-ab86-94d696af4337\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"
Sep 07 10:24:49.865728 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:49.865710    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4596f968-3a01-444a-ab86-94d696af4337-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll\" (UID: \"4596f968-3a01-444a-ab86-94d696af4337\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"
Sep 07 10:24:49.872447 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:49.872420    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzjkb\" (UniqueName: \"kubernetes.io/projected/4596f968-3a01-444a-ab86-94d696af4337-kube-api-access-qzjkb\") pod \"payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll\" (UID: \"4596f968-3a01-444a-ab86-94d696af4337\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"
Sep 07 10:24:50.006316 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:50.006281    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"
Sep 07 10:24:50.283986 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:50.283953    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"]
Sep 07 10:24:50.286026 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:24:50.285996    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4596f968_3a01_444a_ab86_94d696af4337.slice/crio-61335c2c88a03d1fadf14b30d394c04463aa1c9f53d5de5bcc5f9e822d229962 WatchSource:0}: Error finding container 61335c2c88a03d1fadf14b30d394c04463aa1c9f53d5de5bcc5f9e822d229962: Status 404 returned error can't find the container with id 61335c2c88a03d1fadf14b30d394c04463aa1c9f53d5de5bcc5f9e822d229962
Sep 07 10:24:50.812625 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:50.812590    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll" event={"ID":"4596f968-3a01-444a-ab86-94d696af4337","Type":"ContainerStarted","Data":"49ef17b97521f4c2ae816d819053cfe9d6b73c509c3cee243586d3653a1d67d8"}
Sep 07 10:24:50.813060 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:50.812645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll" event={"ID":"4596f968-3a01-444a-ab86-94d696af4337","Type":"ContainerStarted","Data":"61335c2c88a03d1fadf14b30d394c04463aa1c9f53d5de5bcc5f9e822d229962"}
Sep 07 10:24:50.813060 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:50.812703    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"
Sep 07 10:24:50.835275 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:24:50.835219    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll" podStartSLOduration=1.8352012549999999 podStartE2EDuration="1.835201255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:50.832451844 +0000 UTC m=+793.330141779" watchObservedRunningTime="2026-09-07 10:24:50.835201255 +0000 UTC m=+793.332891164"
Sep 07 10:25:01.821381 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:01.821299    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"
Sep 07 10:25:07.944436 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:07.944395    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"]
Sep 07 10:25:07.944836 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:07.944629    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m" podUID="6d183562-1f0f-4cd3-a648-ebe7c58ff476" containerName="maas-api" containerID="cri-o://ef8f0512e17d9c59eae8027927a6c4627d8886f79a78a41924128929c15f4547" gracePeriod=30
Sep 07 10:25:08.037503 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.037464    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"]
Sep 07 10:25:08.037791 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.037718    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl" podUID="7666e1ef-91e5-400b-bfff-21101775b442" containerName="payload-processing" containerID="cri-o://a5728f73889ed148e77c63d1bb7999e54bff5a96b23f82d01392084ce6135cd2" gracePeriod=30
Sep 07 10:25:08.289075 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.289047    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:25:08.322157 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.322121    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxlxx\" (UniqueName: \"kubernetes.io/projected/6d183562-1f0f-4cd3-a648-ebe7c58ff476-kube-api-access-lxlxx\") pod \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") "
Sep 07 10:25:08.322318 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.322226    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d183562-1f0f-4cd3-a648-ebe7c58ff476-maas-api-tls\") pod \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") "
Sep 07 10:25:08.322318 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.322274    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d183562-1f0f-4cd3-a648-ebe7c58ff476-metrics-certs\") pod \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") "
Sep 07 10:25:08.322318 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.322303    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d183562-1f0f-4cd3-a648-ebe7c58ff476-openshift-service-ca\") pod \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\" (UID: \"6d183562-1f0f-4cd3-a648-ebe7c58ff476\") "
Sep 07 10:25:08.323340 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.322935    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d183562-1f0f-4cd3-a648-ebe7c58ff476-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d183562-1f0f-4cd3-a648-ebe7c58ff476" (UID: "6d183562-1f0f-4cd3-a648-ebe7c58ff476"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:25:08.325949 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.325661    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d183562-1f0f-4cd3-a648-ebe7c58ff476-kube-api-access-lxlxx" (OuterVolumeSpecName: "kube-api-access-lxlxx") pod "6d183562-1f0f-4cd3-a648-ebe7c58ff476" (UID: "6d183562-1f0f-4cd3-a648-ebe7c58ff476"). InnerVolumeSpecName "kube-api-access-lxlxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:25:08.326606 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.326574    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d183562-1f0f-4cd3-a648-ebe7c58ff476-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d183562-1f0f-4cd3-a648-ebe7c58ff476" (UID: "6d183562-1f0f-4cd3-a648-ebe7c58ff476"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:25:08.326755 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.326607    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d183562-1f0f-4cd3-a648-ebe7c58ff476-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6d183562-1f0f-4cd3-a648-ebe7c58ff476" (UID: "6d183562-1f0f-4cd3-a648-ebe7c58ff476"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:25:08.332251 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.332225    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"
Sep 07 10:25:08.423372 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.423344    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqchg\" (UniqueName: \"kubernetes.io/projected/7666e1ef-91e5-400b-bfff-21101775b442-kube-api-access-qqchg\") pod \"7666e1ef-91e5-400b-bfff-21101775b442\" (UID: \"7666e1ef-91e5-400b-bfff-21101775b442\") "
Sep 07 10:25:08.423514 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.423486    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7666e1ef-91e5-400b-bfff-21101775b442-plugins-config-volume\") pod \"7666e1ef-91e5-400b-bfff-21101775b442\" (UID: \"7666e1ef-91e5-400b-bfff-21101775b442\") "
Sep 07 10:25:08.423803 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.423753    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxlxx\" (UniqueName: \"kubernetes.io/projected/6d183562-1f0f-4cd3-a648-ebe7c58ff476-kube-api-access-lxlxx\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:25:08.423985 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.423806    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d183562-1f0f-4cd3-a648-ebe7c58ff476-maas-api-tls\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:25:08.423985 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.423817    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d183562-1f0f-4cd3-a648-ebe7c58ff476-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:25:08.423985 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.423826    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d183562-1f0f-4cd3-a648-ebe7c58ff476-openshift-service-ca\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:25:08.423985 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.423849    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7666e1ef-91e5-400b-bfff-21101775b442-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7666e1ef-91e5-400b-bfff-21101775b442" (UID: "7666e1ef-91e5-400b-bfff-21101775b442"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:25:08.425472 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.425454    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7666e1ef-91e5-400b-bfff-21101775b442-kube-api-access-qqchg" (OuterVolumeSpecName: "kube-api-access-qqchg") pod "7666e1ef-91e5-400b-bfff-21101775b442" (UID: "7666e1ef-91e5-400b-bfff-21101775b442"). InnerVolumeSpecName "kube-api-access-qqchg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:25:08.525037 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.524945    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7666e1ef-91e5-400b-bfff-21101775b442-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:25:08.525037 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.524976    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqchg\" (UniqueName: \"kubernetes.io/projected/7666e1ef-91e5-400b-bfff-21101775b442-kube-api-access-qqchg\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:25:08.916742 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.916643    2562 generic.go:356] "Generic (PLEG): container finished" podID="6d183562-1f0f-4cd3-a648-ebe7c58ff476" containerID="ef8f0512e17d9c59eae8027927a6c4627d8886f79a78a41924128929c15f4547" exitCode=0
Sep 07 10:25:08.916742 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.916692    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"
Sep 07 10:25:08.916742 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.916731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m" event={"ID":"6d183562-1f0f-4cd3-a648-ebe7c58ff476","Type":"ContainerDied","Data":"ef8f0512e17d9c59eae8027927a6c4627d8886f79a78a41924128929c15f4547"}
Sep 07 10:25:08.917041 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.916806    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m" event={"ID":"6d183562-1f0f-4cd3-a648-ebe7c58ff476","Type":"ContainerDied","Data":"10631fd73e230bc47ea15104510bb65e0788775f7822de9b364ad2496f0035e6"}
Sep 07 10:25:08.917041 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.916840    2562 scope.go:117] "RemoveContainer" containerID="ef8f0512e17d9c59eae8027927a6c4627d8886f79a78a41924128929c15f4547"
Sep 07 10:25:08.918390 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.918368    2562 generic.go:356] "Generic (PLEG): container finished" podID="7666e1ef-91e5-400b-bfff-21101775b442" containerID="a5728f73889ed148e77c63d1bb7999e54bff5a96b23f82d01392084ce6135cd2" exitCode=0
Sep 07 10:25:08.918493 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.918421    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl" event={"ID":"7666e1ef-91e5-400b-bfff-21101775b442","Type":"ContainerDied","Data":"a5728f73889ed148e77c63d1bb7999e54bff5a96b23f82d01392084ce6135cd2"}
Sep 07 10:25:08.918493 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.918437    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"
Sep 07 10:25:08.918493 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.918465    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl" event={"ID":"7666e1ef-91e5-400b-bfff-21101775b442","Type":"ContainerDied","Data":"3dc10c96901ed06ec02910d70a8f576183f30b2133313bb20814399d3870c2d3"}
Sep 07 10:25:08.927611 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.927592    2562 scope.go:117] "RemoveContainer" containerID="ef8f0512e17d9c59eae8027927a6c4627d8886f79a78a41924128929c15f4547"
Sep 07 10:25:08.927971 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:25:08.927944    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8f0512e17d9c59eae8027927a6c4627d8886f79a78a41924128929c15f4547\": container with ID starting with ef8f0512e17d9c59eae8027927a6c4627d8886f79a78a41924128929c15f4547 not found: ID does not exist" containerID="ef8f0512e17d9c59eae8027927a6c4627d8886f79a78a41924128929c15f4547"
Sep 07 10:25:08.928025 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.927986    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8f0512e17d9c59eae8027927a6c4627d8886f79a78a41924128929c15f4547"} err="failed to get container status \"ef8f0512e17d9c59eae8027927a6c4627d8886f79a78a41924128929c15f4547\": rpc error: code = NotFound desc = could not find container \"ef8f0512e17d9c59eae8027927a6c4627d8886f79a78a41924128929c15f4547\": container with ID starting with ef8f0512e17d9c59eae8027927a6c4627d8886f79a78a41924128929c15f4547 not found: ID does not exist"
Sep 07 10:25:08.928025 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.928011    2562 scope.go:117] "RemoveContainer" containerID="a5728f73889ed148e77c63d1bb7999e54bff5a96b23f82d01392084ce6135cd2"
Sep 07 10:25:08.937311 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.937289    2562 scope.go:117] "RemoveContainer" containerID="a5728f73889ed148e77c63d1bb7999e54bff5a96b23f82d01392084ce6135cd2"
Sep 07 10:25:08.937657 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:25:08.937636    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5728f73889ed148e77c63d1bb7999e54bff5a96b23f82d01392084ce6135cd2\": container with ID starting with a5728f73889ed148e77c63d1bb7999e54bff5a96b23f82d01392084ce6135cd2 not found: ID does not exist" containerID="a5728f73889ed148e77c63d1bb7999e54bff5a96b23f82d01392084ce6135cd2"
Sep 07 10:25:08.937711 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.937664    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5728f73889ed148e77c63d1bb7999e54bff5a96b23f82d01392084ce6135cd2"} err="failed to get container status \"a5728f73889ed148e77c63d1bb7999e54bff5a96b23f82d01392084ce6135cd2\": rpc error: code = NotFound desc = could not find container \"a5728f73889ed148e77c63d1bb7999e54bff5a96b23f82d01392084ce6135cd2\": container with ID starting with a5728f73889ed148e77c63d1bb7999e54bff5a96b23f82d01392084ce6135cd2 not found: ID does not exist"
Sep 07 10:25:08.942359 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.942337    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"]
Sep 07 10:25:08.947226 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.947201    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8ec6b6b5-6c64cb9cc6-7595m"]
Sep 07 10:25:08.961558 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.961533    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"]
Sep 07 10:25:08.964339 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:08.964319    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ec6b6b5-7559b7b69-b65kl"]
Sep 07 10:25:09.966129 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:09.966093    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d183562-1f0f-4cd3-a648-ebe7c58ff476" path="/var/lib/kubelet/pods/6d183562-1f0f-4cd3-a648-ebe7c58ff476/volumes"
Sep 07 10:25:09.966582 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:09.966553    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7666e1ef-91e5-400b-bfff-21101775b442" path="/var/lib/kubelet/pods/7666e1ef-91e5-400b-bfff-21101775b442/volumes"
Sep 07 10:25:12.415508 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.415470    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"]
Sep 07 10:25:12.416109 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.416087    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7666e1ef-91e5-400b-bfff-21101775b442" containerName="payload-processing"
Sep 07 10:25:12.416109 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.416109    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7666e1ef-91e5-400b-bfff-21101775b442" containerName="payload-processing"
Sep 07 10:25:12.416254 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.416122    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d183562-1f0f-4cd3-a648-ebe7c58ff476" containerName="maas-api"
Sep 07 10:25:12.416254 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.416129    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d183562-1f0f-4cd3-a648-ebe7c58ff476" containerName="maas-api"
Sep 07 10:25:12.416254 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.416245    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7666e1ef-91e5-400b-bfff-21101775b442" containerName="payload-processing"
Sep 07 10:25:12.416459 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.416263    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d183562-1f0f-4cd3-a648-ebe7c58ff476" containerName="maas-api"
Sep 07 10:25:12.420905 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.420885    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:12.423373 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.423351    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-de6561a9-serving-cert\""
Sep 07 10:25:12.430936 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.430912    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"]
Sep 07 10:25:12.561675 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.561640    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/792be84f-5991-45ac-9386-2f8322a75a1d-maas-api-tls\") pod \"maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:12.561902 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.561699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/792be84f-5991-45ac-9386-2f8322a75a1d-metrics-certs\") pod \"maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:12.561902 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.561724    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj5b9\" (UniqueName: \"kubernetes.io/projected/792be84f-5991-45ac-9386-2f8322a75a1d-kube-api-access-gj5b9\") pod \"maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:12.561902 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.561815    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/792be84f-5991-45ac-9386-2f8322a75a1d-openshift-service-ca\") pod \"maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:12.662480 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.662446    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/792be84f-5991-45ac-9386-2f8322a75a1d-metrics-certs\") pod \"maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:12.662480 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.662482    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj5b9\" (UniqueName: \"kubernetes.io/projected/792be84f-5991-45ac-9386-2f8322a75a1d-kube-api-access-gj5b9\") pod \"maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:12.662694 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.662539    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/792be84f-5991-45ac-9386-2f8322a75a1d-openshift-service-ca\") pod \"maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:12.662694 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.662584    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/792be84f-5991-45ac-9386-2f8322a75a1d-maas-api-tls\") pod \"maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:12.662694 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:25:12.662682    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-serving-cert: secret "maas-api-e2e-migrate-de6561a9-serving-cert" not found
Sep 07 10:25:12.662888 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:25:12.662753    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/792be84f-5991-45ac-9386-2f8322a75a1d-maas-api-tls podName:792be84f-5991-45ac-9386-2f8322a75a1d nodeName:}" failed. No retries permitted until 2026-09-07 10:25:13.162733912 +0000 UTC m=+815.660423813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/792be84f-5991-45ac-9386-2f8322a75a1d-maas-api-tls") pod "maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf" (UID: "792be84f-5991-45ac-9386-2f8322a75a1d") : secret "maas-api-e2e-migrate-de6561a9-serving-cert" not found
Sep 07 10:25:12.663277 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.663249    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/792be84f-5991-45ac-9386-2f8322a75a1d-openshift-service-ca\") pod \"maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:12.665159 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.665137    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/792be84f-5991-45ac-9386-2f8322a75a1d-metrics-certs\") pod \"maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:12.671506 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:12.671439    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj5b9\" (UniqueName: \"kubernetes.io/projected/792be84f-5991-45ac-9386-2f8322a75a1d-kube-api-access-gj5b9\") pod \"maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:13.167018 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.166975    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/792be84f-5991-45ac-9386-2f8322a75a1d-maas-api-tls\") pod \"maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:13.169558 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.169533    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/792be84f-5991-45ac-9386-2f8322a75a1d-maas-api-tls\") pod \"maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:13.223393 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.223357    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"]
Sep 07 10:25:13.227242 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.227216    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"
Sep 07 10:25:13.230103 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.230077    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-de6561a9\""
Sep 07 10:25:13.239158 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.239126    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"]
Sep 07 10:25:13.267450 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.267410    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9052a2-90fe-471a-94ad-b1eb79892a48-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl\" (UID: \"ba9052a2-90fe-471a-94ad-b1eb79892a48\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"
Sep 07 10:25:13.267642 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.267486    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxbgb\" (UniqueName: \"kubernetes.io/projected/ba9052a2-90fe-471a-94ad-b1eb79892a48-kube-api-access-rxbgb\") pod \"payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl\" (UID: \"ba9052a2-90fe-471a-94ad-b1eb79892a48\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"
Sep 07 10:25:13.332518 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.332480    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:13.368989 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.368952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9052a2-90fe-471a-94ad-b1eb79892a48-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl\" (UID: \"ba9052a2-90fe-471a-94ad-b1eb79892a48\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"
Sep 07 10:25:13.369146 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.369109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbgb\" (UniqueName: \"kubernetes.io/projected/ba9052a2-90fe-471a-94ad-b1eb79892a48-kube-api-access-rxbgb\") pod \"payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl\" (UID: \"ba9052a2-90fe-471a-94ad-b1eb79892a48\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"
Sep 07 10:25:13.369640 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.369618    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9052a2-90fe-471a-94ad-b1eb79892a48-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl\" (UID: \"ba9052a2-90fe-471a-94ad-b1eb79892a48\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"
Sep 07 10:25:13.378406 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.378375    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxbgb\" (UniqueName: \"kubernetes.io/projected/ba9052a2-90fe-471a-94ad-b1eb79892a48-kube-api-access-rxbgb\") pod \"payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl\" (UID: \"ba9052a2-90fe-471a-94ad-b1eb79892a48\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"
Sep 07 10:25:13.540983 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.540939    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"
Sep 07 10:25:13.630448 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.630415    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"]
Sep 07 10:25:13.631120 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:25:13.631094    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792be84f_5991_45ac_9386_2f8322a75a1d.slice/crio-8ab34beace7a8abeb65be9106fb56909a1c51a3cd8512c8128e43a18d74e5f74 WatchSource:0}: Error finding container 8ab34beace7a8abeb65be9106fb56909a1c51a3cd8512c8128e43a18d74e5f74: Status 404 returned error can't find the container with id 8ab34beace7a8abeb65be9106fb56909a1c51a3cd8512c8128e43a18d74e5f74
Sep 07 10:25:13.813991 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.813950    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"]
Sep 07 10:25:13.814981 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:25:13.814957    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9052a2_90fe_471a_94ad_b1eb79892a48.slice/crio-c6cd0eca0f00a200b2a1c803403df084b93e446e15c9b02fb403ed78978d3d7c WatchSource:0}: Error finding container c6cd0eca0f00a200b2a1c803403df084b93e446e15c9b02fb403ed78978d3d7c: Status 404 returned error can't find the container with id c6cd0eca0f00a200b2a1c803403df084b93e446e15c9b02fb403ed78978d3d7c
Sep 07 10:25:13.951126 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.951088    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf" event={"ID":"792be84f-5991-45ac-9386-2f8322a75a1d","Type":"ContainerStarted","Data":"8ab34beace7a8abeb65be9106fb56909a1c51a3cd8512c8128e43a18d74e5f74"}
Sep 07 10:25:13.953258 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.953227    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl" event={"ID":"ba9052a2-90fe-471a-94ad-b1eb79892a48","Type":"ContainerStarted","Data":"9c816731447c58cee00a0b67eaf061559b084de98919cc76996ac1cff38a97aa"}
Sep 07 10:25:13.953407 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.953280    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl" event={"ID":"ba9052a2-90fe-471a-94ad-b1eb79892a48","Type":"ContainerStarted","Data":"c6cd0eca0f00a200b2a1c803403df084b93e446e15c9b02fb403ed78978d3d7c"}
Sep 07 10:25:13.953502 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.953482    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"
Sep 07 10:25:13.972522 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:13.972473    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl" podStartSLOduration=0.972455261 podStartE2EDuration="972.455261ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:25:13.970254766 +0000 UTC m=+816.467944735" watchObservedRunningTime="2026-09-07 10:25:13.972455261 +0000 UTC m=+816.470145162"
Sep 07 10:25:14.969128 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:14.969083    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf" event={"ID":"792be84f-5991-45ac-9386-2f8322a75a1d","Type":"ContainerStarted","Data":"2d6f11e3ccaee13175360c3f9888b44438a4aef167b8bc82e68af934144fc25a"}
Sep 07 10:25:14.969607 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:14.969398    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:14.988414 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:14.988367    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf" podStartSLOduration=2.591338354 podStartE2EDuration="2.988352944s" totalImagesPullingTime="397.01459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:25:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:25:14.986229557 +0000 UTC m=+817.483919490" watchObservedRunningTime="2026-09-07 10:25:14.988352944 +0000 UTC m=+817.486042853"
Sep 07 10:25:20.980054 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:20.980022    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:24.971165 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:24.971132    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"
Sep 07 10:25:35.396296 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.396256    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"]
Sep 07 10:25:35.400417 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.400397    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.402983 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.402961    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-5bc2bc81-kserve-self-signed-certs\""
Sep 07 10:25:35.407430 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.407399    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"]
Sep 07 10:25:35.573077 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.573037    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-tls-certs\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.573567 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.573094    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-home\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.573567 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.573135    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crkwg\" (UniqueName: \"kubernetes.io/projected/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-kube-api-access-crkwg\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.573567 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.573248    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-dshm\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.573567 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.573277    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-model-cache\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.573567 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.573311    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-tmp-dir\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.674420 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.674300    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-home\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.674420 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.674353    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crkwg\" (UniqueName: \"kubernetes.io/projected/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-kube-api-access-crkwg\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.674664 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.674448    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-dshm\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.674664 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.674474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-model-cache\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.674664 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.674520    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-tmp-dir\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.674664 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.674583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-tls-certs\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.674906 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.674781    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-home\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.674906 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.674885    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-model-cache\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.675019 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.674934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-tmp-dir\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.676657 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.676633    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-dshm\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.676995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.676978    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-tls-certs\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.681948 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.681927    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crkwg\" (UniqueName: \"kubernetes.io/projected/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-kube-api-access-crkwg\") pod \"e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") " pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.711017 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.710972    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:35.985379 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:35.985352    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"]
Sep 07 10:25:35.986023 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:25:35.985995    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c28f4a2_7fea_4c99_9b8c_a9ace6f53acb.slice/crio-79a43a62665cf60f15845512f2283899f1f60fdc3300f23e6c14a48a963f72e0 WatchSource:0}: Error finding container 79a43a62665cf60f15845512f2283899f1f60fdc3300f23e6c14a48a963f72e0: Status 404 returned error can't find the container with id 79a43a62665cf60f15845512f2283899f1f60fdc3300f23e6c14a48a963f72e0
Sep 07 10:25:36.092146 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:36.092108    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464" event={"ID":"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb","Type":"ContainerStarted","Data":"f79a50bdf3fc46530d9686c38088ea19e44935195c42345055b92a03d473baf9"}
Sep 07 10:25:36.092307 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:36.092150    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464" event={"ID":"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb","Type":"ContainerStarted","Data":"79a43a62665cf60f15845512f2283899f1f60fdc3300f23e6c14a48a963f72e0"}
Sep 07 10:25:36.109526 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:36.109459    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464" podStartSLOduration=1.109437634 podStartE2EDuration="1.109437634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:25:36.107265063 +0000 UTC m=+838.604954974" watchObservedRunningTime="2026-09-07 10:25:36.109437634 +0000 UTC m=+838.607127550"
Sep 07 10:25:43.926277 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:43.926237    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"]
Sep 07 10:25:43.926858 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:43.926504    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf" podUID="792be84f-5991-45ac-9386-2f8322a75a1d" containerName="maas-api" containerID="cri-o://2d6f11e3ccaee13175360c3f9888b44438a4aef167b8bc82e68af934144fc25a" gracePeriod=30
Sep 07 10:25:43.989166 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:43.989128    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"]
Sep 07 10:25:43.989426 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:43.989393    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl" podUID="ba9052a2-90fe-471a-94ad-b1eb79892a48" containerName="payload-pre-processing" containerID="cri-o://9c816731447c58cee00a0b67eaf061559b084de98919cc76996ac1cff38a97aa" gracePeriod=30
Sep 07 10:25:44.142013 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.141976    2562 generic.go:356] "Generic (PLEG): container finished" podID="792be84f-5991-45ac-9386-2f8322a75a1d" containerID="2d6f11e3ccaee13175360c3f9888b44438a4aef167b8bc82e68af934144fc25a" exitCode=0
Sep 07 10:25:44.142191 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.142079    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf" event={"ID":"792be84f-5991-45ac-9386-2f8322a75a1d","Type":"ContainerDied","Data":"2d6f11e3ccaee13175360c3f9888b44438a4aef167b8bc82e68af934144fc25a"}
Sep 07 10:25:44.144071 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.144039    2562 generic.go:356] "Generic (PLEG): container finished" podID="ba9052a2-90fe-471a-94ad-b1eb79892a48" containerID="9c816731447c58cee00a0b67eaf061559b084de98919cc76996ac1cff38a97aa" exitCode=0
Sep 07 10:25:44.144150 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.144128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl" event={"ID":"ba9052a2-90fe-471a-94ad-b1eb79892a48","Type":"ContainerDied","Data":"9c816731447c58cee00a0b67eaf061559b084de98919cc76996ac1cff38a97aa"}
Sep 07 10:25:44.251815 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.251745    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:44.296101 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.296066    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"
Sep 07 10:25:44.354219 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.354186    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/792be84f-5991-45ac-9386-2f8322a75a1d-metrics-certs\") pod \"792be84f-5991-45ac-9386-2f8322a75a1d\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") "
Sep 07 10:25:44.354378 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.354276    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj5b9\" (UniqueName: \"kubernetes.io/projected/792be84f-5991-45ac-9386-2f8322a75a1d-kube-api-access-gj5b9\") pod \"792be84f-5991-45ac-9386-2f8322a75a1d\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") "
Sep 07 10:25:44.354378 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.354309    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9052a2-90fe-471a-94ad-b1eb79892a48-plugins-config-volume\") pod \"ba9052a2-90fe-471a-94ad-b1eb79892a48\" (UID: \"ba9052a2-90fe-471a-94ad-b1eb79892a48\") "
Sep 07 10:25:44.354378 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.354351    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/792be84f-5991-45ac-9386-2f8322a75a1d-openshift-service-ca\") pod \"792be84f-5991-45ac-9386-2f8322a75a1d\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") "
Sep 07 10:25:44.354541 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.354376    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxbgb\" (UniqueName: \"kubernetes.io/projected/ba9052a2-90fe-471a-94ad-b1eb79892a48-kube-api-access-rxbgb\") pod \"ba9052a2-90fe-471a-94ad-b1eb79892a48\" (UID: \"ba9052a2-90fe-471a-94ad-b1eb79892a48\") "
Sep 07 10:25:44.354541 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.354407    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/792be84f-5991-45ac-9386-2f8322a75a1d-maas-api-tls\") pod \"792be84f-5991-45ac-9386-2f8322a75a1d\" (UID: \"792be84f-5991-45ac-9386-2f8322a75a1d\") "
Sep 07 10:25:44.354791 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.354728    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792be84f-5991-45ac-9386-2f8322a75a1d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "792be84f-5991-45ac-9386-2f8322a75a1d" (UID: "792be84f-5991-45ac-9386-2f8322a75a1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:25:44.354912 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.354792    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba9052a2-90fe-471a-94ad-b1eb79892a48-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ba9052a2-90fe-471a-94ad-b1eb79892a48" (UID: "ba9052a2-90fe-471a-94ad-b1eb79892a48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:25:44.356593 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.356570    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792be84f-5991-45ac-9386-2f8322a75a1d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "792be84f-5991-45ac-9386-2f8322a75a1d" (UID: "792be84f-5991-45ac-9386-2f8322a75a1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:25:44.356918 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.356888    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9052a2-90fe-471a-94ad-b1eb79892a48-kube-api-access-rxbgb" (OuterVolumeSpecName: "kube-api-access-rxbgb") pod "ba9052a2-90fe-471a-94ad-b1eb79892a48" (UID: "ba9052a2-90fe-471a-94ad-b1eb79892a48"). InnerVolumeSpecName "kube-api-access-rxbgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:25:44.357027 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.356922    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792be84f-5991-45ac-9386-2f8322a75a1d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "792be84f-5991-45ac-9386-2f8322a75a1d" (UID: "792be84f-5991-45ac-9386-2f8322a75a1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:25:44.357027 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.356958    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792be84f-5991-45ac-9386-2f8322a75a1d-kube-api-access-gj5b9" (OuterVolumeSpecName: "kube-api-access-gj5b9") pod "792be84f-5991-45ac-9386-2f8322a75a1d" (UID: "792be84f-5991-45ac-9386-2f8322a75a1d"). InnerVolumeSpecName "kube-api-access-gj5b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:25:44.455994 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.455913    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj5b9\" (UniqueName: \"kubernetes.io/projected/792be84f-5991-45ac-9386-2f8322a75a1d-kube-api-access-gj5b9\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:25:44.455994 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.455944    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9052a2-90fe-471a-94ad-b1eb79892a48-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:25:44.455994 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.455954    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/792be84f-5991-45ac-9386-2f8322a75a1d-openshift-service-ca\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:25:44.455994 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.455964    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxbgb\" (UniqueName: \"kubernetes.io/projected/ba9052a2-90fe-471a-94ad-b1eb79892a48-kube-api-access-rxbgb\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:25:44.455994 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.455973    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/792be84f-5991-45ac-9386-2f8322a75a1d-maas-api-tls\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:25:44.455994 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:44.455981    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/792be84f-5991-45ac-9386-2f8322a75a1d-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:25:45.153026 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:45.152993    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"
Sep 07 10:25:45.153429 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:45.153029    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl" event={"ID":"ba9052a2-90fe-471a-94ad-b1eb79892a48","Type":"ContainerDied","Data":"c6cd0eca0f00a200b2a1c803403df084b93e446e15c9b02fb403ed78978d3d7c"}
Sep 07 10:25:45.153429 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:45.153100    2562 scope.go:117] "RemoveContainer" containerID="9c816731447c58cee00a0b67eaf061559b084de98919cc76996ac1cff38a97aa"
Sep 07 10:25:45.154634 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:45.154608    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf" event={"ID":"792be84f-5991-45ac-9386-2f8322a75a1d","Type":"ContainerDied","Data":"8ab34beace7a8abeb65be9106fb56909a1c51a3cd8512c8128e43a18d74e5f74"}
Sep 07 10:25:45.154755 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:45.154645    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"
Sep 07 10:25:45.164126 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:45.164108    2562 scope.go:117] "RemoveContainer" containerID="2d6f11e3ccaee13175360c3f9888b44438a4aef167b8bc82e68af934144fc25a"
Sep 07 10:25:45.180049 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:45.180021    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"]
Sep 07 10:25:45.183593 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:45.183566    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-de6561a9-8c5c6b9bf-fpfcf"]
Sep 07 10:25:45.193583 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:45.193557    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"]
Sep 07 10:25:45.200451 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:45.200429    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-de6561a9-95996f86b-xqjjl"]
Sep 07 10:25:45.711274 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:45.711241    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:45.711472 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:45.711303    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:45.724421 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:45.724396    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:45.967125 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:45.967039    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="792be84f-5991-45ac-9386-2f8322a75a1d" path="/var/lib/kubelet/pods/792be84f-5991-45ac-9386-2f8322a75a1d/volumes"
Sep 07 10:25:45.967929 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:45.967907    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba9052a2-90fe-471a-94ad-b1eb79892a48" path="/var/lib/kubelet/pods/ba9052a2-90fe-471a-94ad-b1eb79892a48/volumes"
Sep 07 10:25:46.172986 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:46.172957    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:25:48.802539 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.802500    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"]
Sep 07 10:25:48.803994 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.803177    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba9052a2-90fe-471a-94ad-b1eb79892a48" containerName="payload-pre-processing"
Sep 07 10:25:48.804138 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.804000    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba9052a2-90fe-471a-94ad-b1eb79892a48" containerName="payload-pre-processing"
Sep 07 10:25:48.804138 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.804063    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792be84f-5991-45ac-9386-2f8322a75a1d" containerName="maas-api"
Sep 07 10:25:48.804138 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.804073    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="792be84f-5991-45ac-9386-2f8322a75a1d" containerName="maas-api"
Sep 07 10:25:48.804243 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.804189    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="792be84f-5991-45ac-9386-2f8322a75a1d" containerName="maas-api"
Sep 07 10:25:48.804243 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.804200    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba9052a2-90fe-471a-94ad-b1eb79892a48" containerName="payload-pre-processing"
Sep 07 10:25:48.809043 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.809024    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:25:48.812182 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.812157    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-31186eae-serving-cert\""
Sep 07 10:25:48.815470 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.815443    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"]
Sep 07 10:25:48.895232 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.895193    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec9d2b24-fbdc-42cd-8d62-73919e37730b-metrics-certs\") pod \"maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv\" (UID: \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:25:48.895381 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.895300    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd8n4\" (UniqueName: \"kubernetes.io/projected/ec9d2b24-fbdc-42cd-8d62-73919e37730b-kube-api-access-nd8n4\") pod \"maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv\" (UID: \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:25:48.895381 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.895332    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9d2b24-fbdc-42cd-8d62-73919e37730b-maas-api-tls\") pod \"maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv\" (UID: \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:25:48.895476 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.895385    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9d2b24-fbdc-42cd-8d62-73919e37730b-openshift-service-ca\") pod \"maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv\" (UID: \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:25:48.996469 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.996426    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8n4\" (UniqueName: \"kubernetes.io/projected/ec9d2b24-fbdc-42cd-8d62-73919e37730b-kube-api-access-nd8n4\") pod \"maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv\" (UID: \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:25:48.996469 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.996470    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9d2b24-fbdc-42cd-8d62-73919e37730b-maas-api-tls\") pod \"maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv\" (UID: \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:25:48.996698 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.996516    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9d2b24-fbdc-42cd-8d62-73919e37730b-openshift-service-ca\") pod \"maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv\" (UID: \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:25:48.996698 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.996550    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec9d2b24-fbdc-42cd-8d62-73919e37730b-metrics-certs\") pod \"maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv\" (UID: \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:25:48.997319 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.997296    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9d2b24-fbdc-42cd-8d62-73919e37730b-openshift-service-ca\") pod \"maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv\" (UID: \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:25:48.999343 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.999313    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec9d2b24-fbdc-42cd-8d62-73919e37730b-metrics-certs\") pod \"maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv\" (UID: \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:25:48.999460 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:48.999313    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9d2b24-fbdc-42cd-8d62-73919e37730b-maas-api-tls\") pod \"maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv\" (UID: \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:25:49.005187 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:49.005168    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd8n4\" (UniqueName: \"kubernetes.io/projected/ec9d2b24-fbdc-42cd-8d62-73919e37730b-kube-api-access-nd8n4\") pod \"maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv\" (UID: \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:25:49.124375 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:49.124287    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:25:49.404807 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:49.400939    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"]
Sep 07 10:25:49.407472 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:49.407450    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"]
Sep 07 10:25:49.407576 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:49.407555    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"
Sep 07 10:25:49.410502 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:49.410300    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-31186eae-dockercfg-2rp8d\""
Sep 07 10:25:49.411425 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:49.411374    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-31186eae\""
Sep 07 10:25:49.413825 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:49.413802    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"]
Sep 07 10:25:49.501942 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:49.501903    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73c860c-7bf0-4eeb-8822-c19ec6754a75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9\" (UID: \"c73c860c-7bf0-4eeb-8822-c19ec6754a75\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"
Sep 07 10:25:49.502107 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:49.502064    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dmz7\" (UniqueName: \"kubernetes.io/projected/c73c860c-7bf0-4eeb-8822-c19ec6754a75-kube-api-access-4dmz7\") pod \"payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9\" (UID: \"c73c860c-7bf0-4eeb-8822-c19ec6754a75\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"
Sep 07 10:25:49.603602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:49.603195    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dmz7\" (UniqueName: \"kubernetes.io/projected/c73c860c-7bf0-4eeb-8822-c19ec6754a75-kube-api-access-4dmz7\") pod \"payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9\" (UID: \"c73c860c-7bf0-4eeb-8822-c19ec6754a75\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"
Sep 07 10:25:49.603838 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:49.603675    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73c860c-7bf0-4eeb-8822-c19ec6754a75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9\" (UID: \"c73c860c-7bf0-4eeb-8822-c19ec6754a75\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"
Sep 07 10:25:49.604432 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:49.604412    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73c860c-7bf0-4eeb-8822-c19ec6754a75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9\" (UID: \"c73c860c-7bf0-4eeb-8822-c19ec6754a75\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"
Sep 07 10:25:49.610682 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:49.610664    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dmz7\" (UniqueName: \"kubernetes.io/projected/c73c860c-7bf0-4eeb-8822-c19ec6754a75-kube-api-access-4dmz7\") pod \"payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9\" (UID: \"c73c860c-7bf0-4eeb-8822-c19ec6754a75\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"
Sep 07 10:25:49.722793 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:49.722730    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"
Sep 07 10:25:50.001030 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:50.000982    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"]
Sep 07 10:25:50.001374 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:25:50.000984    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73c860c_7bf0_4eeb_8822_c19ec6754a75.slice/crio-3ac78c59d91fc6ff39ecd33fc99c459579bc3da8709ffe525107b31943b7a1cc WatchSource:0}: Error finding container 3ac78c59d91fc6ff39ecd33fc99c459579bc3da8709ffe525107b31943b7a1cc: Status 404 returned error can't find the container with id 3ac78c59d91fc6ff39ecd33fc99c459579bc3da8709ffe525107b31943b7a1cc
Sep 07 10:25:50.190062 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:50.190027    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9" event={"ID":"c73c860c-7bf0-4eeb-8822-c19ec6754a75","Type":"ContainerStarted","Data":"32a76aedbfc00f1e2d7c116d07ece256ec190a4b83327a2fcc84d07bddff5f8d"}
Sep 07 10:25:50.190246 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:50.190084    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9" event={"ID":"c73c860c-7bf0-4eeb-8822-c19ec6754a75","Type":"ContainerStarted","Data":"3ac78c59d91fc6ff39ecd33fc99c459579bc3da8709ffe525107b31943b7a1cc"}
Sep 07 10:25:50.190246 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:50.190108    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"
Sep 07 10:25:50.191817 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:50.191790    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv" event={"ID":"ec9d2b24-fbdc-42cd-8d62-73919e37730b","Type":"ContainerStarted","Data":"c799ffeaf2146441b246a45bbd746817409ec76a8c471d127ddddefb826de3b1"}
Sep 07 10:25:50.191942 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:50.191833    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv" event={"ID":"ec9d2b24-fbdc-42cd-8d62-73919e37730b","Type":"ContainerStarted","Data":"68f156209047f9a8916204bcc8b0acf2cc82d8ab8a50410bd19497073a8b5476"}
Sep 07 10:25:50.191942 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:50.191868    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:25:50.207140 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:50.207095    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9" podStartSLOduration=1.20707909 podStartE2EDuration="1.20707909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:25:50.205119543 +0000 UTC m=+852.702809479" watchObservedRunningTime="2026-09-07 10:25:50.20707909 +0000 UTC m=+852.704768999"
Sep 07 10:25:50.222542 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:50.222495    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv" podStartSLOduration=1.853482394 podStartE2EDuration="2.222475097s" totalImagesPullingTime="368.992703ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:25:50.221134294 +0000 UTC m=+852.718824215" watchObservedRunningTime="2026-09-07 10:25:50.222475097 +0000 UTC m=+852.720165045"
Sep 07 10:25:56.202917 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:25:56.202879    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:26:01.200349 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:01.200318    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"
Sep 07 10:26:10.872031 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:10.871995    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"]
Sep 07 10:26:19.520431 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.520393    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"]
Sep 07 10:26:19.520955 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.520601    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv" podUID="ec9d2b24-fbdc-42cd-8d62-73919e37730b" containerName="maas-api" containerID="cri-o://c799ffeaf2146441b246a45bbd746817409ec76a8c471d127ddddefb826de3b1" gracePeriod=30
Sep 07 10:26:19.669349 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.669314    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"]
Sep 07 10:26:19.669845 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.669806    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9" podUID="c73c860c-7bf0-4eeb-8822-c19ec6754a75" containerName="payload-pre-processing" containerID="cri-o://32a76aedbfc00f1e2d7c116d07ece256ec190a4b83327a2fcc84d07bddff5f8d" gracePeriod=30
Sep 07 10:26:19.799000 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.798972    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:26:19.885285 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.885253    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd8n4\" (UniqueName: \"kubernetes.io/projected/ec9d2b24-fbdc-42cd-8d62-73919e37730b-kube-api-access-nd8n4\") pod \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\" (UID: \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\") "
Sep 07 10:26:19.885443 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.885340    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9d2b24-fbdc-42cd-8d62-73919e37730b-maas-api-tls\") pod \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\" (UID: \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\") "
Sep 07 10:26:19.885443 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.885360    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9d2b24-fbdc-42cd-8d62-73919e37730b-openshift-service-ca\") pod \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\" (UID: \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\") "
Sep 07 10:26:19.885443 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.885404    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec9d2b24-fbdc-42cd-8d62-73919e37730b-metrics-certs\") pod \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\" (UID: \"ec9d2b24-fbdc-42cd-8d62-73919e37730b\") "
Sep 07 10:26:19.885848 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.885813    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9d2b24-fbdc-42cd-8d62-73919e37730b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec9d2b24-fbdc-42cd-8d62-73919e37730b" (UID: "ec9d2b24-fbdc-42cd-8d62-73919e37730b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:26:19.887526 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.887498    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec9d2b24-fbdc-42cd-8d62-73919e37730b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec9d2b24-fbdc-42cd-8d62-73919e37730b" (UID: "ec9d2b24-fbdc-42cd-8d62-73919e37730b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:26:19.887526 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.887505    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9d2b24-fbdc-42cd-8d62-73919e37730b-kube-api-access-nd8n4" (OuterVolumeSpecName: "kube-api-access-nd8n4") pod "ec9d2b24-fbdc-42cd-8d62-73919e37730b" (UID: "ec9d2b24-fbdc-42cd-8d62-73919e37730b"). InnerVolumeSpecName "kube-api-access-nd8n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:26:19.887526 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.887517    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec9d2b24-fbdc-42cd-8d62-73919e37730b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ec9d2b24-fbdc-42cd-8d62-73919e37730b" (UID: "ec9d2b24-fbdc-42cd-8d62-73919e37730b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:26:19.932934 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.932910    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"
Sep 07 10:26:19.986079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.986048    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd8n4\" (UniqueName: \"kubernetes.io/projected/ec9d2b24-fbdc-42cd-8d62-73919e37730b-kube-api-access-nd8n4\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:19.986079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.986076    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9d2b24-fbdc-42cd-8d62-73919e37730b-maas-api-tls\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:19.986270 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.986091    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9d2b24-fbdc-42cd-8d62-73919e37730b-openshift-service-ca\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:19.986270 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:19.986100    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec9d2b24-fbdc-42cd-8d62-73919e37730b-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:20.087430 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.087338    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dmz7\" (UniqueName: \"kubernetes.io/projected/c73c860c-7bf0-4eeb-8822-c19ec6754a75-kube-api-access-4dmz7\") pod \"c73c860c-7bf0-4eeb-8822-c19ec6754a75\" (UID: \"c73c860c-7bf0-4eeb-8822-c19ec6754a75\") "
Sep 07 10:26:20.087576 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.087435    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73c860c-7bf0-4eeb-8822-c19ec6754a75-plugins-config-volume\") pod \"c73c860c-7bf0-4eeb-8822-c19ec6754a75\" (UID: \"c73c860c-7bf0-4eeb-8822-c19ec6754a75\") "
Sep 07 10:26:20.087839 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.087808    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73c860c-7bf0-4eeb-8822-c19ec6754a75-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c73c860c-7bf0-4eeb-8822-c19ec6754a75" (UID: "c73c860c-7bf0-4eeb-8822-c19ec6754a75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:26:20.089479 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.089453    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73c860c-7bf0-4eeb-8822-c19ec6754a75-kube-api-access-4dmz7" (OuterVolumeSpecName: "kube-api-access-4dmz7") pod "c73c860c-7bf0-4eeb-8822-c19ec6754a75" (UID: "c73c860c-7bf0-4eeb-8822-c19ec6754a75"). InnerVolumeSpecName "kube-api-access-4dmz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:26:20.188666 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.188627    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dmz7\" (UniqueName: \"kubernetes.io/projected/c73c860c-7bf0-4eeb-8822-c19ec6754a75-kube-api-access-4dmz7\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:20.188666 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.188663    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73c860c-7bf0-4eeb-8822-c19ec6754a75-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:20.364756 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.364672    2562 generic.go:356] "Generic (PLEG): container finished" podID="c73c860c-7bf0-4eeb-8822-c19ec6754a75" containerID="32a76aedbfc00f1e2d7c116d07ece256ec190a4b83327a2fcc84d07bddff5f8d" exitCode=0
Sep 07 10:26:20.364756 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.364729    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"
Sep 07 10:26:20.364979 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.364753    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9" event={"ID":"c73c860c-7bf0-4eeb-8822-c19ec6754a75","Type":"ContainerDied","Data":"32a76aedbfc00f1e2d7c116d07ece256ec190a4b83327a2fcc84d07bddff5f8d"}
Sep 07 10:26:20.364979 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.364830    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9" event={"ID":"c73c860c-7bf0-4eeb-8822-c19ec6754a75","Type":"ContainerDied","Data":"3ac78c59d91fc6ff39ecd33fc99c459579bc3da8709ffe525107b31943b7a1cc"}
Sep 07 10:26:20.364979 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.364853    2562 scope.go:117] "RemoveContainer" containerID="32a76aedbfc00f1e2d7c116d07ece256ec190a4b83327a2fcc84d07bddff5f8d"
Sep 07 10:26:20.366144 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.366121    2562 generic.go:356] "Generic (PLEG): container finished" podID="ec9d2b24-fbdc-42cd-8d62-73919e37730b" containerID="c799ffeaf2146441b246a45bbd746817409ec76a8c471d127ddddefb826de3b1" exitCode=0
Sep 07 10:26:20.366340 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.366155    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv" event={"ID":"ec9d2b24-fbdc-42cd-8d62-73919e37730b","Type":"ContainerDied","Data":"c799ffeaf2146441b246a45bbd746817409ec76a8c471d127ddddefb826de3b1"}
Sep 07 10:26:20.366340 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.366181    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"
Sep 07 10:26:20.366340 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.366190    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv" event={"ID":"ec9d2b24-fbdc-42cd-8d62-73919e37730b","Type":"ContainerDied","Data":"68f156209047f9a8916204bcc8b0acf2cc82d8ab8a50410bd19497073a8b5476"}
Sep 07 10:26:20.374009 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.373980    2562 scope.go:117] "RemoveContainer" containerID="32a76aedbfc00f1e2d7c116d07ece256ec190a4b83327a2fcc84d07bddff5f8d"
Sep 07 10:26:20.374311 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:26:20.374288    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a76aedbfc00f1e2d7c116d07ece256ec190a4b83327a2fcc84d07bddff5f8d\": container with ID starting with 32a76aedbfc00f1e2d7c116d07ece256ec190a4b83327a2fcc84d07bddff5f8d not found: ID does not exist" containerID="32a76aedbfc00f1e2d7c116d07ece256ec190a4b83327a2fcc84d07bddff5f8d"
Sep 07 10:26:20.374407 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.374322    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a76aedbfc00f1e2d7c116d07ece256ec190a4b83327a2fcc84d07bddff5f8d"} err="failed to get container status \"32a76aedbfc00f1e2d7c116d07ece256ec190a4b83327a2fcc84d07bddff5f8d\": rpc error: code = NotFound desc = could not find container \"32a76aedbfc00f1e2d7c116d07ece256ec190a4b83327a2fcc84d07bddff5f8d\": container with ID starting with 32a76aedbfc00f1e2d7c116d07ece256ec190a4b83327a2fcc84d07bddff5f8d not found: ID does not exist"
Sep 07 10:26:20.374407 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.374344    2562 scope.go:117] "RemoveContainer" containerID="c799ffeaf2146441b246a45bbd746817409ec76a8c471d127ddddefb826de3b1"
Sep 07 10:26:20.382872 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.382848    2562 scope.go:117] "RemoveContainer" containerID="c799ffeaf2146441b246a45bbd746817409ec76a8c471d127ddddefb826de3b1"
Sep 07 10:26:20.383207 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:26:20.383181    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c799ffeaf2146441b246a45bbd746817409ec76a8c471d127ddddefb826de3b1\": container with ID starting with c799ffeaf2146441b246a45bbd746817409ec76a8c471d127ddddefb826de3b1 not found: ID does not exist" containerID="c799ffeaf2146441b246a45bbd746817409ec76a8c471d127ddddefb826de3b1"
Sep 07 10:26:20.383278 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.383218    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c799ffeaf2146441b246a45bbd746817409ec76a8c471d127ddddefb826de3b1"} err="failed to get container status \"c799ffeaf2146441b246a45bbd746817409ec76a8c471d127ddddefb826de3b1\": rpc error: code = NotFound desc = could not find container \"c799ffeaf2146441b246a45bbd746817409ec76a8c471d127ddddefb826de3b1\": container with ID starting with c799ffeaf2146441b246a45bbd746817409ec76a8c471d127ddddefb826de3b1 not found: ID does not exist"
Sep 07 10:26:20.384858 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.384838    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"]
Sep 07 10:26:20.390255 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.390232    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-31186eae-d4657bf9f-vwdmv"]
Sep 07 10:26:20.400701 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.400672    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"]
Sep 07 10:26:20.403020 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:20.403002    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-31186eae-5575bb6978-2hzt9"]
Sep 07 10:26:21.964754 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:21.964725    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c73c860c-7bf0-4eeb-8822-c19ec6754a75" path="/var/lib/kubelet/pods/c73c860c-7bf0-4eeb-8822-c19ec6754a75/volumes"
Sep 07 10:26:21.965194 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:21.965169    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec9d2b24-fbdc-42cd-8d62-73919e37730b" path="/var/lib/kubelet/pods/ec9d2b24-fbdc-42cd-8d62-73919e37730b/volumes"
Sep 07 10:26:24.899886 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:24.899852    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"]
Sep 07 10:26:24.900536 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:24.900508    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73c860c-7bf0-4eeb-8822-c19ec6754a75" containerName="payload-pre-processing"
Sep 07 10:26:24.900536 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:24.900536    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73c860c-7bf0-4eeb-8822-c19ec6754a75" containerName="payload-pre-processing"
Sep 07 10:26:24.900701 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:24.900577    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec9d2b24-fbdc-42cd-8d62-73919e37730b" containerName="maas-api"
Sep 07 10:26:24.900701 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:24.900585    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec9d2b24-fbdc-42cd-8d62-73919e37730b" containerName="maas-api"
Sep 07 10:26:24.900701 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:24.900677    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec9d2b24-fbdc-42cd-8d62-73919e37730b" containerName="maas-api"
Sep 07 10:26:24.900701 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:24.900690    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73c860c-7bf0-4eeb-8822-c19ec6754a75" containerName="payload-pre-processing"
Sep 07 10:26:24.905794 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:24.905757    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:24.908289 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:24.908269    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-39223a0b-serving-cert\""
Sep 07 10:26:24.913825 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:24.913235    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"]
Sep 07 10:26:24.938882 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:24.938588    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0de2707-a7dd-47d5-88da-0e774363d983-openshift-service-ca\") pod \"maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:24.938882 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:24.938643    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j85n8\" (UniqueName: \"kubernetes.io/projected/e0de2707-a7dd-47d5-88da-0e774363d983-kube-api-access-j85n8\") pod \"maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:24.938882 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:24.938678    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-maas-api-tls\") pod \"maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:24.939330 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:24.939225    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-metrics-certs\") pod \"maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:25.039738 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.039692    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-metrics-certs\") pod \"maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:25.039954 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.039754    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0de2707-a7dd-47d5-88da-0e774363d983-openshift-service-ca\") pod \"maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:25.039954 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.039801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j85n8\" (UniqueName: \"kubernetes.io/projected/e0de2707-a7dd-47d5-88da-0e774363d983-kube-api-access-j85n8\") pod \"maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:25.039954 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.039835    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-maas-api-tls\") pod \"maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:25.040168 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:26:25.040117    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-serving-cert: secret "maas-api-e2e-derive-39223a0b-serving-cert" not found
Sep 07 10:26:25.040248 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:26:25.040224    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-maas-api-tls podName:e0de2707-a7dd-47d5-88da-0e774363d983 nodeName:}" failed. No retries permitted until 2026-09-07 10:26:25.540202251 +0000 UTC m=+888.037892138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-maas-api-tls") pod "maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t" (UID: "e0de2707-a7dd-47d5-88da-0e774363d983") : secret "maas-api-e2e-derive-39223a0b-serving-cert" not found
Sep 07 10:26:25.041118 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.041095    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0de2707-a7dd-47d5-88da-0e774363d983-openshift-service-ca\") pod \"maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:25.042656 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.042632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-metrics-certs\") pod \"maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:25.051010 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.050986    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j85n8\" (UniqueName: \"kubernetes.io/projected/e0de2707-a7dd-47d5-88da-0e774363d983-kube-api-access-j85n8\") pod \"maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:25.542459 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.542422    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"]
Sep 07 10:26:25.546392 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.546365    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"
Sep 07 10:26:25.549173 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.549136    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-39223a0b-dockercfg-z96jt\""
Sep 07 10:26:25.549173 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.549146    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-39223a0b\""
Sep 07 10:26:25.553051 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.553022    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-maas-api-tls\") pod \"maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:25.553309 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:26:25.553280    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-serving-cert: secret "maas-api-e2e-derive-39223a0b-serving-cert" not found
Sep 07 10:26:25.553399 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:26:25.553359    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-maas-api-tls podName:e0de2707-a7dd-47d5-88da-0e774363d983 nodeName:}" failed. No retries permitted until 2026-09-07 10:26:26.55333941 +0000 UTC m=+889.051029311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-maas-api-tls") pod "maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t" (UID: "e0de2707-a7dd-47d5-88da-0e774363d983") : secret "maas-api-e2e-derive-39223a0b-serving-cert" not found
Sep 07 10:26:25.557432 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.557409    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"]
Sep 07 10:26:25.654094 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.654049    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490b4bce-9bc5-4215-afad-569e6305fdfa-plugins-config-volume\") pod \"payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c\" (UID: \"490b4bce-9bc5-4215-afad-569e6305fdfa\") " pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"
Sep 07 10:26:25.654303 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.654239    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxx67\" (UniqueName: \"kubernetes.io/projected/490b4bce-9bc5-4215-afad-569e6305fdfa-kube-api-access-fxx67\") pod \"payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c\" (UID: \"490b4bce-9bc5-4215-afad-569e6305fdfa\") " pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"
Sep 07 10:26:25.755249 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.755204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxx67\" (UniqueName: \"kubernetes.io/projected/490b4bce-9bc5-4215-afad-569e6305fdfa-kube-api-access-fxx67\") pod \"payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c\" (UID: \"490b4bce-9bc5-4215-afad-569e6305fdfa\") " pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"
Sep 07 10:26:25.755469 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.755268    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490b4bce-9bc5-4215-afad-569e6305fdfa-plugins-config-volume\") pod \"payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c\" (UID: \"490b4bce-9bc5-4215-afad-569e6305fdfa\") " pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"
Sep 07 10:26:25.756029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.756007    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490b4bce-9bc5-4215-afad-569e6305fdfa-plugins-config-volume\") pod \"payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c\" (UID: \"490b4bce-9bc5-4215-afad-569e6305fdfa\") " pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"
Sep 07 10:26:25.764999 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.764957    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxx67\" (UniqueName: \"kubernetes.io/projected/490b4bce-9bc5-4215-afad-569e6305fdfa-kube-api-access-fxx67\") pod \"payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c\" (UID: \"490b4bce-9bc5-4215-afad-569e6305fdfa\") " pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"
Sep 07 10:26:25.860561 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.860474    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"
Sep 07 10:26:25.893504 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:25.893462    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464" podUID="2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb" containerName="main" containerID="cri-o://f79a50bdf3fc46530d9686c38088ea19e44935195c42345055b92a03d473baf9" gracePeriod=45
Sep 07 10:26:26.138001 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.137973    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"]
Sep 07 10:26:26.144723 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:26:26.144688    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490b4bce_9bc5_4215_afad_569e6305fdfa.slice/crio-f4f66c2e679146b9a21d89973cec963daf9f78e15a3e39667999e5467aa2bcbe WatchSource:0}: Error finding container f4f66c2e679146b9a21d89973cec963daf9f78e15a3e39667999e5467aa2bcbe: Status 404 returned error can't find the container with id f4f66c2e679146b9a21d89973cec963daf9f78e15a3e39667999e5467aa2bcbe
Sep 07 10:26:26.160473 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.160448    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:26:26.260355 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.260318    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-dshm\") pod \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") "
Sep 07 10:26:26.260570 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.260372    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-tmp-dir\") pod \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") "
Sep 07 10:26:26.260570 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.260465    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-home\") pod \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") "
Sep 07 10:26:26.260570 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.260503    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-tls-certs\") pod \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") "
Sep 07 10:26:26.260570 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.260547    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-model-cache\") pod \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") "
Sep 07 10:26:26.260792 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.260596    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crkwg\" (UniqueName: \"kubernetes.io/projected/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-kube-api-access-crkwg\") pod \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\" (UID: \"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb\") "
Sep 07 10:26:26.260792 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.260699    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb" (UID: "2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:26:26.260792 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.260721    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-home" (OuterVolumeSpecName: "home") pod "2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb" (UID: "2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:26:26.260976 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.260938    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb" (UID: "2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:26:26.260976 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.260947    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-tmp-dir\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:26.261092 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.260996    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-home\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:26.263155 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.263126    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-dshm" (OuterVolumeSpecName: "dshm") pod "2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb" (UID: "2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:26:26.263295 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.263229    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb" (UID: "2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:26:26.263295 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.263242    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-kube-api-access-crkwg" (OuterVolumeSpecName: "kube-api-access-crkwg") pod "2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb" (UID: "2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb"). InnerVolumeSpecName "kube-api-access-crkwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:26:26.362378 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.362334    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-tls-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:26.362378 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.362375    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-model-cache\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:26.362586 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.362389    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crkwg\" (UniqueName: \"kubernetes.io/projected/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-kube-api-access-crkwg\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:26.362586 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.362402    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb-dshm\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:26.405804 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.405693    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c" event={"ID":"490b4bce-9bc5-4215-afad-569e6305fdfa","Type":"ContainerStarted","Data":"b6a9302b9c1d8ad655afdb4800b9329a58b3b995bdc6157ff30ed34cea9f3559"}
Sep 07 10:26:26.405804 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.405758    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c" event={"ID":"490b4bce-9bc5-4215-afad-569e6305fdfa","Type":"ContainerStarted","Data":"f4f66c2e679146b9a21d89973cec963daf9f78e15a3e39667999e5467aa2bcbe"}
Sep 07 10:26:26.406038 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.405878    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"
Sep 07 10:26:26.407171 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.407148    2562 generic.go:356] "Generic (PLEG): container finished" podID="2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb" containerID="f79a50bdf3fc46530d9686c38088ea19e44935195c42345055b92a03d473baf9" exitCode=0
Sep 07 10:26:26.407297 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.407193    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"
Sep 07 10:26:26.407297 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.407228    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464" event={"ID":"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb","Type":"ContainerDied","Data":"f79a50bdf3fc46530d9686c38088ea19e44935195c42345055b92a03d473baf9"}
Sep 07 10:26:26.407297 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.407266    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464" event={"ID":"2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb","Type":"ContainerDied","Data":"79a43a62665cf60f15845512f2283899f1f60fdc3300f23e6c14a48a963f72e0"}
Sep 07 10:26:26.407297 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.407282    2562 scope.go:117] "RemoveContainer" containerID="f79a50bdf3fc46530d9686c38088ea19e44935195c42345055b92a03d473baf9"
Sep 07 10:26:26.416994 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.416975    2562 scope.go:117] "RemoveContainer" containerID="f79a50bdf3fc46530d9686c38088ea19e44935195c42345055b92a03d473baf9"
Sep 07 10:26:26.417316 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:26:26.417296    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f79a50bdf3fc46530d9686c38088ea19e44935195c42345055b92a03d473baf9\": container with ID starting with f79a50bdf3fc46530d9686c38088ea19e44935195c42345055b92a03d473baf9 not found: ID does not exist" containerID="f79a50bdf3fc46530d9686c38088ea19e44935195c42345055b92a03d473baf9"
Sep 07 10:26:26.417386 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.417328    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f79a50bdf3fc46530d9686c38088ea19e44935195c42345055b92a03d473baf9"} err="failed to get container status \"f79a50bdf3fc46530d9686c38088ea19e44935195c42345055b92a03d473baf9\": rpc error: code = NotFound desc = could not find container \"f79a50bdf3fc46530d9686c38088ea19e44935195c42345055b92a03d473baf9\": container with ID starting with f79a50bdf3fc46530d9686c38088ea19e44935195c42345055b92a03d473baf9 not found: ID does not exist"
Sep 07 10:26:26.423897 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.423856    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c" podStartSLOduration=1.423840849 podStartE2EDuration="1.423840849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:26:26.422558645 +0000 UTC m=+888.920248584" watchObservedRunningTime="2026-09-07 10:26:26.423840849 +0000 UTC m=+888.921530757"
Sep 07 10:26:26.439090 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.438976    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"]
Sep 07 10:26:26.442325 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.442295    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-5bc2bc81-kserve-566b76df8f-bn464"]
Sep 07 10:26:26.564414 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.564375    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-maas-api-tls\") pod \"maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:26.566871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.566849    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-maas-api-tls\") pod \"maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:26.740206 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:26.740160    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:27.020263 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:27.020235    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"]
Sep 07 10:26:27.021986 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:26:27.021957    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0de2707_a7dd_47d5_88da_0e774363d983.slice/crio-13402a97265af20b6ccfb0682f6b1039796fed8fe564d307dbe1723eef1d1db5 WatchSource:0}: Error finding container 13402a97265af20b6ccfb0682f6b1039796fed8fe564d307dbe1723eef1d1db5: Status 404 returned error can't find the container with id 13402a97265af20b6ccfb0682f6b1039796fed8fe564d307dbe1723eef1d1db5
Sep 07 10:26:27.414884 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:27.414848    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t" event={"ID":"e0de2707-a7dd-47d5-88da-0e774363d983","Type":"ContainerStarted","Data":"13402a97265af20b6ccfb0682f6b1039796fed8fe564d307dbe1723eef1d1db5"}
Sep 07 10:26:27.965386 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:27.965354    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb" path="/var/lib/kubelet/pods/2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb/volumes"
Sep 07 10:26:28.427641 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:28.427542    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t" event={"ID":"e0de2707-a7dd-47d5-88da-0e774363d983","Type":"ContainerStarted","Data":"1c846f5bd389b06017ad06fbe3bd42c9a766c9fd2f5587ccc1d95a2f4958fb67"}
Sep 07 10:26:28.428127 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:28.427682    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:28.445832 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:28.445773    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t" podStartSLOduration=4.094197174 podStartE2EDuration="4.445742823s" totalImagesPullingTime="351.545649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:26:28.443188384 +0000 UTC m=+890.940878324" watchObservedRunningTime="2026-09-07 10:26:28.445742823 +0000 UTC m=+890.943432830"
Sep 07 10:26:34.440440 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:34.440362    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:36.177237 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:36.177201    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"]
Sep 07 10:26:36.177677 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:36.177663    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb" containerName="main"
Sep 07 10:26:36.177677 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:36.177678    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb" containerName="main"
Sep 07 10:26:36.177791 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:36.177737    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c28f4a2-7fea-4c99-9b8c-a9ace6f53acb" containerName="main"
Sep 07 10:26:36.180844 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:36.180824    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"
Sep 07 10:26:36.184443 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:36.184425    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-81af70-dockercfg-rkls2\""
Sep 07 10:26:36.184544 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:36.184468    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-81af70\""
Sep 07 10:26:36.198824 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:36.198789    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"]
Sep 07 10:26:36.255819 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:36.255786    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c120adf4-3e62-425e-bd26-e5e3f784653a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb\" (UID: \"c120adf4-3e62-425e-bd26-e5e3f784653a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"
Sep 07 10:26:36.256013 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:36.255859    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzcnz\" (UniqueName: \"kubernetes.io/projected/c120adf4-3e62-425e-bd26-e5e3f784653a-kube-api-access-xzcnz\") pod \"payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb\" (UID: \"c120adf4-3e62-425e-bd26-e5e3f784653a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"
Sep 07 10:26:36.357057 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:36.357015    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c120adf4-3e62-425e-bd26-e5e3f784653a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb\" (UID: \"c120adf4-3e62-425e-bd26-e5e3f784653a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"
Sep 07 10:26:36.357250 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:36.357090    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzcnz\" (UniqueName: \"kubernetes.io/projected/c120adf4-3e62-425e-bd26-e5e3f784653a-kube-api-access-xzcnz\") pod \"payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb\" (UID: \"c120adf4-3e62-425e-bd26-e5e3f784653a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"
Sep 07 10:26:36.357827 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:36.357804    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c120adf4-3e62-425e-bd26-e5e3f784653a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb\" (UID: \"c120adf4-3e62-425e-bd26-e5e3f784653a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"
Sep 07 10:26:36.365111 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:36.365087    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzcnz\" (UniqueName: \"kubernetes.io/projected/c120adf4-3e62-425e-bd26-e5e3f784653a-kube-api-access-xzcnz\") pod \"payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb\" (UID: \"c120adf4-3e62-425e-bd26-e5e3f784653a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"
Sep 07 10:26:36.490621 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:36.490581    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"
Sep 07 10:26:36.769338 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:36.769312    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"]
Sep 07 10:26:36.771755 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:26:36.771728    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc120adf4_3e62_425e_bd26_e5e3f784653a.slice/crio-ff67da70347217680db2755527ea5107cc0b76288bd5838ee4255e045ae51840 WatchSource:0}: Error finding container ff67da70347217680db2755527ea5107cc0b76288bd5838ee4255e045ae51840: Status 404 returned error can't find the container with id ff67da70347217680db2755527ea5107cc0b76288bd5838ee4255e045ae51840
Sep 07 10:26:37.417287 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:37.417257    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"
Sep 07 10:26:37.492833 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:37.492798    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb" event={"ID":"c120adf4-3e62-425e-bd26-e5e3f784653a","Type":"ContainerStarted","Data":"431f94f8db50aae5c3086c21641fa16ec17d953cf079db1aeafc7cd4b9e0a59d"}
Sep 07 10:26:37.492995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:37.492856    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb" event={"ID":"c120adf4-3e62-425e-bd26-e5e3f784653a","Type":"ContainerStarted","Data":"ff67da70347217680db2755527ea5107cc0b76288bd5838ee4255e045ae51840"}
Sep 07 10:26:37.492995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:37.492878    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"
Sep 07 10:26:37.509694 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:37.509632    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb" podStartSLOduration=1.5096153490000002 podStartE2EDuration="1.509615349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:26:37.507352228 +0000 UTC m=+900.005042163" watchObservedRunningTime="2026-09-07 10:26:37.509615349 +0000 UTC m=+900.007305324"
Sep 07 10:26:48.500972 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:48.500933    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"
Sep 07 10:26:55.517976 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:55.517937    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"]
Sep 07 10:26:55.518421 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:55.518204    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t" podUID="e0de2707-a7dd-47d5-88da-0e774363d983" containerName="maas-api" containerID="cri-o://1c846f5bd389b06017ad06fbe3bd42c9a766c9fd2f5587ccc1d95a2f4958fb67" gracePeriod=30
Sep 07 10:26:55.575805 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:55.575758    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"]
Sep 07 10:26:55.576005 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:55.575980    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c" podUID="490b4bce-9bc5-4215-afad-569e6305fdfa" containerName="payload-processing" containerID="cri-o://b6a9302b9c1d8ad655afdb4800b9329a58b3b995bdc6157ff30ed34cea9f3559" gracePeriod=30
Sep 07 10:26:55.828700 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:55.828675    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:55.864960 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:55.864928    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"
Sep 07 10:26:55.942139 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:55.942105    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-metrics-certs\") pod \"e0de2707-a7dd-47d5-88da-0e774363d983\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") "
Sep 07 10:26:55.942325 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:55.942150    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j85n8\" (UniqueName: \"kubernetes.io/projected/e0de2707-a7dd-47d5-88da-0e774363d983-kube-api-access-j85n8\") pod \"e0de2707-a7dd-47d5-88da-0e774363d983\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") "
Sep 07 10:26:55.942325 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:55.942202    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-maas-api-tls\") pod \"e0de2707-a7dd-47d5-88da-0e774363d983\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") "
Sep 07 10:26:55.942325 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:55.942316    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0de2707-a7dd-47d5-88da-0e774363d983-openshift-service-ca\") pod \"e0de2707-a7dd-47d5-88da-0e774363d983\" (UID: \"e0de2707-a7dd-47d5-88da-0e774363d983\") "
Sep 07 10:26:55.942710 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:55.942679    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0de2707-a7dd-47d5-88da-0e774363d983-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e0de2707-a7dd-47d5-88da-0e774363d983" (UID: "e0de2707-a7dd-47d5-88da-0e774363d983"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:26:55.944255 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:55.944228    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e0de2707-a7dd-47d5-88da-0e774363d983" (UID: "e0de2707-a7dd-47d5-88da-0e774363d983"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:26:55.944340 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:55.944277    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e0de2707-a7dd-47d5-88da-0e774363d983" (UID: "e0de2707-a7dd-47d5-88da-0e774363d983"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:26:55.944380 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:55.944346    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0de2707-a7dd-47d5-88da-0e774363d983-kube-api-access-j85n8" (OuterVolumeSpecName: "kube-api-access-j85n8") pod "e0de2707-a7dd-47d5-88da-0e774363d983" (UID: "e0de2707-a7dd-47d5-88da-0e774363d983"). InnerVolumeSpecName "kube-api-access-j85n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:26:56.043886 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.043803    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490b4bce-9bc5-4215-afad-569e6305fdfa-plugins-config-volume\") pod \"490b4bce-9bc5-4215-afad-569e6305fdfa\" (UID: \"490b4bce-9bc5-4215-afad-569e6305fdfa\") "
Sep 07 10:26:56.043886 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.043873    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxx67\" (UniqueName: \"kubernetes.io/projected/490b4bce-9bc5-4215-afad-569e6305fdfa-kube-api-access-fxx67\") pod \"490b4bce-9bc5-4215-afad-569e6305fdfa\" (UID: \"490b4bce-9bc5-4215-afad-569e6305fdfa\") "
Sep 07 10:26:56.044216 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.044198    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0de2707-a7dd-47d5-88da-0e774363d983-openshift-service-ca\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:56.044255 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.044224    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:56.044255 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.044239    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j85n8\" (UniqueName: \"kubernetes.io/projected/e0de2707-a7dd-47d5-88da-0e774363d983-kube-api-access-j85n8\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:56.044329 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.044252    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0de2707-a7dd-47d5-88da-0e774363d983-maas-api-tls\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:56.044329 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.044264    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490b4bce-9bc5-4215-afad-569e6305fdfa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "490b4bce-9bc5-4215-afad-569e6305fdfa" (UID: "490b4bce-9bc5-4215-afad-569e6305fdfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:26:56.045969 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.045942    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490b4bce-9bc5-4215-afad-569e6305fdfa-kube-api-access-fxx67" (OuterVolumeSpecName: "kube-api-access-fxx67") pod "490b4bce-9bc5-4215-afad-569e6305fdfa" (UID: "490b4bce-9bc5-4215-afad-569e6305fdfa"). InnerVolumeSpecName "kube-api-access-fxx67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:26:56.144696 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.144655    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490b4bce-9bc5-4215-afad-569e6305fdfa-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:56.144696 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.144682    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxx67\" (UniqueName: \"kubernetes.io/projected/490b4bce-9bc5-4215-afad-569e6305fdfa-kube-api-access-fxx67\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:26:56.608196 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.608159    2562 generic.go:356] "Generic (PLEG): container finished" podID="490b4bce-9bc5-4215-afad-569e6305fdfa" containerID="b6a9302b9c1d8ad655afdb4800b9329a58b3b995bdc6157ff30ed34cea9f3559" exitCode=0
Sep 07 10:26:56.608654 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.608209    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"
Sep 07 10:26:56.608654 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.608255    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c" event={"ID":"490b4bce-9bc5-4215-afad-569e6305fdfa","Type":"ContainerDied","Data":"b6a9302b9c1d8ad655afdb4800b9329a58b3b995bdc6157ff30ed34cea9f3559"}
Sep 07 10:26:56.608654 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.608314    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c" event={"ID":"490b4bce-9bc5-4215-afad-569e6305fdfa","Type":"ContainerDied","Data":"f4f66c2e679146b9a21d89973cec963daf9f78e15a3e39667999e5467aa2bcbe"}
Sep 07 10:26:56.608654 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.608348    2562 scope.go:117] "RemoveContainer" containerID="b6a9302b9c1d8ad655afdb4800b9329a58b3b995bdc6157ff30ed34cea9f3559"
Sep 07 10:26:56.609976 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.609729    2562 generic.go:356] "Generic (PLEG): container finished" podID="e0de2707-a7dd-47d5-88da-0e774363d983" containerID="1c846f5bd389b06017ad06fbe3bd42c9a766c9fd2f5587ccc1d95a2f4958fb67" exitCode=0
Sep 07 10:26:56.609976 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.609798    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t" event={"ID":"e0de2707-a7dd-47d5-88da-0e774363d983","Type":"ContainerDied","Data":"1c846f5bd389b06017ad06fbe3bd42c9a766c9fd2f5587ccc1d95a2f4958fb67"}
Sep 07 10:26:56.609976 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.609834    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"
Sep 07 10:26:56.609976 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.609845    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t" event={"ID":"e0de2707-a7dd-47d5-88da-0e774363d983","Type":"ContainerDied","Data":"13402a97265af20b6ccfb0682f6b1039796fed8fe564d307dbe1723eef1d1db5"}
Sep 07 10:26:56.618611 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.618594    2562 scope.go:117] "RemoveContainer" containerID="b6a9302b9c1d8ad655afdb4800b9329a58b3b995bdc6157ff30ed34cea9f3559"
Sep 07 10:26:56.618922 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:26:56.618900    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a9302b9c1d8ad655afdb4800b9329a58b3b995bdc6157ff30ed34cea9f3559\": container with ID starting with b6a9302b9c1d8ad655afdb4800b9329a58b3b995bdc6157ff30ed34cea9f3559 not found: ID does not exist" containerID="b6a9302b9c1d8ad655afdb4800b9329a58b3b995bdc6157ff30ed34cea9f3559"
Sep 07 10:26:56.619017 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.618933    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a9302b9c1d8ad655afdb4800b9329a58b3b995bdc6157ff30ed34cea9f3559"} err="failed to get container status \"b6a9302b9c1d8ad655afdb4800b9329a58b3b995bdc6157ff30ed34cea9f3559\": rpc error: code = NotFound desc = could not find container \"b6a9302b9c1d8ad655afdb4800b9329a58b3b995bdc6157ff30ed34cea9f3559\": container with ID starting with b6a9302b9c1d8ad655afdb4800b9329a58b3b995bdc6157ff30ed34cea9f3559 not found: ID does not exist"
Sep 07 10:26:56.619017 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.618956    2562 scope.go:117] "RemoveContainer" containerID="1c846f5bd389b06017ad06fbe3bd42c9a766c9fd2f5587ccc1d95a2f4958fb67"
Sep 07 10:26:56.627793 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.627667    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"]
Sep 07 10:26:56.628015 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.627997    2562 scope.go:117] "RemoveContainer" containerID="1c846f5bd389b06017ad06fbe3bd42c9a766c9fd2f5587ccc1d95a2f4958fb67"
Sep 07 10:26:56.628305 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:26:56.628286    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c846f5bd389b06017ad06fbe3bd42c9a766c9fd2f5587ccc1d95a2f4958fb67\": container with ID starting with 1c846f5bd389b06017ad06fbe3bd42c9a766c9fd2f5587ccc1d95a2f4958fb67 not found: ID does not exist" containerID="1c846f5bd389b06017ad06fbe3bd42c9a766c9fd2f5587ccc1d95a2f4958fb67"
Sep 07 10:26:56.628349 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.628311    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c846f5bd389b06017ad06fbe3bd42c9a766c9fd2f5587ccc1d95a2f4958fb67"} err="failed to get container status \"1c846f5bd389b06017ad06fbe3bd42c9a766c9fd2f5587ccc1d95a2f4958fb67\": rpc error: code = NotFound desc = could not find container \"1c846f5bd389b06017ad06fbe3bd42c9a766c9fd2f5587ccc1d95a2f4958fb67\": container with ID starting with 1c846f5bd389b06017ad06fbe3bd42c9a766c9fd2f5587ccc1d95a2f4958fb67 not found: ID does not exist"
Sep 07 10:26:56.635390 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.635365    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39223a0b-7d4cfb98c4-qck7t"]
Sep 07 10:26:56.645432 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.645408    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"]
Sep 07 10:26:56.649325 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:56.649300    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39223a0b-586458f45c-gkn9c"]
Sep 07 10:26:57.965016 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:57.964983    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="490b4bce-9bc5-4215-afad-569e6305fdfa" path="/var/lib/kubelet/pods/490b4bce-9bc5-4215-afad-569e6305fdfa/volumes"
Sep 07 10:26:57.965404 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:26:57.965390    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0de2707-a7dd-47d5-88da-0e774363d983" path="/var/lib/kubelet/pods/e0de2707-a7dd-47d5-88da-0e774363d983/volumes"
Sep 07 10:27:23.497693 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.497648    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"]
Sep 07 10:27:23.498157 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.498110    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="490b4bce-9bc5-4215-afad-569e6305fdfa" containerName="payload-processing"
Sep 07 10:27:23.498157 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.498125    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="490b4bce-9bc5-4215-afad-569e6305fdfa" containerName="payload-processing"
Sep 07 10:27:23.498157 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.498140    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0de2707-a7dd-47d5-88da-0e774363d983" containerName="maas-api"
Sep 07 10:27:23.498157 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.498146    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0de2707-a7dd-47d5-88da-0e774363d983" containerName="maas-api"
Sep 07 10:27:23.498288 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.498208    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0de2707-a7dd-47d5-88da-0e774363d983" containerName="maas-api"
Sep 07 10:27:23.498288 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.498215    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="490b4bce-9bc5-4215-afad-569e6305fdfa" containerName="payload-processing"
Sep 07 10:27:23.501308 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.501285    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.504849 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.504824    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-81af70\"/\"openshift-service-ca.crt\""
Sep 07 10:27:23.504849 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.504839    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-81af70\"/\"kube-root-ca.crt\""
Sep 07 10:27:23.504849 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.504826    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-81af70\"/\"default-dockercfg-mg4j6\""
Sep 07 10:27:23.505059 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.504833    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-81af70\"/\"explicit-model-81af70-kserve-self-signed-certs\""
Sep 07 10:27:23.510065 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.510043    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"]
Sep 07 10:27:23.581023 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.580957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-model-cache\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.581200 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.581054    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12df332-dc8f-4a6d-b890-416208bcee87-tls-certs\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.581200 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.581079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrcsc\" (UniqueName: \"kubernetes.io/projected/b12df332-dc8f-4a6d-b890-416208bcee87-kube-api-access-lrcsc\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.581200 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.581108    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-dshm\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.581200 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.581190    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-tmp-dir\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.581336 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.581213    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-home\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.682381 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.682340    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12df332-dc8f-4a6d-b890-416208bcee87-tls-certs\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.682381 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.682383    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrcsc\" (UniqueName: \"kubernetes.io/projected/b12df332-dc8f-4a6d-b890-416208bcee87-kube-api-access-lrcsc\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.682609 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.682406    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-dshm\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.682609 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.682449    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-tmp-dir\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.682609 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.682466    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-home\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.682609 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.682514    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-model-cache\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.682960 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.682940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-home\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.683031 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.682972    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-tmp-dir\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.683031 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.683018    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-model-cache\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.684830 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.684812    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-dshm\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.685006 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.684987    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12df332-dc8f-4a6d-b890-416208bcee87-tls-certs\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.690189 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.690166    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrcsc\" (UniqueName: \"kubernetes.io/projected/b12df332-dc8f-4a6d-b890-416208bcee87-kube-api-access-lrcsc\") pod \"explicit-model-81af70-kserve-844ff5677f-6qzbt\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") " pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:23.811884 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:23.811796    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:24.087466 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:24.087437    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"]
Sep 07 10:27:24.088691 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:27:24.088665    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb12df332_dc8f_4a6d_b890_416208bcee87.slice/crio-17bb3e8feb57b154f664d010a52134f19bc8a03ef06ad1fcc7b05f99860c55e6 WatchSource:0}: Error finding container 17bb3e8feb57b154f664d010a52134f19bc8a03ef06ad1fcc7b05f99860c55e6: Status 404 returned error can't find the container with id 17bb3e8feb57b154f664d010a52134f19bc8a03ef06ad1fcc7b05f99860c55e6
Sep 07 10:27:24.779158 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:24.779116    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt" event={"ID":"b12df332-dc8f-4a6d-b890-416208bcee87","Type":"ContainerStarted","Data":"a94f3bc107a35bc696df6fb2bc81286590d0406a8d9023531c2aca8c75242fda"}
Sep 07 10:27:24.779538 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:24.779171    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt" event={"ID":"b12df332-dc8f-4a6d-b890-416208bcee87","Type":"ContainerStarted","Data":"17bb3e8feb57b154f664d010a52134f19bc8a03ef06ad1fcc7b05f99860c55e6"}
Sep 07 10:27:24.795970 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:24.795921    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt" podStartSLOduration=1.795905222 podStartE2EDuration="1.795905222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:27:24.795209417 +0000 UTC m=+947.292899326" watchObservedRunningTime="2026-09-07 10:27:24.795905222 +0000 UTC m=+947.293595154"
Sep 07 10:27:27.374665 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:27.374628    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"]
Sep 07 10:27:33.812330 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:33.812290    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:38.156788 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.156727    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"]
Sep 07 10:27:38.157186 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.156978    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb" podUID="c120adf4-3e62-425e-bd26-e5e3f784653a" containerName="payload-pre-processing" containerID="cri-o://431f94f8db50aae5c3086c21641fa16ec17d953cf079db1aeafc7cd4b9e0a59d" gracePeriod=30
Sep 07 10:27:38.418011 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.417981    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"
Sep 07 10:27:38.520617 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.520580    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzcnz\" (UniqueName: \"kubernetes.io/projected/c120adf4-3e62-425e-bd26-e5e3f784653a-kube-api-access-xzcnz\") pod \"c120adf4-3e62-425e-bd26-e5e3f784653a\" (UID: \"c120adf4-3e62-425e-bd26-e5e3f784653a\") "
Sep 07 10:27:38.520796 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.520628    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c120adf4-3e62-425e-bd26-e5e3f784653a-plugins-config-volume\") pod \"c120adf4-3e62-425e-bd26-e5e3f784653a\" (UID: \"c120adf4-3e62-425e-bd26-e5e3f784653a\") "
Sep 07 10:27:38.521077 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.521053    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c120adf4-3e62-425e-bd26-e5e3f784653a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c120adf4-3e62-425e-bd26-e5e3f784653a" (UID: "c120adf4-3e62-425e-bd26-e5e3f784653a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:27:38.522689 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.522667    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c120adf4-3e62-425e-bd26-e5e3f784653a-kube-api-access-xzcnz" (OuterVolumeSpecName: "kube-api-access-xzcnz") pod "c120adf4-3e62-425e-bd26-e5e3f784653a" (UID: "c120adf4-3e62-425e-bd26-e5e3f784653a"). InnerVolumeSpecName "kube-api-access-xzcnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:27:38.621357 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.621312    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzcnz\" (UniqueName: \"kubernetes.io/projected/c120adf4-3e62-425e-bd26-e5e3f784653a-kube-api-access-xzcnz\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:27:38.621357 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.621349    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c120adf4-3e62-425e-bd26-e5e3f784653a-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:27:38.873356 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.873267    2562 generic.go:356] "Generic (PLEG): container finished" podID="c120adf4-3e62-425e-bd26-e5e3f784653a" containerID="431f94f8db50aae5c3086c21641fa16ec17d953cf079db1aeafc7cd4b9e0a59d" exitCode=0
Sep 07 10:27:38.873356 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.873315    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"
Sep 07 10:27:38.873356 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.873340    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb" event={"ID":"c120adf4-3e62-425e-bd26-e5e3f784653a","Type":"ContainerDied","Data":"431f94f8db50aae5c3086c21641fa16ec17d953cf079db1aeafc7cd4b9e0a59d"}
Sep 07 10:27:38.873584 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.873391    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb" event={"ID":"c120adf4-3e62-425e-bd26-e5e3f784653a","Type":"ContainerDied","Data":"ff67da70347217680db2755527ea5107cc0b76288bd5838ee4255e045ae51840"}
Sep 07 10:27:38.873584 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.873421    2562 scope.go:117] "RemoveContainer" containerID="431f94f8db50aae5c3086c21641fa16ec17d953cf079db1aeafc7cd4b9e0a59d"
Sep 07 10:27:38.882526 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.882507    2562 scope.go:117] "RemoveContainer" containerID="431f94f8db50aae5c3086c21641fa16ec17d953cf079db1aeafc7cd4b9e0a59d"
Sep 07 10:27:38.882784 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:27:38.882749    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"431f94f8db50aae5c3086c21641fa16ec17d953cf079db1aeafc7cd4b9e0a59d\": container with ID starting with 431f94f8db50aae5c3086c21641fa16ec17d953cf079db1aeafc7cd4b9e0a59d not found: ID does not exist" containerID="431f94f8db50aae5c3086c21641fa16ec17d953cf079db1aeafc7cd4b9e0a59d"
Sep 07 10:27:38.882866 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.882791    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431f94f8db50aae5c3086c21641fa16ec17d953cf079db1aeafc7cd4b9e0a59d"} err="failed to get container status \"431f94f8db50aae5c3086c21641fa16ec17d953cf079db1aeafc7cd4b9e0a59d\": rpc error: code = NotFound desc = could not find container \"431f94f8db50aae5c3086c21641fa16ec17d953cf079db1aeafc7cd4b9e0a59d\": container with ID starting with 431f94f8db50aae5c3086c21641fa16ec17d953cf079db1aeafc7cd4b9e0a59d not found: ID does not exist"
Sep 07 10:27:38.895081 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.895049    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"]
Sep 07 10:27:38.901436 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:38.901400    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-81af70-6d4cdbcd5d-sfcfb"]
Sep 07 10:27:39.965578 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:39.965541    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c120adf4-3e62-425e-bd26-e5e3f784653a" path="/var/lib/kubelet/pods/c120adf4-3e62-425e-bd26-e5e3f784653a/volumes"
Sep 07 10:27:42.395011 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.394972    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt" podUID="b12df332-dc8f-4a6d-b890-416208bcee87" containerName="main" containerID="cri-o://a94f3bc107a35bc696df6fb2bc81286590d0406a8d9023531c2aca8c75242fda" gracePeriod=45
Sep 07 10:27:42.651873 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.651801    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:42.760873 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.760839    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-dshm\") pod \"b12df332-dc8f-4a6d-b890-416208bcee87\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") "
Sep 07 10:27:42.761038 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.760895    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrcsc\" (UniqueName: \"kubernetes.io/projected/b12df332-dc8f-4a6d-b890-416208bcee87-kube-api-access-lrcsc\") pod \"b12df332-dc8f-4a6d-b890-416208bcee87\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") "
Sep 07 10:27:42.761038 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.760924    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12df332-dc8f-4a6d-b890-416208bcee87-tls-certs\") pod \"b12df332-dc8f-4a6d-b890-416208bcee87\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") "
Sep 07 10:27:42.761038 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.760987    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-home\") pod \"b12df332-dc8f-4a6d-b890-416208bcee87\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") "
Sep 07 10:27:42.761235 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.761147    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-tmp-dir\") pod \"b12df332-dc8f-4a6d-b890-416208bcee87\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") "
Sep 07 10:27:42.761235 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.761215    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-model-cache\") pod \"b12df332-dc8f-4a6d-b890-416208bcee87\" (UID: \"b12df332-dc8f-4a6d-b890-416208bcee87\") "
Sep 07 10:27:42.761235 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.761227    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-home" (OuterVolumeSpecName: "home") pod "b12df332-dc8f-4a6d-b890-416208bcee87" (UID: "b12df332-dc8f-4a6d-b890-416208bcee87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:27:42.761386 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.761310    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b12df332-dc8f-4a6d-b890-416208bcee87" (UID: "b12df332-dc8f-4a6d-b890-416208bcee87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:27:42.761470 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.761447    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-model-cache" (OuterVolumeSpecName: "model-cache") pod "b12df332-dc8f-4a6d-b890-416208bcee87" (UID: "b12df332-dc8f-4a6d-b890-416208bcee87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:27:42.761594 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.761572    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-home\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:27:42.761594 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.761593    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-tmp-dir\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:27:42.761594 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.761605    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-model-cache\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:27:42.763208 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.763182    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-dshm" (OuterVolumeSpecName: "dshm") pod "b12df332-dc8f-4a6d-b890-416208bcee87" (UID: "b12df332-dc8f-4a6d-b890-416208bcee87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:27:42.763478 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.763461    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b12df332-dc8f-4a6d-b890-416208bcee87-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b12df332-dc8f-4a6d-b890-416208bcee87" (UID: "b12df332-dc8f-4a6d-b890-416208bcee87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:27:42.763557 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.763541    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12df332-dc8f-4a6d-b890-416208bcee87-kube-api-access-lrcsc" (OuterVolumeSpecName: "kube-api-access-lrcsc") pod "b12df332-dc8f-4a6d-b890-416208bcee87" (UID: "b12df332-dc8f-4a6d-b890-416208bcee87"). InnerVolumeSpecName "kube-api-access-lrcsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:27:42.862352 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.862319    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrcsc\" (UniqueName: \"kubernetes.io/projected/b12df332-dc8f-4a6d-b890-416208bcee87-kube-api-access-lrcsc\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:27:42.862352 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.862351    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b12df332-dc8f-4a6d-b890-416208bcee87-tls-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:27:42.862352 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.862362    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b12df332-dc8f-4a6d-b890-416208bcee87-dshm\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:27:42.897233 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.897194    2562 generic.go:356] "Generic (PLEG): container finished" podID="b12df332-dc8f-4a6d-b890-416208bcee87" containerID="a94f3bc107a35bc696df6fb2bc81286590d0406a8d9023531c2aca8c75242fda" exitCode=0
Sep 07 10:27:42.897397 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.897237    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"
Sep 07 10:27:42.897397 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.897276    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt" event={"ID":"b12df332-dc8f-4a6d-b890-416208bcee87","Type":"ContainerDied","Data":"a94f3bc107a35bc696df6fb2bc81286590d0406a8d9023531c2aca8c75242fda"}
Sep 07 10:27:42.897397 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.897318    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt" event={"ID":"b12df332-dc8f-4a6d-b890-416208bcee87","Type":"ContainerDied","Data":"17bb3e8feb57b154f664d010a52134f19bc8a03ef06ad1fcc7b05f99860c55e6"}
Sep 07 10:27:42.897397 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.897341    2562 scope.go:117] "RemoveContainer" containerID="a94f3bc107a35bc696df6fb2bc81286590d0406a8d9023531c2aca8c75242fda"
Sep 07 10:27:42.906454 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.906437    2562 scope.go:117] "RemoveContainer" containerID="a94f3bc107a35bc696df6fb2bc81286590d0406a8d9023531c2aca8c75242fda"
Sep 07 10:27:42.906747 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:27:42.906727    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a94f3bc107a35bc696df6fb2bc81286590d0406a8d9023531c2aca8c75242fda\": container with ID starting with a94f3bc107a35bc696df6fb2bc81286590d0406a8d9023531c2aca8c75242fda not found: ID does not exist" containerID="a94f3bc107a35bc696df6fb2bc81286590d0406a8d9023531c2aca8c75242fda"
Sep 07 10:27:42.906855 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.906757    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94f3bc107a35bc696df6fb2bc81286590d0406a8d9023531c2aca8c75242fda"} err="failed to get container status \"a94f3bc107a35bc696df6fb2bc81286590d0406a8d9023531c2aca8c75242fda\": rpc error: code = NotFound desc = could not find container \"a94f3bc107a35bc696df6fb2bc81286590d0406a8d9023531c2aca8c75242fda\": container with ID starting with a94f3bc107a35bc696df6fb2bc81286590d0406a8d9023531c2aca8c75242fda not found: ID does not exist"
Sep 07 10:27:42.918274 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.918249    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"]
Sep 07 10:27:42.922270 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:42.922247    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-81af70/explicit-model-81af70-kserve-844ff5677f-6qzbt"]
Sep 07 10:27:43.965046 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:27:43.965012    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b12df332-dc8f-4a6d-b890-416208bcee87" path="/var/lib/kubelet/pods/b12df332-dc8f-4a6d-b890-416208bcee87/volumes"
Sep 07 10:28:27.449050 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.449015    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"]
Sep 07 10:28:27.449493 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.449259    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt" podUID="d8c5359b-0c75-4a64-92d4-c73d21bd2058" containerName="maas-api" containerID="cri-o://7f4c69fba2758846cc61d79c7e76bebd2b5233a0d7f8f3d4a73311a18631791f" gracePeriod=30
Sep 07 10:28:27.501220 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.501175    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"]
Sep 07 10:28:27.501507 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.501441    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg" podUID="5243336b-9ae7-469c-a2ef-5941f675876c" containerName="payload-processing" containerID="cri-o://d2072f9d18a17f92ee1f1c348301d29b66cca6d478b8151de4640eba8976b6c1" gracePeriod=30
Sep 07 10:28:27.517351 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.517315    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"]
Sep 07 10:28:27.517609 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.517557    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8" podUID="1b98cd82-6fd9-4640-9cb4-619e4588e92f" containerName="payload-pre-processing" containerID="cri-o://5ded69d26bc4c58d54f4805d0c9ac45ef243d35e602b9a4e17221f956bdc64fc" gracePeriod=30
Sep 07 10:28:27.633408 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.633353    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt" podUID="d8c5359b-0c75-4a64-92d4-c73d21bd2058" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.51:8443/health\": dial tcp 10.134.0.51:8443: connect: connection refused"
Sep 07 10:28:27.785878 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.785853    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:28:27.788849 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.788826    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"
Sep 07 10:28:27.823696 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.823653    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"
Sep 07 10:28:27.889868 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.889832    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv7zz\" (UniqueName: \"kubernetes.io/projected/5243336b-9ae7-469c-a2ef-5941f675876c-kube-api-access-fv7zz\") pod \"5243336b-9ae7-469c-a2ef-5941f675876c\" (UID: \"5243336b-9ae7-469c-a2ef-5941f675876c\") "
Sep 07 10:28:27.890054 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.889880    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5243336b-9ae7-469c-a2ef-5941f675876c-plugins-config-volume\") pod \"5243336b-9ae7-469c-a2ef-5941f675876c\" (UID: \"5243336b-9ae7-469c-a2ef-5941f675876c\") "
Sep 07 10:28:27.890054 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.889921    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c5359b-0c75-4a64-92d4-c73d21bd2058-openshift-service-ca\") pod \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\" (UID: \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\") "
Sep 07 10:28:27.890242 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.890087    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bckr\" (UniqueName: \"kubernetes.io/projected/d8c5359b-0c75-4a64-92d4-c73d21bd2058-kube-api-access-2bckr\") pod \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\" (UID: \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\") "
Sep 07 10:28:27.890242 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.890135    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c5359b-0c75-4a64-92d4-c73d21bd2058-metrics-certs\") pod \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\" (UID: \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\") "
Sep 07 10:28:27.890344 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.890241    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c5359b-0c75-4a64-92d4-c73d21bd2058-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d8c5359b-0c75-4a64-92d4-c73d21bd2058" (UID: "d8c5359b-0c75-4a64-92d4-c73d21bd2058"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:28:27.890344 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.890244    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5243336b-9ae7-469c-a2ef-5941f675876c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5243336b-9ae7-469c-a2ef-5941f675876c" (UID: "5243336b-9ae7-469c-a2ef-5941f675876c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:28:27.890344 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.890264    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c5359b-0c75-4a64-92d4-c73d21bd2058-maas-api-tls\") pod \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\" (UID: \"d8c5359b-0c75-4a64-92d4-c73d21bd2058\") "
Sep 07 10:28:27.890559 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.890535    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5243336b-9ae7-469c-a2ef-5941f675876c-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:28:27.890559 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.890559    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c5359b-0c75-4a64-92d4-c73d21bd2058-openshift-service-ca\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:28:27.892072 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.892044    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5243336b-9ae7-469c-a2ef-5941f675876c-kube-api-access-fv7zz" (OuterVolumeSpecName: "kube-api-access-fv7zz") pod "5243336b-9ae7-469c-a2ef-5941f675876c" (UID: "5243336b-9ae7-469c-a2ef-5941f675876c"). InnerVolumeSpecName "kube-api-access-fv7zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:28:27.892177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.892076    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c5359b-0c75-4a64-92d4-c73d21bd2058-kube-api-access-2bckr" (OuterVolumeSpecName: "kube-api-access-2bckr") pod "d8c5359b-0c75-4a64-92d4-c73d21bd2058" (UID: "d8c5359b-0c75-4a64-92d4-c73d21bd2058"). InnerVolumeSpecName "kube-api-access-2bckr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:28:27.892177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.892114    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c5359b-0c75-4a64-92d4-c73d21bd2058-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d8c5359b-0c75-4a64-92d4-c73d21bd2058" (UID: "d8c5359b-0c75-4a64-92d4-c73d21bd2058"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:28:27.892257 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.892216    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c5359b-0c75-4a64-92d4-c73d21bd2058-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d8c5359b-0c75-4a64-92d4-c73d21bd2058" (UID: "d8c5359b-0c75-4a64-92d4-c73d21bd2058"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:28:27.991843 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.991809    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98cd82-6fd9-4640-9cb4-619e4588e92f-plugins-config-volume\") pod \"1b98cd82-6fd9-4640-9cb4-619e4588e92f\" (UID: \"1b98cd82-6fd9-4640-9cb4-619e4588e92f\") "
Sep 07 10:28:27.992023 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.991863    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bxgq\" (UniqueName: \"kubernetes.io/projected/1b98cd82-6fd9-4640-9cb4-619e4588e92f-kube-api-access-7bxgq\") pod \"1b98cd82-6fd9-4640-9cb4-619e4588e92f\" (UID: \"1b98cd82-6fd9-4640-9cb4-619e4588e92f\") "
Sep 07 10:28:27.992130 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.992089    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bckr\" (UniqueName: \"kubernetes.io/projected/d8c5359b-0c75-4a64-92d4-c73d21bd2058-kube-api-access-2bckr\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:28:27.992130 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.992129    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c5359b-0c75-4a64-92d4-c73d21bd2058-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:28:27.992259 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.992143    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c5359b-0c75-4a64-92d4-c73d21bd2058-maas-api-tls\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:28:27.992259 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.992152    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv7zz\" (UniqueName: \"kubernetes.io/projected/5243336b-9ae7-469c-a2ef-5941f675876c-kube-api-access-fv7zz\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:28:27.992259 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.992240    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b98cd82-6fd9-4640-9cb4-619e4588e92f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1b98cd82-6fd9-4640-9cb4-619e4588e92f" (UID: "1b98cd82-6fd9-4640-9cb4-619e4588e92f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:28:27.993918 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:27.993898    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b98cd82-6fd9-4640-9cb4-619e4588e92f-kube-api-access-7bxgq" (OuterVolumeSpecName: "kube-api-access-7bxgq") pod "1b98cd82-6fd9-4640-9cb4-619e4588e92f" (UID: "1b98cd82-6fd9-4640-9cb4-619e4588e92f"). InnerVolumeSpecName "kube-api-access-7bxgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:28:28.093241 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.093203    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98cd82-6fd9-4640-9cb4-619e4588e92f-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:28:28.093241 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.093231    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bxgq\" (UniqueName: \"kubernetes.io/projected/1b98cd82-6fd9-4640-9cb4-619e4588e92f-kube-api-access-7bxgq\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:28:28.151335 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.151292    2562 generic.go:356] "Generic (PLEG): container finished" podID="1b98cd82-6fd9-4640-9cb4-619e4588e92f" containerID="5ded69d26bc4c58d54f4805d0c9ac45ef243d35e602b9a4e17221f956bdc64fc" exitCode=0
Sep 07 10:28:28.151510 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.151339    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"
Sep 07 10:28:28.151510 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.151374    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8" event={"ID":"1b98cd82-6fd9-4640-9cb4-619e4588e92f","Type":"ContainerDied","Data":"5ded69d26bc4c58d54f4805d0c9ac45ef243d35e602b9a4e17221f956bdc64fc"}
Sep 07 10:28:28.151510 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.151432    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8" event={"ID":"1b98cd82-6fd9-4640-9cb4-619e4588e92f","Type":"ContainerDied","Data":"1ef17b1a52fe185b8d4c776171e63abfd131ded0646d4ff7676e6795d9c0bb51"}
Sep 07 10:28:28.151510 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.151463    2562 scope.go:117] "RemoveContainer" containerID="5ded69d26bc4c58d54f4805d0c9ac45ef243d35e602b9a4e17221f956bdc64fc"
Sep 07 10:28:28.153000 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.152979    2562 generic.go:356] "Generic (PLEG): container finished" podID="5243336b-9ae7-469c-a2ef-5941f675876c" containerID="d2072f9d18a17f92ee1f1c348301d29b66cca6d478b8151de4640eba8976b6c1" exitCode=0
Sep 07 10:28:28.153111 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.153018    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"
Sep 07 10:28:28.153111 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.153062    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg" event={"ID":"5243336b-9ae7-469c-a2ef-5941f675876c","Type":"ContainerDied","Data":"d2072f9d18a17f92ee1f1c348301d29b66cca6d478b8151de4640eba8976b6c1"}
Sep 07 10:28:28.153224 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.153115    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg" event={"ID":"5243336b-9ae7-469c-a2ef-5941f675876c","Type":"ContainerDied","Data":"e418de3fb3ca2e3002f928c6b02ac941a17973fbab29fae5f4ecf44f3176aa93"}
Sep 07 10:28:28.154816 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.154792    2562 generic.go:356] "Generic (PLEG): container finished" podID="d8c5359b-0c75-4a64-92d4-c73d21bd2058" containerID="7f4c69fba2758846cc61d79c7e76bebd2b5233a0d7f8f3d4a73311a18631791f" exitCode=0
Sep 07 10:28:28.154953 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.154838    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt" event={"ID":"d8c5359b-0c75-4a64-92d4-c73d21bd2058","Type":"ContainerDied","Data":"7f4c69fba2758846cc61d79c7e76bebd2b5233a0d7f8f3d4a73311a18631791f"}
Sep 07 10:28:28.154953 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.154876    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt" event={"ID":"d8c5359b-0c75-4a64-92d4-c73d21bd2058","Type":"ContainerDied","Data":"50a6003b06d755577df0d9cd4408edfb1e397b9cd9f6249391eb49f90dcd4522"}
Sep 07 10:28:28.154953 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.154934    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"
Sep 07 10:28:28.162826 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.162810    2562 scope.go:117] "RemoveContainer" containerID="5ded69d26bc4c58d54f4805d0c9ac45ef243d35e602b9a4e17221f956bdc64fc"
Sep 07 10:28:28.163085 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:28:28.163067    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ded69d26bc4c58d54f4805d0c9ac45ef243d35e602b9a4e17221f956bdc64fc\": container with ID starting with 5ded69d26bc4c58d54f4805d0c9ac45ef243d35e602b9a4e17221f956bdc64fc not found: ID does not exist" containerID="5ded69d26bc4c58d54f4805d0c9ac45ef243d35e602b9a4e17221f956bdc64fc"
Sep 07 10:28:28.163133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.163093    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ded69d26bc4c58d54f4805d0c9ac45ef243d35e602b9a4e17221f956bdc64fc"} err="failed to get container status \"5ded69d26bc4c58d54f4805d0c9ac45ef243d35e602b9a4e17221f956bdc64fc\": rpc error: code = NotFound desc = could not find container \"5ded69d26bc4c58d54f4805d0c9ac45ef243d35e602b9a4e17221f956bdc64fc\": container with ID starting with 5ded69d26bc4c58d54f4805d0c9ac45ef243d35e602b9a4e17221f956bdc64fc not found: ID does not exist"
Sep 07 10:28:28.163133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.163107    2562 scope.go:117] "RemoveContainer" containerID="d2072f9d18a17f92ee1f1c348301d29b66cca6d478b8151de4640eba8976b6c1"
Sep 07 10:28:28.171623 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.171597    2562 scope.go:117] "RemoveContainer" containerID="d2072f9d18a17f92ee1f1c348301d29b66cca6d478b8151de4640eba8976b6c1"
Sep 07 10:28:28.171877 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:28:28.171860    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2072f9d18a17f92ee1f1c348301d29b66cca6d478b8151de4640eba8976b6c1\": container with ID starting with d2072f9d18a17f92ee1f1c348301d29b66cca6d478b8151de4640eba8976b6c1 not found: ID does not exist" containerID="d2072f9d18a17f92ee1f1c348301d29b66cca6d478b8151de4640eba8976b6c1"
Sep 07 10:28:28.171954 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.171885    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2072f9d18a17f92ee1f1c348301d29b66cca6d478b8151de4640eba8976b6c1"} err="failed to get container status \"d2072f9d18a17f92ee1f1c348301d29b66cca6d478b8151de4640eba8976b6c1\": rpc error: code = NotFound desc = could not find container \"d2072f9d18a17f92ee1f1c348301d29b66cca6d478b8151de4640eba8976b6c1\": container with ID starting with d2072f9d18a17f92ee1f1c348301d29b66cca6d478b8151de4640eba8976b6c1 not found: ID does not exist"
Sep 07 10:28:28.171954 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.171899    2562 scope.go:117] "RemoveContainer" containerID="7f4c69fba2758846cc61d79c7e76bebd2b5233a0d7f8f3d4a73311a18631791f"
Sep 07 10:28:28.176190 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.176164    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"]
Sep 07 10:28:28.181601 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.181585    2562 scope.go:117] "RemoveContainer" containerID="7f4c69fba2758846cc61d79c7e76bebd2b5233a0d7f8f3d4a73311a18631791f"
Sep 07 10:28:28.181904 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:28:28.181877    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f4c69fba2758846cc61d79c7e76bebd2b5233a0d7f8f3d4a73311a18631791f\": container with ID starting with 7f4c69fba2758846cc61d79c7e76bebd2b5233a0d7f8f3d4a73311a18631791f not found: ID does not exist" containerID="7f4c69fba2758846cc61d79c7e76bebd2b5233a0d7f8f3d4a73311a18631791f"
Sep 07 10:28:28.181994 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.181914    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f4c69fba2758846cc61d79c7e76bebd2b5233a0d7f8f3d4a73311a18631791f"} err="failed to get container status \"7f4c69fba2758846cc61d79c7e76bebd2b5233a0d7f8f3d4a73311a18631791f\": rpc error: code = NotFound desc = could not find container \"7f4c69fba2758846cc61d79c7e76bebd2b5233a0d7f8f3d4a73311a18631791f\": container with ID starting with 7f4c69fba2758846cc61d79c7e76bebd2b5233a0d7f8f3d4a73311a18631791f not found: ID does not exist"
Sep 07 10:28:28.182320 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.182300    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05ecbf-64999b4f9f-jxwkt"]
Sep 07 10:28:28.197454 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.197427    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"]
Sep 07 10:28:28.204247 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.204222    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05ecbf-6f475fff94-x29n8"]
Sep 07 10:28:28.214876 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.214837    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"]
Sep 07 10:28:28.218884 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:28.218862    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05ecbf-7b5664467f-pd5xg"]
Sep 07 10:28:29.965286 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:29.965252    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b98cd82-6fd9-4640-9cb4-619e4588e92f" path="/var/lib/kubelet/pods/1b98cd82-6fd9-4640-9cb4-619e4588e92f/volumes"
Sep 07 10:28:29.965666 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:29.965605    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5243336b-9ae7-469c-a2ef-5941f675876c" path="/var/lib/kubelet/pods/5243336b-9ae7-469c-a2ef-5941f675876c/volumes"
Sep 07 10:28:29.966004 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:29.965988    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8c5359b-0c75-4a64-92d4-c73d21bd2058" path="/var/lib/kubelet/pods/d8c5359b-0c75-4a64-92d4-c73d21bd2058/volumes"
Sep 07 10:28:34.166636 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.166582    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"]
Sep 07 10:28:34.167133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.167012    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c120adf4-3e62-425e-bd26-e5e3f784653a" containerName="payload-pre-processing"
Sep 07 10:28:34.167133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.167026    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c120adf4-3e62-425e-bd26-e5e3f784653a" containerName="payload-pre-processing"
Sep 07 10:28:34.167133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.167038    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b12df332-dc8f-4a6d-b890-416208bcee87" containerName="main"
Sep 07 10:28:34.167133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.167044    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b12df332-dc8f-4a6d-b890-416208bcee87" containerName="main"
Sep 07 10:28:34.167133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.167055    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8c5359b-0c75-4a64-92d4-c73d21bd2058" containerName="maas-api"
Sep 07 10:28:34.167133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.167061    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8c5359b-0c75-4a64-92d4-c73d21bd2058" containerName="maas-api"
Sep 07 10:28:34.167133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.167067    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5243336b-9ae7-469c-a2ef-5941f675876c" containerName="payload-processing"
Sep 07 10:28:34.167133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.167073    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5243336b-9ae7-469c-a2ef-5941f675876c" containerName="payload-processing"
Sep 07 10:28:34.167133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.167087    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b98cd82-6fd9-4640-9cb4-619e4588e92f" containerName="payload-pre-processing"
Sep 07 10:28:34.167133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.167092    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b98cd82-6fd9-4640-9cb4-619e4588e92f" containerName="payload-pre-processing"
Sep 07 10:28:34.167664 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.167154    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8c5359b-0c75-4a64-92d4-c73d21bd2058" containerName="maas-api"
Sep 07 10:28:34.167664 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.167164    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5243336b-9ae7-469c-a2ef-5941f675876c" containerName="payload-processing"
Sep 07 10:28:34.167664 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.167170    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c120adf4-3e62-425e-bd26-e5e3f784653a" containerName="payload-pre-processing"
Sep 07 10:28:34.167664 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.167177    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b98cd82-6fd9-4640-9cb4-619e4588e92f" containerName="payload-pre-processing"
Sep 07 10:28:34.167664 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.167181    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b12df332-dc8f-4a6d-b890-416208bcee87" containerName="main"
Sep 07 10:28:34.172394 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.172367    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:34.174709 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.174687    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2c28e354-serving-cert\""
Sep 07 10:28:34.183068 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.183039    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"]
Sep 07 10:28:34.345513 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.345475    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daff2afc-cce6-4753-bed8-a2efbf4e984b-maas-api-tls\") pod \"maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:34.345716 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.345566    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlrnj\" (UniqueName: \"kubernetes.io/projected/daff2afc-cce6-4753-bed8-a2efbf4e984b-kube-api-access-vlrnj\") pod \"maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:34.345716 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.345678    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daff2afc-cce6-4753-bed8-a2efbf4e984b-metrics-certs\") pod \"maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:34.345820 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.345721    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daff2afc-cce6-4753-bed8-a2efbf4e984b-openshift-service-ca\") pod \"maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:34.446952 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.446867    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daff2afc-cce6-4753-bed8-a2efbf4e984b-maas-api-tls\") pod \"maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:34.446952 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.446905    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlrnj\" (UniqueName: \"kubernetes.io/projected/daff2afc-cce6-4753-bed8-a2efbf4e984b-kube-api-access-vlrnj\") pod \"maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:34.447155 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.447024    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daff2afc-cce6-4753-bed8-a2efbf4e984b-metrics-certs\") pod \"maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:34.447155 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:28:34.447053    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-serving-cert: secret "maas-api-e2e-mt-2c28e354-serving-cert" not found
Sep 07 10:28:34.447155 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:28:34.447128    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daff2afc-cce6-4753-bed8-a2efbf4e984b-maas-api-tls podName:daff2afc-cce6-4753-bed8-a2efbf4e984b nodeName:}" failed. No retries permitted until 2026-09-07 10:28:34.947106701 +0000 UTC m=+1017.444796604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/daff2afc-cce6-4753-bed8-a2efbf4e984b-maas-api-tls") pod "maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf" (UID: "daff2afc-cce6-4753-bed8-a2efbf4e984b") : secret "maas-api-e2e-mt-2c28e354-serving-cert" not found
Sep 07 10:28:34.447155 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.447062    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daff2afc-cce6-4753-bed8-a2efbf4e984b-openshift-service-ca\") pod \"maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:34.447825 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.447804    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daff2afc-cce6-4753-bed8-a2efbf4e984b-openshift-service-ca\") pod \"maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:34.450108 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.450083    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daff2afc-cce6-4753-bed8-a2efbf4e984b-metrics-certs\") pod \"maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:34.455737 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.455712    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlrnj\" (UniqueName: \"kubernetes.io/projected/daff2afc-cce6-4753-bed8-a2efbf4e984b-kube-api-access-vlrnj\") pod \"maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:34.799435 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.799390    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"]
Sep 07 10:28:34.803073 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.803055    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"
Sep 07 10:28:34.805646 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.805617    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2c28e354\""
Sep 07 10:28:34.816050 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.816021    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"]
Sep 07 10:28:34.847497 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.847464    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"]
Sep 07 10:28:34.850999 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.850974    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"
Sep 07 10:28:34.859849 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.859580    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2c28e354-dockercfg-99xq9\""
Sep 07 10:28:34.862418 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.862385    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"]
Sep 07 10:28:34.953052 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.953010    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daff2afc-cce6-4753-bed8-a2efbf4e984b-maas-api-tls\") pod \"maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:34.953237 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.953091    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfcr9\" (UniqueName: \"kubernetes.io/projected/90b62764-e517-4856-94ff-88016d8af33c-kube-api-access-jfcr9\") pod \"payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t\" (UID: \"90b62764-e517-4856-94ff-88016d8af33c\") " pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"
Sep 07 10:28:34.953237 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.953138    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c6bb\" (UniqueName: \"kubernetes.io/projected/964260c1-7f5a-4f85-9586-4e7e4924fd48-kube-api-access-9c6bb\") pod \"payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7\" (UID: \"964260c1-7f5a-4f85-9586-4e7e4924fd48\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"
Sep 07 10:28:34.953237 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.953195    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b62764-e517-4856-94ff-88016d8af33c-plugins-config-volume\") pod \"payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t\" (UID: \"90b62764-e517-4856-94ff-88016d8af33c\") " pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"
Sep 07 10:28:34.953373 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.953271    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964260c1-7f5a-4f85-9586-4e7e4924fd48-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7\" (UID: \"964260c1-7f5a-4f85-9586-4e7e4924fd48\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"
Sep 07 10:28:34.955369 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:34.955347    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daff2afc-cce6-4753-bed8-a2efbf4e984b-maas-api-tls\") pod \"maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:35.054935 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:35.054821    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfcr9\" (UniqueName: \"kubernetes.io/projected/90b62764-e517-4856-94ff-88016d8af33c-kube-api-access-jfcr9\") pod \"payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t\" (UID: \"90b62764-e517-4856-94ff-88016d8af33c\") " pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"
Sep 07 10:28:35.054935 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:35.054894    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c6bb\" (UniqueName: \"kubernetes.io/projected/964260c1-7f5a-4f85-9586-4e7e4924fd48-kube-api-access-9c6bb\") pod \"payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7\" (UID: \"964260c1-7f5a-4f85-9586-4e7e4924fd48\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"
Sep 07 10:28:35.054935 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:35.054922    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b62764-e517-4856-94ff-88016d8af33c-plugins-config-volume\") pod \"payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t\" (UID: \"90b62764-e517-4856-94ff-88016d8af33c\") " pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"
Sep 07 10:28:35.055228 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:35.054981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964260c1-7f5a-4f85-9586-4e7e4924fd48-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7\" (UID: \"964260c1-7f5a-4f85-9586-4e7e4924fd48\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"
Sep 07 10:28:35.055895 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:35.055869    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b62764-e517-4856-94ff-88016d8af33c-plugins-config-volume\") pod \"payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t\" (UID: \"90b62764-e517-4856-94ff-88016d8af33c\") " pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"
Sep 07 10:28:35.056045 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:35.055900    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964260c1-7f5a-4f85-9586-4e7e4924fd48-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7\" (UID: \"964260c1-7f5a-4f85-9586-4e7e4924fd48\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"
Sep 07 10:28:35.063269 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:35.063238    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c6bb\" (UniqueName: \"kubernetes.io/projected/964260c1-7f5a-4f85-9586-4e7e4924fd48-kube-api-access-9c6bb\") pod \"payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7\" (UID: \"964260c1-7f5a-4f85-9586-4e7e4924fd48\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"
Sep 07 10:28:35.063403 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:35.063268    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfcr9\" (UniqueName: \"kubernetes.io/projected/90b62764-e517-4856-94ff-88016d8af33c-kube-api-access-jfcr9\") pod \"payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t\" (UID: \"90b62764-e517-4856-94ff-88016d8af33c\") " pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"
Sep 07 10:28:35.086854 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:35.086821    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:35.114859 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:35.114816    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"
Sep 07 10:28:35.168045 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:35.168004    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"
Sep 07 10:28:35.364751 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:35.364723    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"]
Sep 07 10:28:35.366543 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:28:35.366519    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaff2afc_cce6_4753_bed8_a2efbf4e984b.slice/crio-33f57981ad025a2b7539cfca9995f9417ecbbcd96851fd795412a124d92c12ce WatchSource:0}: Error finding container 33f57981ad025a2b7539cfca9995f9417ecbbcd96851fd795412a124d92c12ce: Status 404 returned error can't find the container with id 33f57981ad025a2b7539cfca9995f9417ecbbcd96851fd795412a124d92c12ce
Sep 07 10:28:35.368457 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:35.368438    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 10:28:35.402411 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:35.402379    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"]
Sep 07 10:28:35.403304 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:28:35.403280    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964260c1_7f5a_4f85_9586_4e7e4924fd48.slice/crio-2fa6b67de485d28bbbd13017b5657c5b7fc97292a6ddb8f29bcbbc8262242db4 WatchSource:0}: Error finding container 2fa6b67de485d28bbbd13017b5657c5b7fc97292a6ddb8f29bcbbc8262242db4: Status 404 returned error can't find the container with id 2fa6b67de485d28bbbd13017b5657c5b7fc97292a6ddb8f29bcbbc8262242db4
Sep 07 10:28:35.450420 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:35.450396    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"]
Sep 07 10:28:35.452155 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:28:35.452131    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b62764_e517_4856_94ff_88016d8af33c.slice/crio-f5fdd44247ad87e1de2944a6a6eb38c561726e28f2cac0eedb2df3ee3bcf4d93 WatchSource:0}: Error finding container f5fdd44247ad87e1de2944a6a6eb38c561726e28f2cac0eedb2df3ee3bcf4d93: Status 404 returned error can't find the container with id f5fdd44247ad87e1de2944a6a6eb38c561726e28f2cac0eedb2df3ee3bcf4d93
Sep 07 10:28:36.210216 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:36.210174    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf" event={"ID":"daff2afc-cce6-4753-bed8-a2efbf4e984b","Type":"ContainerStarted","Data":"9b8c2ea7c09f0a3d740d6d8c0a6be3c6648eaa8cb7c735ce2828af13f16f69a5"}
Sep 07 10:28:36.210690 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:36.210236    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf" event={"ID":"daff2afc-cce6-4753-bed8-a2efbf4e984b","Type":"ContainerStarted","Data":"33f57981ad025a2b7539cfca9995f9417ecbbcd96851fd795412a124d92c12ce"}
Sep 07 10:28:36.210690 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:36.210264    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:36.211829 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:36.211800    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t" event={"ID":"90b62764-e517-4856-94ff-88016d8af33c","Type":"ContainerStarted","Data":"6d4b75ab61efaf3d64a7246774265ad12c64f932e9d88112d1e5f5ece6c02922"}
Sep 07 10:28:36.211938 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:36.211850    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t" event={"ID":"90b62764-e517-4856-94ff-88016d8af33c","Type":"ContainerStarted","Data":"f5fdd44247ad87e1de2944a6a6eb38c561726e28f2cac0eedb2df3ee3bcf4d93"}
Sep 07 10:28:36.211938 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:36.211880    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"
Sep 07 10:28:36.213340 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:36.213315    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7" event={"ID":"964260c1-7f5a-4f85-9586-4e7e4924fd48","Type":"ContainerStarted","Data":"d5b2f26001a0b711a202fff86a74c992fb6ada94e55853b51f028cf210eb2ed8"}
Sep 07 10:28:36.213468 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:36.213361    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7" event={"ID":"964260c1-7f5a-4f85-9586-4e7e4924fd48","Type":"ContainerStarted","Data":"2fa6b67de485d28bbbd13017b5657c5b7fc97292a6ddb8f29bcbbc8262242db4"}
Sep 07 10:28:36.213468 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:36.213383    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"
Sep 07 10:28:36.230239 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:36.230174    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf" podStartSLOduration=1.825239087 podStartE2EDuration="2.230153816s" totalImagesPullingTime="404.914729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:28:36.22794213 +0000 UTC m=+1018.725632071" watchObservedRunningTime="2026-09-07 10:28:36.230153816 +0000 UTC m=+1018.727843729"
Sep 07 10:28:36.244333 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:36.244282    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t" podStartSLOduration=2.244270829 podStartE2EDuration="2.244270829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:28:36.243388834 +0000 UTC m=+1018.741078758" watchObservedRunningTime="2026-09-07 10:28:36.244270829 +0000 UTC m=+1018.741960791"
Sep 07 10:28:36.260622 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:36.260567    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7" podStartSLOduration=2.260547751 podStartE2EDuration="2.260547751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:28:36.258914847 +0000 UTC m=+1018.756604785" watchObservedRunningTime="2026-09-07 10:28:36.260547751 +0000 UTC m=+1018.758237663"
Sep 07 10:28:42.225837 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:42.225807    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:28:45.662431 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:45.662394    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"]
Sep 07 10:28:45.662926 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:45.662673    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll" podUID="4596f968-3a01-444a-ab86-94d696af4337" containerName="payload-pre-processing" containerID="cri-o://49ef17b97521f4c2ae816d819053cfe9d6b73c509c3cee243586d3653a1d67d8" gracePeriod=30
Sep 07 10:28:45.919989 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:45.919925    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"
Sep 07 10:28:45.954177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:45.954145    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4596f968-3a01-444a-ab86-94d696af4337-plugins-config-volume\") pod \"4596f968-3a01-444a-ab86-94d696af4337\" (UID: \"4596f968-3a01-444a-ab86-94d696af4337\") "
Sep 07 10:28:45.954340 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:45.954193    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzjkb\" (UniqueName: \"kubernetes.io/projected/4596f968-3a01-444a-ab86-94d696af4337-kube-api-access-qzjkb\") pod \"4596f968-3a01-444a-ab86-94d696af4337\" (UID: \"4596f968-3a01-444a-ab86-94d696af4337\") "
Sep 07 10:28:45.954577 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:45.954551    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4596f968-3a01-444a-ab86-94d696af4337-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4596f968-3a01-444a-ab86-94d696af4337" (UID: "4596f968-3a01-444a-ab86-94d696af4337"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:28:45.956309 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:45.956286    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4596f968-3a01-444a-ab86-94d696af4337-kube-api-access-qzjkb" (OuterVolumeSpecName: "kube-api-access-qzjkb") pod "4596f968-3a01-444a-ab86-94d696af4337" (UID: "4596f968-3a01-444a-ab86-94d696af4337"). InnerVolumeSpecName "kube-api-access-qzjkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:28:46.055348 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:46.055310    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4596f968-3a01-444a-ab86-94d696af4337-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:28:46.055348 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:46.055342    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzjkb\" (UniqueName: \"kubernetes.io/projected/4596f968-3a01-444a-ab86-94d696af4337-kube-api-access-qzjkb\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:28:46.273146 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:46.273117    2562 generic.go:356] "Generic (PLEG): container finished" podID="4596f968-3a01-444a-ab86-94d696af4337" containerID="49ef17b97521f4c2ae816d819053cfe9d6b73c509c3cee243586d3653a1d67d8" exitCode=0
Sep 07 10:28:46.273333 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:46.273174    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"
Sep 07 10:28:46.273333 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:46.273193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll" event={"ID":"4596f968-3a01-444a-ab86-94d696af4337","Type":"ContainerDied","Data":"49ef17b97521f4c2ae816d819053cfe9d6b73c509c3cee243586d3653a1d67d8"}
Sep 07 10:28:46.273333 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:46.273252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll" event={"ID":"4596f968-3a01-444a-ab86-94d696af4337","Type":"ContainerDied","Data":"61335c2c88a03d1fadf14b30d394c04463aa1c9f53d5de5bcc5f9e822d229962"}
Sep 07 10:28:46.273333 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:46.273279    2562 scope.go:117] "RemoveContainer" containerID="49ef17b97521f4c2ae816d819053cfe9d6b73c509c3cee243586d3653a1d67d8"
Sep 07 10:28:46.282222 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:46.282203    2562 scope.go:117] "RemoveContainer" containerID="49ef17b97521f4c2ae816d819053cfe9d6b73c509c3cee243586d3653a1d67d8"
Sep 07 10:28:46.282519 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:28:46.282500    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ef17b97521f4c2ae816d819053cfe9d6b73c509c3cee243586d3653a1d67d8\": container with ID starting with 49ef17b97521f4c2ae816d819053cfe9d6b73c509c3cee243586d3653a1d67d8 not found: ID does not exist" containerID="49ef17b97521f4c2ae816d819053cfe9d6b73c509c3cee243586d3653a1d67d8"
Sep 07 10:28:46.282567 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:46.282528    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ef17b97521f4c2ae816d819053cfe9d6b73c509c3cee243586d3653a1d67d8"} err="failed to get container status \"49ef17b97521f4c2ae816d819053cfe9d6b73c509c3cee243586d3653a1d67d8\": rpc error: code = NotFound desc = could not find container \"49ef17b97521f4c2ae816d819053cfe9d6b73c509c3cee243586d3653a1d67d8\": container with ID starting with 49ef17b97521f4c2ae816d819053cfe9d6b73c509c3cee243586d3653a1d67d8 not found: ID does not exist"
Sep 07 10:28:46.289391 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:46.289369    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"]
Sep 07 10:28:46.295402 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:46.295380    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f17c1f-797c45955d-zzzll"]
Sep 07 10:28:47.221943 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:47.221911    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"
Sep 07 10:28:47.222391 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:47.221989    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"
Sep 07 10:28:47.964141 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:47.964107    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4596f968-3a01-444a-ab86-94d696af4337" path="/var/lib/kubelet/pods/4596f968-3a01-444a-ab86-94d696af4337/volumes"
Sep 07 10:28:58.027021 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:58.026980    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"]
Sep 07 10:28:58.027478 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:58.027365    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4596f968-3a01-444a-ab86-94d696af4337" containerName="payload-pre-processing"
Sep 07 10:28:58.027478 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:58.027375    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4596f968-3a01-444a-ab86-94d696af4337" containerName="payload-pre-processing"
Sep 07 10:28:58.027478 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:58.027446    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4596f968-3a01-444a-ab86-94d696af4337" containerName="payload-pre-processing"
Sep 07 10:28:58.031624 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:58.031606    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"
Sep 07 10:28:58.035175 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:58.035149    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-0a577c\""
Sep 07 10:28:58.035175 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:58.035176    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-0a577c-dockercfg-ssl48\""
Sep 07 10:28:58.043063 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:58.043036    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"]
Sep 07 10:28:58.171775 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:58.171732    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkwkb\" (UniqueName: \"kubernetes.io/projected/9901089c-ffc7-434b-8ab1-246e6d33c408-kube-api-access-zkwkb\") pod \"payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw\" (UID: \"9901089c-ffc7-434b-8ab1-246e6d33c408\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"
Sep 07 10:28:58.171976 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:58.171875    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9901089c-ffc7-434b-8ab1-246e6d33c408-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw\" (UID: \"9901089c-ffc7-434b-8ab1-246e6d33c408\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"
Sep 07 10:28:58.273128 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:58.273081    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9901089c-ffc7-434b-8ab1-246e6d33c408-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw\" (UID: \"9901089c-ffc7-434b-8ab1-246e6d33c408\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"
Sep 07 10:28:58.273330 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:58.273222    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkwkb\" (UniqueName: \"kubernetes.io/projected/9901089c-ffc7-434b-8ab1-246e6d33c408-kube-api-access-zkwkb\") pod \"payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw\" (UID: \"9901089c-ffc7-434b-8ab1-246e6d33c408\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"
Sep 07 10:28:58.273847 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:58.273825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9901089c-ffc7-434b-8ab1-246e6d33c408-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw\" (UID: \"9901089c-ffc7-434b-8ab1-246e6d33c408\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"
Sep 07 10:28:58.281684 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:58.281609    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkwkb\" (UniqueName: \"kubernetes.io/projected/9901089c-ffc7-434b-8ab1-246e6d33c408-kube-api-access-zkwkb\") pod \"payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw\" (UID: \"9901089c-ffc7-434b-8ab1-246e6d33c408\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"
Sep 07 10:28:58.344852 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:58.344817    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"
Sep 07 10:28:58.627431 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:58.627404    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"]
Sep 07 10:28:58.629021 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:28:58.628993    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9901089c_ffc7_434b_8ab1_246e6d33c408.slice/crio-7aff9b2ab8d60452abaccda0d94481431502b80871052cf1ff84be3de8740a20 WatchSource:0}: Error finding container 7aff9b2ab8d60452abaccda0d94481431502b80871052cf1ff84be3de8740a20: Status 404 returned error can't find the container with id 7aff9b2ab8d60452abaccda0d94481431502b80871052cf1ff84be3de8740a20
Sep 07 10:28:59.349634 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:59.349594    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw" event={"ID":"9901089c-ffc7-434b-8ab1-246e6d33c408","Type":"ContainerStarted","Data":"1008fba885f5cf16eeb8d1353df0e5531955b7e8e2027127d6211385b1747ca0"}
Sep 07 10:28:59.350036 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:59.349659    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw" event={"ID":"9901089c-ffc7-434b-8ab1-246e6d33c408","Type":"ContainerStarted","Data":"7aff9b2ab8d60452abaccda0d94481431502b80871052cf1ff84be3de8740a20"}
Sep 07 10:28:59.350036 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:59.349688    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"
Sep 07 10:28:59.379903 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:28:59.379845    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw" podStartSLOduration=1.3798276170000001 podStartE2EDuration="1.379827617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:28:59.378947201 +0000 UTC m=+1041.876637148" watchObservedRunningTime="2026-09-07 10:28:59.379827617 +0000 UTC m=+1041.877517546"
Sep 07 10:29:10.357107 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:10.357076    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"
Sep 07 10:29:15.469380 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.469347    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"]
Sep 07 10:29:15.469776 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.469584    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf" podUID="daff2afc-cce6-4753-bed8-a2efbf4e984b" containerName="maas-api" containerID="cri-o://9b8c2ea7c09f0a3d740d6d8c0a6be3c6648eaa8cb7c735ce2828af13f16f69a5" gracePeriod=30
Sep 07 10:29:15.524944 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.524907    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"]
Sep 07 10:29:15.525158 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.525129    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t" podUID="90b62764-e517-4856-94ff-88016d8af33c" containerName="payload-processing" containerID="cri-o://6d4b75ab61efaf3d64a7246774265ad12c64f932e9d88112d1e5f5ece6c02922" gracePeriod=30
Sep 07 10:29:15.531256 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.531224    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"]
Sep 07 10:29:15.531481 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.531455    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7" podUID="964260c1-7f5a-4f85-9586-4e7e4924fd48" containerName="payload-pre-processing" containerID="cri-o://d5b2f26001a0b711a202fff86a74c992fb6ada94e55853b51f028cf210eb2ed8" gracePeriod=30
Sep 07 10:29:15.777047 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.777013    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:29:15.784450 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.784421    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"
Sep 07 10:29:15.821012 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.820987    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"
Sep 07 10:29:15.930572 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.930530    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daff2afc-cce6-4753-bed8-a2efbf4e984b-openshift-service-ca\") pod \"daff2afc-cce6-4753-bed8-a2efbf4e984b\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") "
Sep 07 10:29:15.930835 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.930595    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c6bb\" (UniqueName: \"kubernetes.io/projected/964260c1-7f5a-4f85-9586-4e7e4924fd48-kube-api-access-9c6bb\") pod \"964260c1-7f5a-4f85-9586-4e7e4924fd48\" (UID: \"964260c1-7f5a-4f85-9586-4e7e4924fd48\") "
Sep 07 10:29:15.930835 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.930650    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlrnj\" (UniqueName: \"kubernetes.io/projected/daff2afc-cce6-4753-bed8-a2efbf4e984b-kube-api-access-vlrnj\") pod \"daff2afc-cce6-4753-bed8-a2efbf4e984b\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") "
Sep 07 10:29:15.930835 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.930692    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daff2afc-cce6-4753-bed8-a2efbf4e984b-metrics-certs\") pod \"daff2afc-cce6-4753-bed8-a2efbf4e984b\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") "
Sep 07 10:29:15.930835 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.930714    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964260c1-7f5a-4f85-9586-4e7e4924fd48-plugins-config-volume\") pod \"964260c1-7f5a-4f85-9586-4e7e4924fd48\" (UID: \"964260c1-7f5a-4f85-9586-4e7e4924fd48\") "
Sep 07 10:29:15.930835 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.930757    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daff2afc-cce6-4753-bed8-a2efbf4e984b-maas-api-tls\") pod \"daff2afc-cce6-4753-bed8-a2efbf4e984b\" (UID: \"daff2afc-cce6-4753-bed8-a2efbf4e984b\") "
Sep 07 10:29:15.930835 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.930834    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfcr9\" (UniqueName: \"kubernetes.io/projected/90b62764-e517-4856-94ff-88016d8af33c-kube-api-access-jfcr9\") pod \"90b62764-e517-4856-94ff-88016d8af33c\" (UID: \"90b62764-e517-4856-94ff-88016d8af33c\") "
Sep 07 10:29:15.931177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.930858    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b62764-e517-4856-94ff-88016d8af33c-plugins-config-volume\") pod \"90b62764-e517-4856-94ff-88016d8af33c\" (UID: \"90b62764-e517-4856-94ff-88016d8af33c\") "
Sep 07 10:29:15.931177 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.931100    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daff2afc-cce6-4753-bed8-a2efbf4e984b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "daff2afc-cce6-4753-bed8-a2efbf4e984b" (UID: "daff2afc-cce6-4753-bed8-a2efbf4e984b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:29:15.931278 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.931180    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daff2afc-cce6-4753-bed8-a2efbf4e984b-openshift-service-ca\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:29:15.931278 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.931170    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964260c1-7f5a-4f85-9586-4e7e4924fd48-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "964260c1-7f5a-4f85-9586-4e7e4924fd48" (UID: "964260c1-7f5a-4f85-9586-4e7e4924fd48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:29:15.931442 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.931417    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b62764-e517-4856-94ff-88016d8af33c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90b62764-e517-4856-94ff-88016d8af33c" (UID: "90b62764-e517-4856-94ff-88016d8af33c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:29:15.932964 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.932934    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daff2afc-cce6-4753-bed8-a2efbf4e984b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "daff2afc-cce6-4753-bed8-a2efbf4e984b" (UID: "daff2afc-cce6-4753-bed8-a2efbf4e984b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:29:15.933339 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.933317    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964260c1-7f5a-4f85-9586-4e7e4924fd48-kube-api-access-9c6bb" (OuterVolumeSpecName: "kube-api-access-9c6bb") pod "964260c1-7f5a-4f85-9586-4e7e4924fd48" (UID: "964260c1-7f5a-4f85-9586-4e7e4924fd48"). InnerVolumeSpecName "kube-api-access-9c6bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:29:15.933452 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.933343    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b62764-e517-4856-94ff-88016d8af33c-kube-api-access-jfcr9" (OuterVolumeSpecName: "kube-api-access-jfcr9") pod "90b62764-e517-4856-94ff-88016d8af33c" (UID: "90b62764-e517-4856-94ff-88016d8af33c"). InnerVolumeSpecName "kube-api-access-jfcr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:29:15.933452 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.933387    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daff2afc-cce6-4753-bed8-a2efbf4e984b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "daff2afc-cce6-4753-bed8-a2efbf4e984b" (UID: "daff2afc-cce6-4753-bed8-a2efbf4e984b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:29:15.933452 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:15.933399    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daff2afc-cce6-4753-bed8-a2efbf4e984b-kube-api-access-vlrnj" (OuterVolumeSpecName: "kube-api-access-vlrnj") pod "daff2afc-cce6-4753-bed8-a2efbf4e984b" (UID: "daff2afc-cce6-4753-bed8-a2efbf4e984b"). InnerVolumeSpecName "kube-api-access-vlrnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:29:16.032172 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.032099    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlrnj\" (UniqueName: \"kubernetes.io/projected/daff2afc-cce6-4753-bed8-a2efbf4e984b-kube-api-access-vlrnj\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:29:16.032172 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.032125    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daff2afc-cce6-4753-bed8-a2efbf4e984b-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:29:16.032172 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.032135    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964260c1-7f5a-4f85-9586-4e7e4924fd48-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:29:16.032172 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.032143    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daff2afc-cce6-4753-bed8-a2efbf4e984b-maas-api-tls\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:29:16.032172 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.032151    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfcr9\" (UniqueName: \"kubernetes.io/projected/90b62764-e517-4856-94ff-88016d8af33c-kube-api-access-jfcr9\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:29:16.032172 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.032159    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b62764-e517-4856-94ff-88016d8af33c-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:29:16.032172 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.032167    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c6bb\" (UniqueName: \"kubernetes.io/projected/964260c1-7f5a-4f85-9586-4e7e4924fd48-kube-api-access-9c6bb\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:29:16.446027 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.445931    2562 generic.go:356] "Generic (PLEG): container finished" podID="964260c1-7f5a-4f85-9586-4e7e4924fd48" containerID="d5b2f26001a0b711a202fff86a74c992fb6ada94e55853b51f028cf210eb2ed8" exitCode=0
Sep 07 10:29:16.446027 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.445978    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"
Sep 07 10:29:16.446246 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.446016    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7" event={"ID":"964260c1-7f5a-4f85-9586-4e7e4924fd48","Type":"ContainerDied","Data":"d5b2f26001a0b711a202fff86a74c992fb6ada94e55853b51f028cf210eb2ed8"}
Sep 07 10:29:16.446246 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.446079    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7" event={"ID":"964260c1-7f5a-4f85-9586-4e7e4924fd48","Type":"ContainerDied","Data":"2fa6b67de485d28bbbd13017b5657c5b7fc97292a6ddb8f29bcbbc8262242db4"}
Sep 07 10:29:16.446246 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.446113    2562 scope.go:117] "RemoveContainer" containerID="d5b2f26001a0b711a202fff86a74c992fb6ada94e55853b51f028cf210eb2ed8"
Sep 07 10:29:16.447509 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.447487    2562 generic.go:356] "Generic (PLEG): container finished" podID="daff2afc-cce6-4753-bed8-a2efbf4e984b" containerID="9b8c2ea7c09f0a3d740d6d8c0a6be3c6648eaa8cb7c735ce2828af13f16f69a5" exitCode=0
Sep 07 10:29:16.447628 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.447526    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"
Sep 07 10:29:16.447628 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.447578    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf" event={"ID":"daff2afc-cce6-4753-bed8-a2efbf4e984b","Type":"ContainerDied","Data":"9b8c2ea7c09f0a3d740d6d8c0a6be3c6648eaa8cb7c735ce2828af13f16f69a5"}
Sep 07 10:29:16.447749 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.447643    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf" event={"ID":"daff2afc-cce6-4753-bed8-a2efbf4e984b","Type":"ContainerDied","Data":"33f57981ad025a2b7539cfca9995f9417ecbbcd96851fd795412a124d92c12ce"}
Sep 07 10:29:16.449270 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.449187    2562 generic.go:356] "Generic (PLEG): container finished" podID="90b62764-e517-4856-94ff-88016d8af33c" containerID="6d4b75ab61efaf3d64a7246774265ad12c64f932e9d88112d1e5f5ece6c02922" exitCode=0
Sep 07 10:29:16.449270 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.449223    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t" event={"ID":"90b62764-e517-4856-94ff-88016d8af33c","Type":"ContainerDied","Data":"6d4b75ab61efaf3d64a7246774265ad12c64f932e9d88112d1e5f5ece6c02922"}
Sep 07 10:29:16.449408 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.449271    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t" event={"ID":"90b62764-e517-4856-94ff-88016d8af33c","Type":"ContainerDied","Data":"f5fdd44247ad87e1de2944a6a6eb38c561726e28f2cac0eedb2df3ee3bcf4d93"}
Sep 07 10:29:16.449408 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.449228    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"
Sep 07 10:29:16.456414 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.456393    2562 scope.go:117] "RemoveContainer" containerID="d5b2f26001a0b711a202fff86a74c992fb6ada94e55853b51f028cf210eb2ed8"
Sep 07 10:29:16.456701 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:29:16.456673    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b2f26001a0b711a202fff86a74c992fb6ada94e55853b51f028cf210eb2ed8\": container with ID starting with d5b2f26001a0b711a202fff86a74c992fb6ada94e55853b51f028cf210eb2ed8 not found: ID does not exist" containerID="d5b2f26001a0b711a202fff86a74c992fb6ada94e55853b51f028cf210eb2ed8"
Sep 07 10:29:16.456785 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.456713    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b2f26001a0b711a202fff86a74c992fb6ada94e55853b51f028cf210eb2ed8"} err="failed to get container status \"d5b2f26001a0b711a202fff86a74c992fb6ada94e55853b51f028cf210eb2ed8\": rpc error: code = NotFound desc = could not find container \"d5b2f26001a0b711a202fff86a74c992fb6ada94e55853b51f028cf210eb2ed8\": container with ID starting with d5b2f26001a0b711a202fff86a74c992fb6ada94e55853b51f028cf210eb2ed8 not found: ID does not exist"
Sep 07 10:29:16.456785 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.456736    2562 scope.go:117] "RemoveContainer" containerID="9b8c2ea7c09f0a3d740d6d8c0a6be3c6648eaa8cb7c735ce2828af13f16f69a5"
Sep 07 10:29:16.464945 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.464924    2562 scope.go:117] "RemoveContainer" containerID="9b8c2ea7c09f0a3d740d6d8c0a6be3c6648eaa8cb7c735ce2828af13f16f69a5"
Sep 07 10:29:16.465333 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:29:16.465306    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8c2ea7c09f0a3d740d6d8c0a6be3c6648eaa8cb7c735ce2828af13f16f69a5\": container with ID starting with 9b8c2ea7c09f0a3d740d6d8c0a6be3c6648eaa8cb7c735ce2828af13f16f69a5 not found: ID does not exist" containerID="9b8c2ea7c09f0a3d740d6d8c0a6be3c6648eaa8cb7c735ce2828af13f16f69a5"
Sep 07 10:29:16.465401 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.465344    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8c2ea7c09f0a3d740d6d8c0a6be3c6648eaa8cb7c735ce2828af13f16f69a5"} err="failed to get container status \"9b8c2ea7c09f0a3d740d6d8c0a6be3c6648eaa8cb7c735ce2828af13f16f69a5\": rpc error: code = NotFound desc = could not find container \"9b8c2ea7c09f0a3d740d6d8c0a6be3c6648eaa8cb7c735ce2828af13f16f69a5\": container with ID starting with 9b8c2ea7c09f0a3d740d6d8c0a6be3c6648eaa8cb7c735ce2828af13f16f69a5 not found: ID does not exist"
Sep 07 10:29:16.465401 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.465368    2562 scope.go:117] "RemoveContainer" containerID="6d4b75ab61efaf3d64a7246774265ad12c64f932e9d88112d1e5f5ece6c02922"
Sep 07 10:29:16.467262 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.467244    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"]
Sep 07 10:29:16.471407 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.471387    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2c28e354-6c99558fb9-zff5t"]
Sep 07 10:29:16.474063 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.473942    2562 scope.go:117] "RemoveContainer" containerID="6d4b75ab61efaf3d64a7246774265ad12c64f932e9d88112d1e5f5ece6c02922"
Sep 07 10:29:16.474248 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:29:16.474229    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d4b75ab61efaf3d64a7246774265ad12c64f932e9d88112d1e5f5ece6c02922\": container with ID starting with 6d4b75ab61efaf3d64a7246774265ad12c64f932e9d88112d1e5f5ece6c02922 not found: ID does not exist" containerID="6d4b75ab61efaf3d64a7246774265ad12c64f932e9d88112d1e5f5ece6c02922"
Sep 07 10:29:16.474292 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.474254    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d4b75ab61efaf3d64a7246774265ad12c64f932e9d88112d1e5f5ece6c02922"} err="failed to get container status \"6d4b75ab61efaf3d64a7246774265ad12c64f932e9d88112d1e5f5ece6c02922\": rpc error: code = NotFound desc = could not find container \"6d4b75ab61efaf3d64a7246774265ad12c64f932e9d88112d1e5f5ece6c02922\": container with ID starting with 6d4b75ab61efaf3d64a7246774265ad12c64f932e9d88112d1e5f5ece6c02922 not found: ID does not exist"
Sep 07 10:29:16.482503 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.482473    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"]
Sep 07 10:29:16.485126 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.484988    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2c28e354-66998b4cc4-mhgzf"]
Sep 07 10:29:16.493038 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.493019    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"]
Sep 07 10:29:16.496958 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:16.496939    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2c28e354-544c9849f8-4tkn7"]
Sep 07 10:29:17.964413 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:17.964379    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90b62764-e517-4856-94ff-88016d8af33c" path="/var/lib/kubelet/pods/90b62764-e517-4856-94ff-88016d8af33c/volumes"
Sep 07 10:29:17.964811 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:17.964728    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="964260c1-7f5a-4f85-9586-4e7e4924fd48" path="/var/lib/kubelet/pods/964260c1-7f5a-4f85-9586-4e7e4924fd48/volumes"
Sep 07 10:29:17.965096 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:17.965083    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daff2afc-cce6-4753-bed8-a2efbf4e984b" path="/var/lib/kubelet/pods/daff2afc-cce6-4753-bed8-a2efbf4e984b/volumes"
Sep 07 10:29:25.700910 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.700868    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"]
Sep 07 10:29:25.702924 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.702474    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b62764-e517-4856-94ff-88016d8af33c" containerName="payload-processing"
Sep 07 10:29:25.703106 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.703085    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b62764-e517-4856-94ff-88016d8af33c" containerName="payload-processing"
Sep 07 10:29:25.703220 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.703204    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daff2afc-cce6-4753-bed8-a2efbf4e984b" containerName="maas-api"
Sep 07 10:29:25.703317 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.703307    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="daff2afc-cce6-4753-bed8-a2efbf4e984b" containerName="maas-api"
Sep 07 10:29:25.703429 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.703417    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="964260c1-7f5a-4f85-9586-4e7e4924fd48" containerName="payload-pre-processing"
Sep 07 10:29:25.703518 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.703508    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="964260c1-7f5a-4f85-9586-4e7e4924fd48" containerName="payload-pre-processing"
Sep 07 10:29:25.703865 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.703846    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="90b62764-e517-4856-94ff-88016d8af33c" containerName="payload-processing"
Sep 07 10:29:25.703983 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.703970    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="daff2afc-cce6-4753-bed8-a2efbf4e984b" containerName="maas-api"
Sep 07 10:29:25.704087 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.704075    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="964260c1-7f5a-4f85-9586-4e7e4924fd48" containerName="payload-pre-processing"
Sep 07 10:29:25.714536 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.713058    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:25.714930 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.714905    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"]
Sep 07 10:29:25.715814 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.715790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20aa2d9-d7db-4d07-a263-f393b64403f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:25.716059 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.716039    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b20aa2d9-d7db-4d07-a263-f393b64403f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:25.716204 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.716188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20aa2d9-d7db-4d07-a263-f393b64403f0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:25.716295 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.716275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8ck9\" (UniqueName: \"kubernetes.io/projected/b20aa2d9-d7db-4d07-a263-f393b64403f0-kube-api-access-s8ck9\") pod \"maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:25.716349 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.716156    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-93e41e-serving-cert\""
Sep 07 10:29:25.817023 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.816989    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20aa2d9-d7db-4d07-a263-f393b64403f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:25.817230 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.817061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b20aa2d9-d7db-4d07-a263-f393b64403f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:25.817230 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.817081    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20aa2d9-d7db-4d07-a263-f393b64403f0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:25.817230 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.817096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8ck9\" (UniqueName: \"kubernetes.io/projected/b20aa2d9-d7db-4d07-a263-f393b64403f0-kube-api-access-s8ck9\") pod \"maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:25.817230 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:29:25.817220    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-serving-cert: secret "maas-api-e2e-shared-b-w3-93e41e-serving-cert" not found
Sep 07 10:29:25.817436 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:29:25.817302    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20aa2d9-d7db-4d07-a263-f393b64403f0-maas-api-tls podName:b20aa2d9-d7db-4d07-a263-f393b64403f0 nodeName:}" failed. No retries permitted until 2026-09-07 10:29:26.317280814 +0000 UTC m=+1068.814970701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b20aa2d9-d7db-4d07-a263-f393b64403f0-maas-api-tls") pod "maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw" (UID: "b20aa2d9-d7db-4d07-a263-f393b64403f0") : secret "maas-api-e2e-shared-b-w3-93e41e-serving-cert" not found
Sep 07 10:29:25.817772 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.817733    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20aa2d9-d7db-4d07-a263-f393b64403f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:25.819501 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.819472    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20aa2d9-d7db-4d07-a263-f393b64403f0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:25.827621 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:25.827596    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8ck9\" (UniqueName: \"kubernetes.io/projected/b20aa2d9-d7db-4d07-a263-f393b64403f0-kube-api-access-s8ck9\") pod \"maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:26.316348 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.316312    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"]
Sep 07 10:29:26.319958 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.319939    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"
Sep 07 10:29:26.322486 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.322460    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-93e41e\""
Sep 07 10:29:26.322618 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.322557    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920d81be-e890-43ad-81b2-dd8460395676-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r\" (UID: \"920d81be-e890-43ad-81b2-dd8460395676\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"
Sep 07 10:29:26.322667 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.322636    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b20aa2d9-d7db-4d07-a263-f393b64403f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:26.322714 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.322666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngj9n\" (UniqueName: \"kubernetes.io/projected/920d81be-e890-43ad-81b2-dd8460395676-kube-api-access-ngj9n\") pod \"payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r\" (UID: \"920d81be-e890-43ad-81b2-dd8460395676\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"
Sep 07 10:29:26.322799 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.322724    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-93e41e-dockercfg-m9wwn\""
Sep 07 10:29:26.325606 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.325583    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b20aa2d9-d7db-4d07-a263-f393b64403f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:26.325728 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.325704    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:26.333188 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.333159    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"]
Sep 07 10:29:26.424034 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.424000    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920d81be-e890-43ad-81b2-dd8460395676-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r\" (UID: \"920d81be-e890-43ad-81b2-dd8460395676\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"
Sep 07 10:29:26.424221 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.424050    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngj9n\" (UniqueName: \"kubernetes.io/projected/920d81be-e890-43ad-81b2-dd8460395676-kube-api-access-ngj9n\") pod \"payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r\" (UID: \"920d81be-e890-43ad-81b2-dd8460395676\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"
Sep 07 10:29:26.424715 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.424692    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920d81be-e890-43ad-81b2-dd8460395676-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r\" (UID: \"920d81be-e890-43ad-81b2-dd8460395676\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"
Sep 07 10:29:26.431905 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.431881    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngj9n\" (UniqueName: \"kubernetes.io/projected/920d81be-e890-43ad-81b2-dd8460395676-kube-api-access-ngj9n\") pod \"payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r\" (UID: \"920d81be-e890-43ad-81b2-dd8460395676\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"
Sep 07 10:29:26.606746 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.606720    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"]
Sep 07 10:29:26.608377 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:29:26.608349    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20aa2d9_d7db_4d07_a263_f393b64403f0.slice/crio-d54288d73e3ad535986b3d0f2e6d0a2bf4b64ee1bef8f6b671c561e31fb7ed2a WatchSource:0}: Error finding container d54288d73e3ad535986b3d0f2e6d0a2bf4b64ee1bef8f6b671c561e31fb7ed2a: Status 404 returned error can't find the container with id d54288d73e3ad535986b3d0f2e6d0a2bf4b64ee1bef8f6b671c561e31fb7ed2a
Sep 07 10:29:26.658284 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.658253    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"
Sep 07 10:29:26.931147 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:26.931038    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"]
Sep 07 10:29:26.938855 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:29:26.938827    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920d81be_e890_43ad_81b2_dd8460395676.slice/crio-a9bfcf05c8c7c5a03cb48e1b67cece64ae7b9a396fbd05bd360e4b9ab876d521 WatchSource:0}: Error finding container a9bfcf05c8c7c5a03cb48e1b67cece64ae7b9a396fbd05bd360e4b9ab876d521: Status 404 returned error can't find the container with id a9bfcf05c8c7c5a03cb48e1b67cece64ae7b9a396fbd05bd360e4b9ab876d521
Sep 07 10:29:27.518925 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:27.518875    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw" event={"ID":"b20aa2d9-d7db-4d07-a263-f393b64403f0","Type":"ContainerStarted","Data":"c8c005ad4437591401eaf546a8ff71b2b24caf4fb81d95eda576247127dd3c33"}
Sep 07 10:29:27.519144 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:27.518952    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw" event={"ID":"b20aa2d9-d7db-4d07-a263-f393b64403f0","Type":"ContainerStarted","Data":"d54288d73e3ad535986b3d0f2e6d0a2bf4b64ee1bef8f6b671c561e31fb7ed2a"}
Sep 07 10:29:27.519144 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:27.518987    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:27.520418 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:27.520394    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r" event={"ID":"920d81be-e890-43ad-81b2-dd8460395676","Type":"ContainerStarted","Data":"cabd7d2644cace78f861523074c4fcd6db2ae90e1f21e528cec1d4a7efaa8ed7"}
Sep 07 10:29:27.520512 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:27.520432    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r" event={"ID":"920d81be-e890-43ad-81b2-dd8460395676","Type":"ContainerStarted","Data":"a9bfcf05c8c7c5a03cb48e1b67cece64ae7b9a396fbd05bd360e4b9ab876d521"}
Sep 07 10:29:27.520571 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:27.520520    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"
Sep 07 10:29:27.536711 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:27.536081    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw" podStartSLOduration=2.255186812 podStartE2EDuration="2.536063255s" totalImagesPullingTime="280.876443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:29:27.535848849 +0000 UTC m=+1070.033538774" watchObservedRunningTime="2026-09-07 10:29:27.536063255 +0000 UTC m=+1070.033753164"
Sep 07 10:29:27.554851 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:27.554794    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r" podStartSLOduration=1.554755941 podStartE2EDuration="1.554755941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:29:27.552417019 +0000 UTC m=+1070.050106932" watchObservedRunningTime="2026-09-07 10:29:27.554755941 +0000 UTC m=+1070.052445852"
Sep 07 10:29:33.530802 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:33.530720    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:29:38.527711 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:38.527676    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"
Sep 07 10:29:40.644628 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:40.644567    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"]
Sep 07 10:29:40.650049 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:40.650024    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:40.652557 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:40.652493    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-7bdcc7-serving-cert\""
Sep 07 10:29:40.659236 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:40.659205    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"]
Sep 07 10:29:40.745921 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:40.745885    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4800952d-0d92-43bc-9996-5f380634da57-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:40.746082 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:40.745927    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8l2l\" (UniqueName: \"kubernetes.io/projected/4800952d-0d92-43bc-9996-5f380634da57-kube-api-access-x8l2l\") pod \"maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:40.746082 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:40.746032    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4800952d-0d92-43bc-9996-5f380634da57-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:40.746172 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:40.746122    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4800952d-0d92-43bc-9996-5f380634da57-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:40.847309 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:40.847265    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4800952d-0d92-43bc-9996-5f380634da57-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:40.847309 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:40.847303    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8l2l\" (UniqueName: \"kubernetes.io/projected/4800952d-0d92-43bc-9996-5f380634da57-kube-api-access-x8l2l\") pod \"maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:40.847570 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:40.847331    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4800952d-0d92-43bc-9996-5f380634da57-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:40.847570 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:40.847382    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4800952d-0d92-43bc-9996-5f380634da57-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:40.847570 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:29:40.847432    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-serving-cert: secret "maas-api-e2e-shared-a-w2-7bdcc7-serving-cert" not found
Sep 07 10:29:40.847570 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:29:40.847498    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4800952d-0d92-43bc-9996-5f380634da57-maas-api-tls podName:4800952d-0d92-43bc-9996-5f380634da57 nodeName:}" failed. No retries permitted until 2026-09-07 10:29:41.347478268 +0000 UTC m=+1083.845168166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4800952d-0d92-43bc-9996-5f380634da57-maas-api-tls") pod "maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk" (UID: "4800952d-0d92-43bc-9996-5f380634da57") : secret "maas-api-e2e-shared-a-w2-7bdcc7-serving-cert" not found
Sep 07 10:29:40.848153 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:40.848129    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4800952d-0d92-43bc-9996-5f380634da57-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:40.849983 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:40.849956    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4800952d-0d92-43bc-9996-5f380634da57-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:40.855994 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:40.855966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8l2l\" (UniqueName: \"kubernetes.io/projected/4800952d-0d92-43bc-9996-5f380634da57-kube-api-access-x8l2l\") pod \"maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:41.353004 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:41.352967    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4800952d-0d92-43bc-9996-5f380634da57-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:41.355405 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:41.355384    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4800952d-0d92-43bc-9996-5f380634da57-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:41.536798 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:41.536747    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"]
Sep 07 10:29:41.540538 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:41.540491    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"
Sep 07 10:29:41.543659 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:41.543622    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-7bdcc7\""
Sep 07 10:29:41.544210 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:41.544193    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-7bdcc7-dockercfg-mkrfw\""
Sep 07 10:29:41.553176 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:41.553152    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"]
Sep 07 10:29:41.570664 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:41.570636    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:41.656132 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:41.655803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26nhh\" (UniqueName: \"kubernetes.io/projected/22dfb45e-95e7-4293-acef-03fde4269a51-kube-api-access-26nhh\") pod \"payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9\" (UID: \"22dfb45e-95e7-4293-acef-03fde4269a51\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"
Sep 07 10:29:41.656132 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:41.655884    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22dfb45e-95e7-4293-acef-03fde4269a51-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9\" (UID: \"22dfb45e-95e7-4293-acef-03fde4269a51\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"
Sep 07 10:29:41.756743 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:41.756708    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22dfb45e-95e7-4293-acef-03fde4269a51-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9\" (UID: \"22dfb45e-95e7-4293-acef-03fde4269a51\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"
Sep 07 10:29:41.756934 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:41.756834    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26nhh\" (UniqueName: \"kubernetes.io/projected/22dfb45e-95e7-4293-acef-03fde4269a51-kube-api-access-26nhh\") pod \"payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9\" (UID: \"22dfb45e-95e7-4293-acef-03fde4269a51\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"
Sep 07 10:29:41.757445 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:41.757422    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22dfb45e-95e7-4293-acef-03fde4269a51-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9\" (UID: \"22dfb45e-95e7-4293-acef-03fde4269a51\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"
Sep 07 10:29:41.765003 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:41.764977    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26nhh\" (UniqueName: \"kubernetes.io/projected/22dfb45e-95e7-4293-acef-03fde4269a51-kube-api-access-26nhh\") pod \"payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9\" (UID: \"22dfb45e-95e7-4293-acef-03fde4269a51\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"
Sep 07 10:29:41.848952 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:41.848924    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"]
Sep 07 10:29:41.851053 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:29:41.851022    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4800952d_0d92_43bc_9996_5f380634da57.slice/crio-ed37f96c31df8ab6d1770a3793980d3db5e202e342855d362df8a7bd591272f6 WatchSource:0}: Error finding container ed37f96c31df8ab6d1770a3793980d3db5e202e342855d362df8a7bd591272f6: Status 404 returned error can't find the container with id ed37f96c31df8ab6d1770a3793980d3db5e202e342855d362df8a7bd591272f6
Sep 07 10:29:41.851746 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:41.851728    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"
Sep 07 10:29:42.126810 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:42.126676    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"]
Sep 07 10:29:42.128953 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:29:42.128924    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22dfb45e_95e7_4293_acef_03fde4269a51.slice/crio-02a239fd592bd603632137b702611b433ce73aea3faa8c206dadea6961d8d81a WatchSource:0}: Error finding container 02a239fd592bd603632137b702611b433ce73aea3faa8c206dadea6961d8d81a: Status 404 returned error can't find the container with id 02a239fd592bd603632137b702611b433ce73aea3faa8c206dadea6961d8d81a
Sep 07 10:29:42.607389 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:42.607353    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9" event={"ID":"22dfb45e-95e7-4293-acef-03fde4269a51","Type":"ContainerStarted","Data":"7ca90df97953184c878f8c782c4cf06984eaf43a1ffbaf95912a8aefb152cbb1"}
Sep 07 10:29:42.607588 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:42.607415    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"
Sep 07 10:29:42.607588 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:42.607437    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9" event={"ID":"22dfb45e-95e7-4293-acef-03fde4269a51","Type":"ContainerStarted","Data":"02a239fd592bd603632137b702611b433ce73aea3faa8c206dadea6961d8d81a"}
Sep 07 10:29:42.608949 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:42.608925    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk" event={"ID":"4800952d-0d92-43bc-9996-5f380634da57","Type":"ContainerStarted","Data":"b50a0adfb87cc2a8f64d429b9c1338607267ff11bad67f3ac8ed020347cbfa98"}
Sep 07 10:29:42.609043 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:42.608975    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:42.609043 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:42.608992    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk" event={"ID":"4800952d-0d92-43bc-9996-5f380634da57","Type":"ContainerStarted","Data":"ed37f96c31df8ab6d1770a3793980d3db5e202e342855d362df8a7bd591272f6"}
Sep 07 10:29:42.627548 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:42.627485    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9" podStartSLOduration=1.627469814 podStartE2EDuration="1.627469814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:29:42.625504236 +0000 UTC m=+1085.123194143" watchObservedRunningTime="2026-09-07 10:29:42.627469814 +0000 UTC m=+1085.125159724"
Sep 07 10:29:42.645313 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:42.645260    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk" podStartSLOduration=2.234850506 podStartE2EDuration="2.64524367s" totalImagesPullingTime="410.393164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:29:42.642976692 +0000 UTC m=+1085.140666628" watchObservedRunningTime="2026-09-07 10:29:42.64524367 +0000 UTC m=+1085.142933577"
Sep 07 10:29:48.620923 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:48.620884    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:29:53.616928 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:29:53.616896    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"
Sep 07 10:30:09.484038 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:09.484001    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"]
Sep 07 10:30:09.487412 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:09.487392    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"
Sep 07 10:30:09.489877 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:09.489856    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-319d4c\""
Sep 07 10:30:09.489990 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:09.489941    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-319d4c-dockercfg-8bc5l\""
Sep 07 10:30:09.498957 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:09.498930    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"]
Sep 07 10:30:09.607365 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:09.607331    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37e438a-00cd-4a5e-822a-034d9674a18b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh\" (UID: \"b37e438a-00cd-4a5e-822a-034d9674a18b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"
Sep 07 10:30:09.607534 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:09.607378    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x24pz\" (UniqueName: \"kubernetes.io/projected/b37e438a-00cd-4a5e-822a-034d9674a18b-kube-api-access-x24pz\") pod \"payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh\" (UID: \"b37e438a-00cd-4a5e-822a-034d9674a18b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"
Sep 07 10:30:09.708752 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:09.708704    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37e438a-00cd-4a5e-822a-034d9674a18b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh\" (UID: \"b37e438a-00cd-4a5e-822a-034d9674a18b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"
Sep 07 10:30:09.708963 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:09.708792    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x24pz\" (UniqueName: \"kubernetes.io/projected/b37e438a-00cd-4a5e-822a-034d9674a18b-kube-api-access-x24pz\") pod \"payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh\" (UID: \"b37e438a-00cd-4a5e-822a-034d9674a18b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"
Sep 07 10:30:09.709527 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:09.709508    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37e438a-00cd-4a5e-822a-034d9674a18b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh\" (UID: \"b37e438a-00cd-4a5e-822a-034d9674a18b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"
Sep 07 10:30:09.720250 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:09.720219    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x24pz\" (UniqueName: \"kubernetes.io/projected/b37e438a-00cd-4a5e-822a-034d9674a18b-kube-api-access-x24pz\") pod \"payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh\" (UID: \"b37e438a-00cd-4a5e-822a-034d9674a18b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"
Sep 07 10:30:09.799573 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:09.799483    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"
Sep 07 10:30:10.071256 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:10.071230    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"]
Sep 07 10:30:10.073888 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:30:10.073857    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37e438a_00cd_4a5e_822a_034d9674a18b.slice/crio-0861208983b4fbfd853559e8118e8db71f16839267c854c92a9b17c794b86f82 WatchSource:0}: Error finding container 0861208983b4fbfd853559e8118e8db71f16839267c854c92a9b17c794b86f82: Status 404 returned error can't find the container with id 0861208983b4fbfd853559e8118e8db71f16839267c854c92a9b17c794b86f82
Sep 07 10:30:10.767046 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:10.767011    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh" event={"ID":"b37e438a-00cd-4a5e-822a-034d9674a18b","Type":"ContainerStarted","Data":"9fa4ea6035724a3720cdd6240354d047a040d2954a6458f47a7ee70809457dc7"}
Sep 07 10:30:10.767525 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:10.767071    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh" event={"ID":"b37e438a-00cd-4a5e-822a-034d9674a18b","Type":"ContainerStarted","Data":"0861208983b4fbfd853559e8118e8db71f16839267c854c92a9b17c794b86f82"}
Sep 07 10:30:10.767525 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:10.767146    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"
Sep 07 10:30:10.784304 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:10.784257    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh" podStartSLOduration=1.7842442649999999 podStartE2EDuration="1.784244265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:30:10.78271516 +0000 UTC m=+1113.280405132" watchObservedRunningTime="2026-09-07 10:30:10.784244265 +0000 UTC m=+1113.281934430"
Sep 07 10:30:21.775864 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:21.775830    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"
Sep 07 10:30:37.344309 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:37.344271    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8014-140224719722080"]
Sep 07 10:30:37.347902 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:37.347882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-140224719722080"
Sep 07 10:30:37.350402 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:37.350376    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qpjjb\""
Sep 07 10:30:37.353849 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:37.353823    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8014-140224719722080"]
Sep 07 10:30:37.467058 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:37.467013    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dzww\" (UniqueName: \"kubernetes.io/projected/ff752d38-8754-49ca-8847-56175d92ea0d-kube-api-access-7dzww\") pod \"test-curl-8014-140224719722080\" (UID: \"ff752d38-8754-49ca-8847-56175d92ea0d\") " pod="opendatahub/test-curl-8014-140224719722080"
Sep 07 10:30:37.568019 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:37.567983    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dzww\" (UniqueName: \"kubernetes.io/projected/ff752d38-8754-49ca-8847-56175d92ea0d-kube-api-access-7dzww\") pod \"test-curl-8014-140224719722080\" (UID: \"ff752d38-8754-49ca-8847-56175d92ea0d\") " pod="opendatahub/test-curl-8014-140224719722080"
Sep 07 10:30:37.577543 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:37.577511    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dzww\" (UniqueName: \"kubernetes.io/projected/ff752d38-8754-49ca-8847-56175d92ea0d-kube-api-access-7dzww\") pod \"test-curl-8014-140224719722080\" (UID: \"ff752d38-8754-49ca-8847-56175d92ea0d\") " pod="opendatahub/test-curl-8014-140224719722080"
Sep 07 10:30:37.669961 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:37.669870    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-140224719722080"
Sep 07 10:30:37.938307 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:37.938281    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8014-140224719722080"]
Sep 07 10:30:37.940440 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:30:37.940406    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff752d38_8754_49ca_8847_56175d92ea0d.slice/crio-ad3c4f011428f1d3a13b9e97834711e54678b3ac6a844265726d3f3b07f650f7 WatchSource:0}: Error finding container ad3c4f011428f1d3a13b9e97834711e54678b3ac6a844265726d3f3b07f650f7: Status 404 returned error can't find the container with id ad3c4f011428f1d3a13b9e97834711e54678b3ac6a844265726d3f3b07f650f7
Sep 07 10:30:38.921184 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:38.921142    2562 generic.go:356] "Generic (PLEG): container finished" podID="ff752d38-8754-49ca-8847-56175d92ea0d" containerID="ea078a79164178918864194c552e90533085447e6b501d6f35ddeab9eb2e4a35" exitCode=0
Sep 07 10:30:38.921558 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:38.921221    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8014-140224719722080" event={"ID":"ff752d38-8754-49ca-8847-56175d92ea0d","Type":"ContainerDied","Data":"ea078a79164178918864194c552e90533085447e6b501d6f35ddeab9eb2e4a35"}
Sep 07 10:30:38.921558 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:38.921258    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8014-140224719722080" event={"ID":"ff752d38-8754-49ca-8847-56175d92ea0d","Type":"ContainerStarted","Data":"ad3c4f011428f1d3a13b9e97834711e54678b3ac6a844265726d3f3b07f650f7"}
Sep 07 10:30:40.070039 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:40.070010    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-140224719722080"
Sep 07 10:30:40.103132 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:40.103104    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-8014-140224719722080_ff752d38-8754-49ca-8847-56175d92ea0d/test-curl-8014-140224719722080/0.log"
Sep 07 10:30:40.126128 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:40.126086    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8014-140224719722080"]
Sep 07 10:30:40.129832 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:40.129800    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8014-140224719722080"]
Sep 07 10:30:40.194252 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:40.194161    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dzww\" (UniqueName: \"kubernetes.io/projected/ff752d38-8754-49ca-8847-56175d92ea0d-kube-api-access-7dzww\") pod \"ff752d38-8754-49ca-8847-56175d92ea0d\" (UID: \"ff752d38-8754-49ca-8847-56175d92ea0d\") "
Sep 07 10:30:40.196250 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:40.196223    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff752d38-8754-49ca-8847-56175d92ea0d-kube-api-access-7dzww" (OuterVolumeSpecName: "kube-api-access-7dzww") pod "ff752d38-8754-49ca-8847-56175d92ea0d" (UID: "ff752d38-8754-49ca-8847-56175d92ea0d"). InnerVolumeSpecName "kube-api-access-7dzww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:30:40.295777 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:40.295728    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dzww\" (UniqueName: \"kubernetes.io/projected/ff752d38-8754-49ca-8847-56175d92ea0d-kube-api-access-7dzww\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:30:40.933477 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:40.933448    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-140224719722080"
Sep 07 10:30:40.933734 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:40.933447    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad3c4f011428f1d3a13b9e97834711e54678b3ac6a844265726d3f3b07f650f7"
Sep 07 10:30:40.946709 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:30:40.946665    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8014-140224719722080\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="ff752d38-8754-49ca-8847-56175d92ea0d" pod="opendatahub/test-curl-8014-140224719722080"
Sep 07 10:30:41.966366 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:41.966331    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff752d38-8754-49ca-8847-56175d92ea0d" path="/var/lib/kubelet/pods/ff752d38-8754-49ca-8847-56175d92ea0d/volumes"
Sep 07 10:30:43.417383 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:43.417343    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8014-140224719722656"]
Sep 07 10:30:43.417806 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:43.417790    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff752d38-8754-49ca-8847-56175d92ea0d" containerName="test-curl-8014-140224719722080"
Sep 07 10:30:43.417858 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:43.417809    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff752d38-8754-49ca-8847-56175d92ea0d" containerName="test-curl-8014-140224719722080"
Sep 07 10:30:43.417894 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:43.417881    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff752d38-8754-49ca-8847-56175d92ea0d" containerName="test-curl-8014-140224719722080"
Sep 07 10:30:43.421192 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:43.421173    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-140224719722656"
Sep 07 10:30:43.423632 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:43.423614    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qpjjb\""
Sep 07 10:30:43.427395 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:43.427370    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8014-140224719722656"]
Sep 07 10:30:43.524264 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:43.524223    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgjfd\" (UniqueName: \"kubernetes.io/projected/3454575b-6516-423e-9bbe-ffa807c85fdd-kube-api-access-hgjfd\") pod \"test-curl-8014-140224719722656\" (UID: \"3454575b-6516-423e-9bbe-ffa807c85fdd\") " pod="opendatahub/test-curl-8014-140224719722656"
Sep 07 10:30:43.625792 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:43.625748    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgjfd\" (UniqueName: \"kubernetes.io/projected/3454575b-6516-423e-9bbe-ffa807c85fdd-kube-api-access-hgjfd\") pod \"test-curl-8014-140224719722656\" (UID: \"3454575b-6516-423e-9bbe-ffa807c85fdd\") " pod="opendatahub/test-curl-8014-140224719722656"
Sep 07 10:30:43.634567 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:43.634539    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgjfd\" (UniqueName: \"kubernetes.io/projected/3454575b-6516-423e-9bbe-ffa807c85fdd-kube-api-access-hgjfd\") pod \"test-curl-8014-140224719722656\" (UID: \"3454575b-6516-423e-9bbe-ffa807c85fdd\") " pod="opendatahub/test-curl-8014-140224719722656"
Sep 07 10:30:43.731956 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:43.731913    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-140224719722656"
Sep 07 10:30:44.008215 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:44.008183    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8014-140224719722656"]
Sep 07 10:30:44.009434 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:30:44.009410    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3454575b_6516_423e_9bbe_ffa807c85fdd.slice/crio-a5d8b299947647e1d6e4533822598fc6c683649b799cdfe897029b56b696a11c WatchSource:0}: Error finding container a5d8b299947647e1d6e4533822598fc6c683649b799cdfe897029b56b696a11c: Status 404 returned error can't find the container with id a5d8b299947647e1d6e4533822598fc6c683649b799cdfe897029b56b696a11c
Sep 07 10:30:44.960888 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:44.960851    2562 generic.go:356] "Generic (PLEG): container finished" podID="3454575b-6516-423e-9bbe-ffa807c85fdd" containerID="5364b45d1712154d8b450f5efd5e3ee11357013b7ce23edbaa891f4bc446c3e4" exitCode=0
Sep 07 10:30:44.961295 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:44.960933    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8014-140224719722656" event={"ID":"3454575b-6516-423e-9bbe-ffa807c85fdd","Type":"ContainerDied","Data":"5364b45d1712154d8b450f5efd5e3ee11357013b7ce23edbaa891f4bc446c3e4"}
Sep 07 10:30:44.961295 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:44.960964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8014-140224719722656" event={"ID":"3454575b-6516-423e-9bbe-ffa807c85fdd","Type":"ContainerStarted","Data":"a5d8b299947647e1d6e4533822598fc6c683649b799cdfe897029b56b696a11c"}
Sep 07 10:30:46.107731 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:46.107698    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-140224719722656"
Sep 07 10:30:46.127960 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:46.127892    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-8014-140224719722656_3454575b-6516-423e-9bbe-ffa807c85fdd/test-curl-8014-140224719722656/0.log"
Sep 07 10:30:46.149988 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:46.149954    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8014-140224719722656"]
Sep 07 10:30:46.152988 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:46.152957    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8014-140224719722656"]
Sep 07 10:30:46.251565 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:46.251485    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgjfd\" (UniqueName: \"kubernetes.io/projected/3454575b-6516-423e-9bbe-ffa807c85fdd-kube-api-access-hgjfd\") pod \"3454575b-6516-423e-9bbe-ffa807c85fdd\" (UID: \"3454575b-6516-423e-9bbe-ffa807c85fdd\") "
Sep 07 10:30:46.253570 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:46.253541    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3454575b-6516-423e-9bbe-ffa807c85fdd-kube-api-access-hgjfd" (OuterVolumeSpecName: "kube-api-access-hgjfd") pod "3454575b-6516-423e-9bbe-ffa807c85fdd" (UID: "3454575b-6516-423e-9bbe-ffa807c85fdd"). InnerVolumeSpecName "kube-api-access-hgjfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:30:46.352241 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:46.352195    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgjfd\" (UniqueName: \"kubernetes.io/projected/3454575b-6516-423e-9bbe-ffa807c85fdd-kube-api-access-hgjfd\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:30:46.978049 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:46.978016    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-140224719722656"
Sep 07 10:30:46.978197 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:46.978056    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5d8b299947647e1d6e4533822598fc6c683649b799cdfe897029b56b696a11c"
Sep 07 10:30:46.988017 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:30:46.987985    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8014-140224719722656\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="3454575b-6516-423e-9bbe-ffa807c85fdd" pod="opendatahub/test-curl-8014-140224719722656"
Sep 07 10:30:47.963559 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:30:47.963516    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8014-140224719722656\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="3454575b-6516-423e-9bbe-ffa807c85fdd" pod="opendatahub/test-curl-8014-140224719722656"
Sep 07 10:30:47.965319 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:47.965296    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3454575b-6516-423e-9bbe-ffa807c85fdd" path="/var/lib/kubelet/pods/3454575b-6516-423e-9bbe-ffa807c85fdd/volumes"
Sep 07 10:30:51.157049 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.156938    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"]
Sep 07 10:30:51.157994 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.157969    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3454575b-6516-423e-9bbe-ffa807c85fdd" containerName="test-curl-8014-140224719722656"
Sep 07 10:30:51.157994 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.157993    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3454575b-6516-423e-9bbe-ffa807c85fdd" containerName="test-curl-8014-140224719722656"
Sep 07 10:30:51.158203 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.158173    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3454575b-6516-423e-9bbe-ffa807c85fdd" containerName="test-curl-8014-140224719722656"
Sep 07 10:30:51.162581 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.162553    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:30:51.165639 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.165611    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f957f836-serving-cert\""
Sep 07 10:30:51.168155 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.168129    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"]
Sep 07 10:30:51.299945 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.299904    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgpvp\" (UniqueName: \"kubernetes.io/projected/a64e851b-f325-4cd6-8aeb-590de751604f-kube-api-access-hgpvp\") pod \"maas-api-e2e-mt-f957f836-576dcd6b96-552tp\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:30:51.300134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.299955    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64e851b-f325-4cd6-8aeb-590de751604f-maas-api-tls\") pod \"maas-api-e2e-mt-f957f836-576dcd6b96-552tp\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:30:51.300134 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.300019    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64e851b-f325-4cd6-8aeb-590de751604f-metrics-certs\") pod \"maas-api-e2e-mt-f957f836-576dcd6b96-552tp\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:30:51.300233 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.300146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64e851b-f325-4cd6-8aeb-590de751604f-openshift-service-ca\") pod \"maas-api-e2e-mt-f957f836-576dcd6b96-552tp\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:30:51.401095 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.401053    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgpvp\" (UniqueName: \"kubernetes.io/projected/a64e851b-f325-4cd6-8aeb-590de751604f-kube-api-access-hgpvp\") pod \"maas-api-e2e-mt-f957f836-576dcd6b96-552tp\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:30:51.401095 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.401096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64e851b-f325-4cd6-8aeb-590de751604f-maas-api-tls\") pod \"maas-api-e2e-mt-f957f836-576dcd6b96-552tp\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:30:51.401357 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.401117    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64e851b-f325-4cd6-8aeb-590de751604f-metrics-certs\") pod \"maas-api-e2e-mt-f957f836-576dcd6b96-552tp\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:30:51.401357 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.401158    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64e851b-f325-4cd6-8aeb-590de751604f-openshift-service-ca\") pod \"maas-api-e2e-mt-f957f836-576dcd6b96-552tp\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:30:51.401357 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:30:51.401206    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-serving-cert: secret "maas-api-e2e-mt-f957f836-serving-cert" not found
Sep 07 10:30:51.401357 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:30:51.401285    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a64e851b-f325-4cd6-8aeb-590de751604f-maas-api-tls podName:a64e851b-f325-4cd6-8aeb-590de751604f nodeName:}" failed. No retries permitted until 2026-09-07 10:30:51.901265685 +0000 UTC m=+1154.398955586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a64e851b-f325-4cd6-8aeb-590de751604f-maas-api-tls") pod "maas-api-e2e-mt-f957f836-576dcd6b96-552tp" (UID: "a64e851b-f325-4cd6-8aeb-590de751604f") : secret "maas-api-e2e-mt-f957f836-serving-cert" not found
Sep 07 10:30:51.401878 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.401860    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64e851b-f325-4cd6-8aeb-590de751604f-openshift-service-ca\") pod \"maas-api-e2e-mt-f957f836-576dcd6b96-552tp\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:30:51.403596 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.403577    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64e851b-f325-4cd6-8aeb-590de751604f-metrics-certs\") pod \"maas-api-e2e-mt-f957f836-576dcd6b96-552tp\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:30:51.409897 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.409842    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgpvp\" (UniqueName: \"kubernetes.io/projected/a64e851b-f325-4cd6-8aeb-590de751604f-kube-api-access-hgpvp\") pod \"maas-api-e2e-mt-f957f836-576dcd6b96-552tp\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:30:51.805170 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.805132    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"]
Sep 07 10:30:51.808944 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.808916    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"
Sep 07 10:30:51.811688 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.811650    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f957f836\""
Sep 07 10:30:51.811865 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.811754    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f957f836-dockercfg-lbbl5\""
Sep 07 10:30:51.826058 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.826023    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"]
Sep 07 10:30:51.906185 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.906144    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqqqk\" (UniqueName: \"kubernetes.io/projected/ec8d49b4-3a81-436b-b96e-afc115b4f236-kube-api-access-zqqqk\") pod \"payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9\" (UID: \"ec8d49b4-3a81-436b-b96e-afc115b4f236\") " pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"
Sep 07 10:30:51.906185 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.906187    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8d49b4-3a81-436b-b96e-afc115b4f236-plugins-config-volume\") pod \"payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9\" (UID: \"ec8d49b4-3a81-436b-b96e-afc115b4f236\") " pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"
Sep 07 10:30:51.906427 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.906222    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64e851b-f325-4cd6-8aeb-590de751604f-maas-api-tls\") pod \"maas-api-e2e-mt-f957f836-576dcd6b96-552tp\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:30:51.908561 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:51.908538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64e851b-f325-4cd6-8aeb-590de751604f-maas-api-tls\") pod \"maas-api-e2e-mt-f957f836-576dcd6b96-552tp\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:30:52.007464 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:52.007434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqqqk\" (UniqueName: \"kubernetes.io/projected/ec8d49b4-3a81-436b-b96e-afc115b4f236-kube-api-access-zqqqk\") pod \"payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9\" (UID: \"ec8d49b4-3a81-436b-b96e-afc115b4f236\") " pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"
Sep 07 10:30:52.007649 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:52.007471    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8d49b4-3a81-436b-b96e-afc115b4f236-plugins-config-volume\") pod \"payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9\" (UID: \"ec8d49b4-3a81-436b-b96e-afc115b4f236\") " pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"
Sep 07 10:30:52.008253 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:52.008233    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8d49b4-3a81-436b-b96e-afc115b4f236-plugins-config-volume\") pod \"payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9\" (UID: \"ec8d49b4-3a81-436b-b96e-afc115b4f236\") " pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"
Sep 07 10:30:52.017476 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:52.017444    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqqqk\" (UniqueName: \"kubernetes.io/projected/ec8d49b4-3a81-436b-b96e-afc115b4f236-kube-api-access-zqqqk\") pod \"payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9\" (UID: \"ec8d49b4-3a81-436b-b96e-afc115b4f236\") " pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"
Sep 07 10:30:52.075840 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:52.075708    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:30:52.119920 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:52.119877    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"
Sep 07 10:30:52.354573 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:52.354546    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"]
Sep 07 10:30:52.355632 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:30:52.355607    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64e851b_f325_4cd6_8aeb_590de751604f.slice/crio-55c85998659eef6d01d8486cd0aa83d2eca6ca9d8b3d9c48303ada02c0a6047c WatchSource:0}: Error finding container 55c85998659eef6d01d8486cd0aa83d2eca6ca9d8b3d9c48303ada02c0a6047c: Status 404 returned error can't find the container with id 55c85998659eef6d01d8486cd0aa83d2eca6ca9d8b3d9c48303ada02c0a6047c
Sep 07 10:30:52.410050 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:52.410018    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"]
Sep 07 10:30:52.411757 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:30:52.411731    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8d49b4_3a81_436b_b96e_afc115b4f236.slice/crio-66e59114670647fa1fc0ccab784761d3f818c1a65aaa7926c499a8491a786f1d WatchSource:0}: Error finding container 66e59114670647fa1fc0ccab784761d3f818c1a65aaa7926c499a8491a786f1d: Status 404 returned error can't find the container with id 66e59114670647fa1fc0ccab784761d3f818c1a65aaa7926c499a8491a786f1d
Sep 07 10:30:53.014358 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:53.014258    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp" event={"ID":"a64e851b-f325-4cd6-8aeb-590de751604f","Type":"ContainerStarted","Data":"6c0266972cf2295c410cb2551b1d0ace8c98d3b2303f524ab483e8a5f7cf62d0"}
Sep 07 10:30:53.014358 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:53.014335    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:30:53.014590 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:53.014362    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp" event={"ID":"a64e851b-f325-4cd6-8aeb-590de751604f","Type":"ContainerStarted","Data":"55c85998659eef6d01d8486cd0aa83d2eca6ca9d8b3d9c48303ada02c0a6047c"}
Sep 07 10:30:53.016123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:53.016098    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9" event={"ID":"ec8d49b4-3a81-436b-b96e-afc115b4f236","Type":"ContainerStarted","Data":"642699f11f742dc77265bfe42e11be4f2b2f5490b41921c3735cdaadc97d7253"}
Sep 07 10:30:53.016260 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:53.016147    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9" event={"ID":"ec8d49b4-3a81-436b-b96e-afc115b4f236","Type":"ContainerStarted","Data":"66e59114670647fa1fc0ccab784761d3f818c1a65aaa7926c499a8491a786f1d"}
Sep 07 10:30:53.016260 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:53.016176    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"
Sep 07 10:30:53.034115 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:53.034043    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp" podStartSLOduration=1.631309369 podStartE2EDuration="2.034014508s" totalImagesPullingTime="402.705139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:30:53.031177972 +0000 UTC m=+1155.528867901" watchObservedRunningTime="2026-09-07 10:30:53.034014508 +0000 UTC m=+1155.531704418"
Sep 07 10:30:53.049007 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:53.048941    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9" podStartSLOduration=2.048918629 podStartE2EDuration="2.048918629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:30:53.04797875 +0000 UTC m=+1155.545668686" watchObservedRunningTime="2026-09-07 10:30:53.048918629 +0000 UTC m=+1155.546608539"
Sep 07 10:30:59.027860 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:30:59.027829    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:31:04.024442 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:31:04.024358    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"
Sep 07 10:32:46.254618 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.254534    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"]
Sep 07 10:32:46.256951 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.254809    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp" podUID="a64e851b-f325-4cd6-8aeb-590de751604f" containerName="maas-api" containerID="cri-o://6c0266972cf2295c410cb2551b1d0ace8c98d3b2303f524ab483e8a5f7cf62d0" gracePeriod=30
Sep 07 10:32:46.299911 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.299875    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"]
Sep 07 10:32:46.300147 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.300120    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9" podUID="ec8d49b4-3a81-436b-b96e-afc115b4f236" containerName="payload-processing" containerID="cri-o://642699f11f742dc77265bfe42e11be4f2b2f5490b41921c3735cdaadc97d7253" gracePeriod=30
Sep 07 10:32:46.584502 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.584475    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:32:46.587446 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.587424    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"
Sep 07 10:32:46.661625 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.661586    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64e851b-f325-4cd6-8aeb-590de751604f-openshift-service-ca\") pod \"a64e851b-f325-4cd6-8aeb-590de751604f\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") "
Sep 07 10:32:46.661844 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.661636    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqqqk\" (UniqueName: \"kubernetes.io/projected/ec8d49b4-3a81-436b-b96e-afc115b4f236-kube-api-access-zqqqk\") pod \"ec8d49b4-3a81-436b-b96e-afc115b4f236\" (UID: \"ec8d49b4-3a81-436b-b96e-afc115b4f236\") "
Sep 07 10:32:46.661844 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.661673    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64e851b-f325-4cd6-8aeb-590de751604f-metrics-certs\") pod \"a64e851b-f325-4cd6-8aeb-590de751604f\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") "
Sep 07 10:32:46.661844 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.661707    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgpvp\" (UniqueName: \"kubernetes.io/projected/a64e851b-f325-4cd6-8aeb-590de751604f-kube-api-access-hgpvp\") pod \"a64e851b-f325-4cd6-8aeb-590de751604f\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") "
Sep 07 10:32:46.661844 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.661838    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64e851b-f325-4cd6-8aeb-590de751604f-maas-api-tls\") pod \"a64e851b-f325-4cd6-8aeb-590de751604f\" (UID: \"a64e851b-f325-4cd6-8aeb-590de751604f\") "
Sep 07 10:32:46.662087 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.661919    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8d49b4-3a81-436b-b96e-afc115b4f236-plugins-config-volume\") pod \"ec8d49b4-3a81-436b-b96e-afc115b4f236\" (UID: \"ec8d49b4-3a81-436b-b96e-afc115b4f236\") "
Sep 07 10:32:46.662149 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.662079    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a64e851b-f325-4cd6-8aeb-590de751604f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a64e851b-f325-4cd6-8aeb-590de751604f" (UID: "a64e851b-f325-4cd6-8aeb-590de751604f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:32:46.662289 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.662243    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64e851b-f325-4cd6-8aeb-590de751604f-openshift-service-ca\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:32:46.662491 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.662469    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8d49b4-3a81-436b-b96e-afc115b4f236-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec8d49b4-3a81-436b-b96e-afc115b4f236" (UID: "ec8d49b4-3a81-436b-b96e-afc115b4f236"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:32:46.664031 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.664003    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64e851b-f325-4cd6-8aeb-590de751604f-kube-api-access-hgpvp" (OuterVolumeSpecName: "kube-api-access-hgpvp") pod "a64e851b-f325-4cd6-8aeb-590de751604f" (UID: "a64e851b-f325-4cd6-8aeb-590de751604f"). InnerVolumeSpecName "kube-api-access-hgpvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:32:46.664031 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.664027    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8d49b4-3a81-436b-b96e-afc115b4f236-kube-api-access-zqqqk" (OuterVolumeSpecName: "kube-api-access-zqqqk") pod "ec8d49b4-3a81-436b-b96e-afc115b4f236" (UID: "ec8d49b4-3a81-436b-b96e-afc115b4f236"). InnerVolumeSpecName "kube-api-access-zqqqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:32:46.664167 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.664003    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64e851b-f325-4cd6-8aeb-590de751604f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a64e851b-f325-4cd6-8aeb-590de751604f" (UID: "a64e851b-f325-4cd6-8aeb-590de751604f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:32:46.664167 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.664099    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64e851b-f325-4cd6-8aeb-590de751604f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a64e851b-f325-4cd6-8aeb-590de751604f" (UID: "a64e851b-f325-4cd6-8aeb-590de751604f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:32:46.674418 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.674388    2562 generic.go:356] "Generic (PLEG): container finished" podID="ec8d49b4-3a81-436b-b96e-afc115b4f236" containerID="642699f11f742dc77265bfe42e11be4f2b2f5490b41921c3735cdaadc97d7253" exitCode=0
Sep 07 10:32:46.674543 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.674431    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"
Sep 07 10:32:46.674543 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.674475    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9" event={"ID":"ec8d49b4-3a81-436b-b96e-afc115b4f236","Type":"ContainerDied","Data":"642699f11f742dc77265bfe42e11be4f2b2f5490b41921c3735cdaadc97d7253"}
Sep 07 10:32:46.674543 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.674533    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9" event={"ID":"ec8d49b4-3a81-436b-b96e-afc115b4f236","Type":"ContainerDied","Data":"66e59114670647fa1fc0ccab784761d3f818c1a65aaa7926c499a8491a786f1d"}
Sep 07 10:32:46.674706 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.674564    2562 scope.go:117] "RemoveContainer" containerID="642699f11f742dc77265bfe42e11be4f2b2f5490b41921c3735cdaadc97d7253"
Sep 07 10:32:46.675958 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.675928    2562 generic.go:356] "Generic (PLEG): container finished" podID="a64e851b-f325-4cd6-8aeb-590de751604f" containerID="6c0266972cf2295c410cb2551b1d0ace8c98d3b2303f524ab483e8a5f7cf62d0" exitCode=0
Sep 07 10:32:46.676074 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.675959    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp" event={"ID":"a64e851b-f325-4cd6-8aeb-590de751604f","Type":"ContainerDied","Data":"6c0266972cf2295c410cb2551b1d0ace8c98d3b2303f524ab483e8a5f7cf62d0"}
Sep 07 10:32:46.676074 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.676020    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp" event={"ID":"a64e851b-f325-4cd6-8aeb-590de751604f","Type":"ContainerDied","Data":"55c85998659eef6d01d8486cd0aa83d2eca6ca9d8b3d9c48303ada02c0a6047c"}
Sep 07 10:32:46.676074 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.675966    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"
Sep 07 10:32:46.684852 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.684834    2562 scope.go:117] "RemoveContainer" containerID="642699f11f742dc77265bfe42e11be4f2b2f5490b41921c3735cdaadc97d7253"
Sep 07 10:32:46.685133 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:32:46.685106    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642699f11f742dc77265bfe42e11be4f2b2f5490b41921c3735cdaadc97d7253\": container with ID starting with 642699f11f742dc77265bfe42e11be4f2b2f5490b41921c3735cdaadc97d7253 not found: ID does not exist" containerID="642699f11f742dc77265bfe42e11be4f2b2f5490b41921c3735cdaadc97d7253"
Sep 07 10:32:46.685203 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.685144    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642699f11f742dc77265bfe42e11be4f2b2f5490b41921c3735cdaadc97d7253"} err="failed to get container status \"642699f11f742dc77265bfe42e11be4f2b2f5490b41921c3735cdaadc97d7253\": rpc error: code = NotFound desc = could not find container \"642699f11f742dc77265bfe42e11be4f2b2f5490b41921c3735cdaadc97d7253\": container with ID starting with 642699f11f742dc77265bfe42e11be4f2b2f5490b41921c3735cdaadc97d7253 not found: ID does not exist"
Sep 07 10:32:46.685203 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.685165    2562 scope.go:117] "RemoveContainer" containerID="6c0266972cf2295c410cb2551b1d0ace8c98d3b2303f524ab483e8a5f7cf62d0"
Sep 07 10:32:46.693629 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.693610    2562 scope.go:117] "RemoveContainer" containerID="6c0266972cf2295c410cb2551b1d0ace8c98d3b2303f524ab483e8a5f7cf62d0"
Sep 07 10:32:46.693935 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:32:46.693915    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0266972cf2295c410cb2551b1d0ace8c98d3b2303f524ab483e8a5f7cf62d0\": container with ID starting with 6c0266972cf2295c410cb2551b1d0ace8c98d3b2303f524ab483e8a5f7cf62d0 not found: ID does not exist" containerID="6c0266972cf2295c410cb2551b1d0ace8c98d3b2303f524ab483e8a5f7cf62d0"
Sep 07 10:32:46.693991 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.693942    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0266972cf2295c410cb2551b1d0ace8c98d3b2303f524ab483e8a5f7cf62d0"} err="failed to get container status \"6c0266972cf2295c410cb2551b1d0ace8c98d3b2303f524ab483e8a5f7cf62d0\": rpc error: code = NotFound desc = could not find container \"6c0266972cf2295c410cb2551b1d0ace8c98d3b2303f524ab483e8a5f7cf62d0\": container with ID starting with 6c0266972cf2295c410cb2551b1d0ace8c98d3b2303f524ab483e8a5f7cf62d0 not found: ID does not exist"
Sep 07 10:32:46.700682 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.700661    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"]
Sep 07 10:32:46.704392 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.704370    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f957f836-576dcd6b96-552tp"]
Sep 07 10:32:46.713663 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.713643    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"]
Sep 07 10:32:46.717717 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.717698    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f957f836-db994fdf7-hwnc9"]
Sep 07 10:32:46.763508 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.763472    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqqqk\" (UniqueName: \"kubernetes.io/projected/ec8d49b4-3a81-436b-b96e-afc115b4f236-kube-api-access-zqqqk\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:32:46.763508 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.763504    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64e851b-f325-4cd6-8aeb-590de751604f-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:32:46.763725 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.763517    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgpvp\" (UniqueName: \"kubernetes.io/projected/a64e851b-f325-4cd6-8aeb-590de751604f-kube-api-access-hgpvp\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:32:46.763725 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.763527    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64e851b-f325-4cd6-8aeb-590de751604f-maas-api-tls\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:32:46.763725 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:46.763535    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8d49b4-3a81-436b-b96e-afc115b4f236-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:32:47.966327 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:47.966290    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a64e851b-f325-4cd6-8aeb-590de751604f" path="/var/lib/kubelet/pods/a64e851b-f325-4cd6-8aeb-590de751604f/volumes"
Sep 07 10:32:47.966697 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:32:47.966685    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec8d49b4-3a81-436b-b96e-afc115b4f236" path="/var/lib/kubelet/pods/ec8d49b4-3a81-436b-b96e-afc115b4f236/volumes"
Sep 07 10:33:46.678367 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:46.678324    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-8017-65f309"]
Sep 07 10:33:46.678997 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:46.678976    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a64e851b-f325-4cd6-8aeb-590de751604f" containerName="maas-api"
Sep 07 10:33:46.679085 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:46.679000    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a64e851b-f325-4cd6-8aeb-590de751604f" containerName="maas-api"
Sep 07 10:33:46.679085 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:46.679024    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec8d49b4-3a81-436b-b96e-afc115b4f236" containerName="payload-processing"
Sep 07 10:33:46.679085 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:46.679033    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec8d49b4-3a81-436b-b96e-afc115b4f236" containerName="payload-processing"
Sep 07 10:33:46.679235 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:46.679135    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec8d49b4-3a81-436b-b96e-afc115b4f236" containerName="payload-processing"
Sep 07 10:33:46.679235 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:46.679147    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a64e851b-f325-4cd6-8aeb-590de751604f" containerName="maas-api"
Sep 07 10:33:46.683746 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:46.683725    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8017-65f309"
Sep 07 10:33:46.687220 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:46.687169    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qpjjb\""
Sep 07 10:33:46.688677 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:46.688654    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-8017-65f309"]
Sep 07 10:33:46.697308 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:46.697285    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5cp\" (UniqueName: \"kubernetes.io/projected/1f3d4619-78f3-41fe-b68b-30337a2d0539-kube-api-access-bx5cp\") pod \"mt-curl-8017-65f309\" (UID: \"1f3d4619-78f3-41fe-b68b-30337a2d0539\") " pod="opendatahub/mt-curl-8017-65f309"
Sep 07 10:33:46.798413 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:46.798373    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5cp\" (UniqueName: \"kubernetes.io/projected/1f3d4619-78f3-41fe-b68b-30337a2d0539-kube-api-access-bx5cp\") pod \"mt-curl-8017-65f309\" (UID: \"1f3d4619-78f3-41fe-b68b-30337a2d0539\") " pod="opendatahub/mt-curl-8017-65f309"
Sep 07 10:33:46.807344 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:46.807310    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5cp\" (UniqueName: \"kubernetes.io/projected/1f3d4619-78f3-41fe-b68b-30337a2d0539-kube-api-access-bx5cp\") pod \"mt-curl-8017-65f309\" (UID: \"1f3d4619-78f3-41fe-b68b-30337a2d0539\") " pod="opendatahub/mt-curl-8017-65f309"
Sep 07 10:33:46.994839 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:46.994791    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8017-65f309"
Sep 07 10:33:47.285600 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:47.285569    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-8017-65f309"]
Sep 07 10:33:47.286695 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:33:47.286660    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f3d4619_78f3_41fe_b68b_30337a2d0539.slice/crio-5ccd8237b5b124ca723a3849b95ada8f2ee208c14bfef02ddb4d4680a5829d25 WatchSource:0}: Error finding container 5ccd8237b5b124ca723a3849b95ada8f2ee208c14bfef02ddb4d4680a5829d25: Status 404 returned error can't find the container with id 5ccd8237b5b124ca723a3849b95ada8f2ee208c14bfef02ddb4d4680a5829d25
Sep 07 10:33:47.288074 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:47.288051    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 10:33:48.033756 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:48.033731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-8017-65f309" event={"ID":"1f3d4619-78f3-41fe-b68b-30337a2d0539","Type":"ContainerStarted","Data":"5ccd8237b5b124ca723a3849b95ada8f2ee208c14bfef02ddb4d4680a5829d25"}
Sep 07 10:33:49.042437 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:49.042401    2562 generic.go:356] "Generic (PLEG): container finished" podID="1f3d4619-78f3-41fe-b68b-30337a2d0539" containerID="fb99ca0deb5f9ecee3e881003af3086da5285133bea4abd46ee3722a709803c4" exitCode=0
Sep 07 10:33:49.042861 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:49.042485    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-8017-65f309" event={"ID":"1f3d4619-78f3-41fe-b68b-30337a2d0539","Type":"ContainerDied","Data":"fb99ca0deb5f9ecee3e881003af3086da5285133bea4abd46ee3722a709803c4"}
Sep 07 10:33:50.186745 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:50.186720    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8017-65f309"
Sep 07 10:33:50.215751 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:50.215712    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-8017-65f309_1f3d4619-78f3-41fe-b68b-30337a2d0539/mt-curl-8017-65f309/0.log"
Sep 07 10:33:50.229446 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:50.229410    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx5cp\" (UniqueName: \"kubernetes.io/projected/1f3d4619-78f3-41fe-b68b-30337a2d0539-kube-api-access-bx5cp\") pod \"1f3d4619-78f3-41fe-b68b-30337a2d0539\" (UID: \"1f3d4619-78f3-41fe-b68b-30337a2d0539\") "
Sep 07 10:33:50.231430 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:50.231395    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3d4619-78f3-41fe-b68b-30337a2d0539-kube-api-access-bx5cp" (OuterVolumeSpecName: "kube-api-access-bx5cp") pod "1f3d4619-78f3-41fe-b68b-30337a2d0539" (UID: "1f3d4619-78f3-41fe-b68b-30337a2d0539"). InnerVolumeSpecName "kube-api-access-bx5cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:33:50.237614 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:50.237587    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-8017-65f309"]
Sep 07 10:33:50.240970 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:50.240941    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-8017-65f309"]
Sep 07 10:33:50.330640 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:50.330561    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx5cp\" (UniqueName: \"kubernetes.io/projected/1f3d4619-78f3-41fe-b68b-30337a2d0539-kube-api-access-bx5cp\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:33:51.057263 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:51.057233    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8017-65f309"
Sep 07 10:33:51.057450 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:51.057272    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ccd8237b5b124ca723a3849b95ada8f2ee208c14bfef02ddb4d4680a5829d25"
Sep 07 10:33:51.067359 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:33:51.067328    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-8017-65f309\" is forbidden: User \"system:node:ip-10-0-141-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-62.ec2.internal' and this object" podUID="1f3d4619-78f3-41fe-b68b-30337a2d0539" pod="opendatahub/mt-curl-8017-65f309"
Sep 07 10:33:51.964927 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:51.964889    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f3d4619-78f3-41fe-b68b-30337a2d0539" path="/var/lib/kubelet/pods/1f3d4619-78f3-41fe-b68b-30337a2d0539/volumes"
Sep 07 10:33:56.397729 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.397693    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"]
Sep 07 10:33:56.398290 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.398140    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f3d4619-78f3-41fe-b68b-30337a2d0539" containerName="mt-curl-8017-65f309"
Sep 07 10:33:56.398290 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.398155    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f3d4619-78f3-41fe-b68b-30337a2d0539" containerName="mt-curl-8017-65f309"
Sep 07 10:33:56.398290 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.398218    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f3d4619-78f3-41fe-b68b-30337a2d0539" containerName="mt-curl-8017-65f309"
Sep 07 10:33:56.401736 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.401711    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.405632 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.405607    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a577c\"/\"kube-root-ca.crt\""
Sep 07 10:33:56.405632 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.405620    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a577c\"/\"openshift-service-ca.crt\""
Sep 07 10:33:56.405855 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.405620    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a577c\"/\"default-dockercfg-c5g9z\""
Sep 07 10:33:56.405855 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.405620    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a577c\"/\"sub-test-model-0a577c-kserve-self-signed-certs\""
Sep 07 10:33:56.410619 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.410597    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"]
Sep 07 10:33:56.492104 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.492069    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-dshm\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.492104 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.492108    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-model-cache\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.492338 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.492137    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-tls-certs\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.492338 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.492253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhgjk\" (UniqueName: \"kubernetes.io/projected/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-kube-api-access-rhgjk\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.492338 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.492284    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-home\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.492338 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.492301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-tmp-dir\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.592950 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.592909    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-dshm\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.592950 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.592950    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-model-cache\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.593213 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.592982    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-tls-certs\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.593213 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.593029    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhgjk\" (UniqueName: \"kubernetes.io/projected/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-kube-api-access-rhgjk\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.593213 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.593048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-home\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.593213 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.593062    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-tmp-dir\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.593487 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.593453    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-model-cache\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.593487 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.593483    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-tmp-dir\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.593651 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.593523    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-home\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.595225 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.595205    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-dshm\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.595638 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.595619    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-tls-certs\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.601686 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.601665    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhgjk\" (UniqueName: \"kubernetes.io/projected/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-kube-api-access-rhgjk\") pod \"sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.712076 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.712034    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:33:56.987976 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:56.987911    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"]
Sep 07 10:33:56.991283 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:33:56.991252    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ab1cfa7_1daa_41fa_8b7d_fc4dd6e029bb.slice/crio-d6d5d71f882d259b1026d9c156b372cc044e934377ba16034eb92199469f064a WatchSource:0}: Error finding container d6d5d71f882d259b1026d9c156b372cc044e934377ba16034eb92199469f064a: Status 404 returned error can't find the container with id d6d5d71f882d259b1026d9c156b372cc044e934377ba16034eb92199469f064a
Sep 07 10:33:57.093446 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:57.093411    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk" event={"ID":"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb","Type":"ContainerStarted","Data":"d43fb42103604565c75b172bbc68360858cf59bc4ddf325c86dd86938ee5c7af"}
Sep 07 10:33:57.093628 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:57.093465    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk" event={"ID":"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb","Type":"ContainerStarted","Data":"d6d5d71f882d259b1026d9c156b372cc044e934377ba16034eb92199469f064a"}
Sep 07 10:33:57.110526 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:33:57.110277    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk" podStartSLOduration=1.110256081 podStartE2EDuration="1.110256081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:33:57.10922896 +0000 UTC m=+1339.606918894" watchObservedRunningTime="2026-09-07 10:33:57.110256081 +0000 UTC m=+1339.607945989"
Sep 07 10:34:06.713182 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:06.713139    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:34:06.713617 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:06.713235    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:34:06.725803 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:06.725748    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:34:07.168927 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:07.168832    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:34:36.235266 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:36.235226    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"]
Sep 07 10:34:36.240329 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:36.240305    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"
Sep 07 10:34:36.243219 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:36.243183    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fbea94-dockercfg-ld5zd\""
Sep 07 10:34:36.243219 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:36.243199    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fbea94\""
Sep 07 10:34:36.252035 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:36.252009    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"]
Sep 07 10:34:36.266189 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:36.266148    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7216486c-9719-428c-89b5-a30e80feccd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx\" (UID: \"7216486c-9719-428c-89b5-a30e80feccd6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"
Sep 07 10:34:36.266375 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:36.266200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j42jp\" (UniqueName: \"kubernetes.io/projected/7216486c-9719-428c-89b5-a30e80feccd6-kube-api-access-j42jp\") pod \"payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx\" (UID: \"7216486c-9719-428c-89b5-a30e80feccd6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"
Sep 07 10:34:36.367461 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:36.367421    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7216486c-9719-428c-89b5-a30e80feccd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx\" (UID: \"7216486c-9719-428c-89b5-a30e80feccd6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"
Sep 07 10:34:36.367461 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:36.367467    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j42jp\" (UniqueName: \"kubernetes.io/projected/7216486c-9719-428c-89b5-a30e80feccd6-kube-api-access-j42jp\") pod \"payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx\" (UID: \"7216486c-9719-428c-89b5-a30e80feccd6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"
Sep 07 10:34:36.368224 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:36.368203    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7216486c-9719-428c-89b5-a30e80feccd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx\" (UID: \"7216486c-9719-428c-89b5-a30e80feccd6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"
Sep 07 10:34:36.375858 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:36.375826    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j42jp\" (UniqueName: \"kubernetes.io/projected/7216486c-9719-428c-89b5-a30e80feccd6-kube-api-access-j42jp\") pod \"payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx\" (UID: \"7216486c-9719-428c-89b5-a30e80feccd6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"
Sep 07 10:34:36.552627 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:36.552549    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"
Sep 07 10:34:36.827017 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:36.826987    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"]
Sep 07 10:34:36.828990 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:34:36.828963    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7216486c_9719_428c_89b5_a30e80feccd6.slice/crio-525a2af9acf8c2c3f4bf6ca4e642b65a94366cddb0eefd4f0551f5ed10aa0013 WatchSource:0}: Error finding container 525a2af9acf8c2c3f4bf6ca4e642b65a94366cddb0eefd4f0551f5ed10aa0013: Status 404 returned error can't find the container with id 525a2af9acf8c2c3f4bf6ca4e642b65a94366cddb0eefd4f0551f5ed10aa0013
Sep 07 10:34:37.338388 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:37.338348    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx" event={"ID":"7216486c-9719-428c-89b5-a30e80feccd6","Type":"ContainerStarted","Data":"d9228ff3e572989c78e7e6ae2b4e3078ac8876a010c56a5441a2e89355c7027d"}
Sep 07 10:34:37.338836 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:37.338412    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx" event={"ID":"7216486c-9719-428c-89b5-a30e80feccd6","Type":"ContainerStarted","Data":"525a2af9acf8c2c3f4bf6ca4e642b65a94366cddb0eefd4f0551f5ed10aa0013"}
Sep 07 10:34:37.338836 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:37.338519    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"
Sep 07 10:34:37.358453 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:37.358389    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx" podStartSLOduration=1.358367168 podStartE2EDuration="1.358367168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:34:37.355921346 +0000 UTC m=+1379.853611273" watchObservedRunningTime="2026-09-07 10:34:37.358367168 +0000 UTC m=+1379.856057081"
Sep 07 10:34:48.347181 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:34:48.347148    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"
Sep 07 10:35:04.706726 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:04.706685    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"]
Sep 07 10:35:04.711277 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:04.711248    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:35:04.713895 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:04.713867    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-695a13-serving-cert\""
Sep 07 10:35:04.724186 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:04.724125    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"]
Sep 07 10:35:04.840058 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:04.840016    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ba4ee9f-077a-40f1-938b-2036a404008a-metrics-certs\") pod \"maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7\" (UID: \"2ba4ee9f-077a-40f1-938b-2036a404008a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:35:04.840255 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:04.840068    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r65xw\" (UniqueName: \"kubernetes.io/projected/2ba4ee9f-077a-40f1-938b-2036a404008a-kube-api-access-r65xw\") pod \"maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7\" (UID: \"2ba4ee9f-077a-40f1-938b-2036a404008a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:35:04.840255 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:04.840128    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba4ee9f-077a-40f1-938b-2036a404008a-openshift-service-ca\") pod \"maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7\" (UID: \"2ba4ee9f-077a-40f1-938b-2036a404008a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:35:04.840255 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:04.840163    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ba4ee9f-077a-40f1-938b-2036a404008a-maas-api-tls\") pod \"maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7\" (UID: \"2ba4ee9f-077a-40f1-938b-2036a404008a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:35:04.941593 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:04.941555    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ba4ee9f-077a-40f1-938b-2036a404008a-metrics-certs\") pod \"maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7\" (UID: \"2ba4ee9f-077a-40f1-938b-2036a404008a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:35:04.941593 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:04.941594    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r65xw\" (UniqueName: \"kubernetes.io/projected/2ba4ee9f-077a-40f1-938b-2036a404008a-kube-api-access-r65xw\") pod \"maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7\" (UID: \"2ba4ee9f-077a-40f1-938b-2036a404008a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:35:04.941921 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:04.941627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba4ee9f-077a-40f1-938b-2036a404008a-openshift-service-ca\") pod \"maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7\" (UID: \"2ba4ee9f-077a-40f1-938b-2036a404008a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:35:04.941921 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:04.941667    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ba4ee9f-077a-40f1-938b-2036a404008a-maas-api-tls\") pod \"maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7\" (UID: \"2ba4ee9f-077a-40f1-938b-2036a404008a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:35:04.942444 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:04.942416    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba4ee9f-077a-40f1-938b-2036a404008a-openshift-service-ca\") pod \"maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7\" (UID: \"2ba4ee9f-077a-40f1-938b-2036a404008a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:35:04.944264 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:04.944234    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ba4ee9f-077a-40f1-938b-2036a404008a-metrics-certs\") pod \"maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7\" (UID: \"2ba4ee9f-077a-40f1-938b-2036a404008a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:35:04.944378 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:04.944277    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ba4ee9f-077a-40f1-938b-2036a404008a-maas-api-tls\") pod \"maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7\" (UID: \"2ba4ee9f-077a-40f1-938b-2036a404008a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:35:04.957427 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:04.957350    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r65xw\" (UniqueName: \"kubernetes.io/projected/2ba4ee9f-077a-40f1-938b-2036a404008a-kube-api-access-r65xw\") pod \"maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7\" (UID: \"2ba4ee9f-077a-40f1-938b-2036a404008a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:35:05.025455 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.025414    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:35:05.304715 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.304685    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"]
Sep 07 10:35:05.306228 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:35:05.306202    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba4ee9f_077a_40f1_938b_2036a404008a.slice/crio-46097bc6b96ea19345df516940e584d662285c748b280d3a04e3a4b70664afe6 WatchSource:0}: Error finding container 46097bc6b96ea19345df516940e584d662285c748b280d3a04e3a4b70664afe6: Status 404 returned error can't find the container with id 46097bc6b96ea19345df516940e584d662285c748b280d3a04e3a4b70664afe6
Sep 07 10:35:05.509404 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.509368    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7" event={"ID":"2ba4ee9f-077a-40f1-938b-2036a404008a","Type":"ContainerStarted","Data":"46097bc6b96ea19345df516940e584d662285c748b280d3a04e3a4b70664afe6"}
Sep 07 10:35:05.532104 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.532067    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"]
Sep 07 10:35:05.537101 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.537073    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"
Sep 07 10:35:05.540857 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.540826    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-695a13\""
Sep 07 10:35:05.557747 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.554532    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"]
Sep 07 10:35:05.573228 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.573187    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"]
Sep 07 10:35:05.577142 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.577116    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"
Sep 07 10:35:05.581495 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.581474    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-695a13-dockercfg-gcn77\""
Sep 07 10:35:05.591752 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.591721    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"]
Sep 07 10:35:05.648518 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.648470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx8dr\" (UniqueName: \"kubernetes.io/projected/122582ef-848a-4551-8ac4-121103416746-kube-api-access-cx8dr\") pod \"payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq\" (UID: \"122582ef-848a-4551-8ac4-121103416746\") " pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"
Sep 07 10:35:05.648703 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.648595    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122582ef-848a-4551-8ac4-121103416746-plugins-config-volume\") pod \"payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq\" (UID: \"122582ef-848a-4551-8ac4-121103416746\") " pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"
Sep 07 10:35:05.749497 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.749454    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzqvs\" (UniqueName: \"kubernetes.io/projected/e6a1b364-2910-4b02-801a-940de8b5a286-kube-api-access-nzqvs\") pod \"payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4\" (UID: \"e6a1b364-2910-4b02-801a-940de8b5a286\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"
Sep 07 10:35:05.749943 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.749565    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a1b364-2910-4b02-801a-940de8b5a286-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4\" (UID: \"e6a1b364-2910-4b02-801a-940de8b5a286\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"
Sep 07 10:35:05.749943 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.749638    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122582ef-848a-4551-8ac4-121103416746-plugins-config-volume\") pod \"payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq\" (UID: \"122582ef-848a-4551-8ac4-121103416746\") " pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"
Sep 07 10:35:05.749943 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.749733    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx8dr\" (UniqueName: \"kubernetes.io/projected/122582ef-848a-4551-8ac4-121103416746-kube-api-access-cx8dr\") pod \"payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq\" (UID: \"122582ef-848a-4551-8ac4-121103416746\") " pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"
Sep 07 10:35:05.750477 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.750447    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122582ef-848a-4551-8ac4-121103416746-plugins-config-volume\") pod \"payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq\" (UID: \"122582ef-848a-4551-8ac4-121103416746\") " pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"
Sep 07 10:35:05.757541 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.757513    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx8dr\" (UniqueName: \"kubernetes.io/projected/122582ef-848a-4551-8ac4-121103416746-kube-api-access-cx8dr\") pod \"payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq\" (UID: \"122582ef-848a-4551-8ac4-121103416746\") " pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"
Sep 07 10:35:05.849522 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.849487    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"
Sep 07 10:35:05.850850 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.850823    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzqvs\" (UniqueName: \"kubernetes.io/projected/e6a1b364-2910-4b02-801a-940de8b5a286-kube-api-access-nzqvs\") pod \"payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4\" (UID: \"e6a1b364-2910-4b02-801a-940de8b5a286\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"
Sep 07 10:35:05.850970 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.850899    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a1b364-2910-4b02-801a-940de8b5a286-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4\" (UID: \"e6a1b364-2910-4b02-801a-940de8b5a286\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"
Sep 07 10:35:05.851534 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.851514    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a1b364-2910-4b02-801a-940de8b5a286-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4\" (UID: \"e6a1b364-2910-4b02-801a-940de8b5a286\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"
Sep 07 10:35:05.859032 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.858996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzqvs\" (UniqueName: \"kubernetes.io/projected/e6a1b364-2910-4b02-801a-940de8b5a286-kube-api-access-nzqvs\") pod \"payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4\" (UID: \"e6a1b364-2910-4b02-801a-940de8b5a286\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"
Sep 07 10:35:05.887826 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:05.887792    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"
Sep 07 10:35:06.131225 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:06.131191    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"]
Sep 07 10:35:06.132737 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:35:06.132680    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122582ef_848a_4551_8ac4_121103416746.slice/crio-3b23f616e6fc96396cfd6a82f40204380317bccb123b1c63f77e021429a33c10 WatchSource:0}: Error finding container 3b23f616e6fc96396cfd6a82f40204380317bccb123b1c63f77e021429a33c10: Status 404 returned error can't find the container with id 3b23f616e6fc96396cfd6a82f40204380317bccb123b1c63f77e021429a33c10
Sep 07 10:35:06.185232 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:06.185207    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"]
Sep 07 10:35:06.186902 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:35:06.186876    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a1b364_2910_4b02_801a_940de8b5a286.slice/crio-9ef8f310d3c27f5ccb003c8d5a75fddbb0a9f4b0c638ab8f06cd3394489acb88 WatchSource:0}: Error finding container 9ef8f310d3c27f5ccb003c8d5a75fddbb0a9f4b0c638ab8f06cd3394489acb88: Status 404 returned error can't find the container with id 9ef8f310d3c27f5ccb003c8d5a75fddbb0a9f4b0c638ab8f06cd3394489acb88
Sep 07 10:35:06.518007 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:06.517966    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4" event={"ID":"e6a1b364-2910-4b02-801a-940de8b5a286","Type":"ContainerStarted","Data":"b5ceeab8ba339d54152a7de8c500eba0b31eda11dd46944f8cd62685f2acbe4b"}
Sep 07 10:35:06.518198 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:06.518034    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4" event={"ID":"e6a1b364-2910-4b02-801a-940de8b5a286","Type":"ContainerStarted","Data":"9ef8f310d3c27f5ccb003c8d5a75fddbb0a9f4b0c638ab8f06cd3394489acb88"}
Sep 07 10:35:06.518198 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:06.518155    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"
Sep 07 10:35:06.519899 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:06.519866    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq" event={"ID":"122582ef-848a-4551-8ac4-121103416746","Type":"ContainerStarted","Data":"e4902cbb332c70b151cc838171312fed29d8e18aec36bddd79758206d081327a"}
Sep 07 10:35:06.520046 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:06.519930    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"
Sep 07 10:35:06.520046 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:06.519956    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq" event={"ID":"122582ef-848a-4551-8ac4-121103416746","Type":"ContainerStarted","Data":"3b23f616e6fc96396cfd6a82f40204380317bccb123b1c63f77e021429a33c10"}
Sep 07 10:35:06.521670 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:06.521645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7" event={"ID":"2ba4ee9f-077a-40f1-938b-2036a404008a","Type":"ContainerStarted","Data":"0ab08604d6ce12c7a7e0f51d3d41dbd6ac4da2a9d762c25a98ea169116286e66"}
Sep 07 10:35:06.521812 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:06.521709    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:35:06.535479 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:06.535423    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4" podStartSLOduration=1.5354027609999998 podStartE2EDuration="1.535402761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:35:06.534259955 +0000 UTC m=+1409.031949871" watchObservedRunningTime="2026-09-07 10:35:06.535402761 +0000 UTC m=+1409.033092662"
Sep 07 10:35:06.557118 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:06.557062    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7" podStartSLOduration=2.084868186 podStartE2EDuration="2.557044361s" totalImagesPullingTime="472.176175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:35:06.554786889 +0000 UTC m=+1409.052476818" watchObservedRunningTime="2026-09-07 10:35:06.557044361 +0000 UTC m=+1409.054734327"
Sep 07 10:35:06.572912 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:06.572847    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq" podStartSLOduration=1.572825242 podStartE2EDuration="1.572825242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:35:06.572325009 +0000 UTC m=+1409.070014975" watchObservedRunningTime="2026-09-07 10:35:06.572825242 +0000 UTC m=+1409.070515151"
Sep 07 10:35:12.533500 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:12.533458    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:35:17.530192 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:17.530164    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"
Sep 07 10:35:17.530566 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:17.530236    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"
Sep 07 10:35:46.919123 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:46.919037    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"]
Sep 07 10:35:46.919706 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:46.919266    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx" podUID="7216486c-9719-428c-89b5-a30e80feccd6" containerName="payload-pre-processing" containerID="cri-o://d9228ff3e572989c78e7e6ae2b4e3078ac8876a010c56a5441a2e89355c7027d" gracePeriod=30
Sep 07 10:35:47.185489 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.185406    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"
Sep 07 10:35:47.256989 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.256957    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j42jp\" (UniqueName: \"kubernetes.io/projected/7216486c-9719-428c-89b5-a30e80feccd6-kube-api-access-j42jp\") pod \"7216486c-9719-428c-89b5-a30e80feccd6\" (UID: \"7216486c-9719-428c-89b5-a30e80feccd6\") "
Sep 07 10:35:47.257161 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.257000    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7216486c-9719-428c-89b5-a30e80feccd6-plugins-config-volume\") pod \"7216486c-9719-428c-89b5-a30e80feccd6\" (UID: \"7216486c-9719-428c-89b5-a30e80feccd6\") "
Sep 07 10:35:47.257417 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.257398    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7216486c-9719-428c-89b5-a30e80feccd6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7216486c-9719-428c-89b5-a30e80feccd6" (UID: "7216486c-9719-428c-89b5-a30e80feccd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:35:47.259082 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.259059    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7216486c-9719-428c-89b5-a30e80feccd6-kube-api-access-j42jp" (OuterVolumeSpecName: "kube-api-access-j42jp") pod "7216486c-9719-428c-89b5-a30e80feccd6" (UID: "7216486c-9719-428c-89b5-a30e80feccd6"). InnerVolumeSpecName "kube-api-access-j42jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:35:47.358308 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.358270    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j42jp\" (UniqueName: \"kubernetes.io/projected/7216486c-9719-428c-89b5-a30e80feccd6-kube-api-access-j42jp\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:35:47.358308 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.358299    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7216486c-9719-428c-89b5-a30e80feccd6-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:35:47.773133 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.773087    2562 generic.go:356] "Generic (PLEG): container finished" podID="7216486c-9719-428c-89b5-a30e80feccd6" containerID="d9228ff3e572989c78e7e6ae2b4e3078ac8876a010c56a5441a2e89355c7027d" exitCode=0
Sep 07 10:35:47.773351 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.773152    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"
Sep 07 10:35:47.773351 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.773178    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx" event={"ID":"7216486c-9719-428c-89b5-a30e80feccd6","Type":"ContainerDied","Data":"d9228ff3e572989c78e7e6ae2b4e3078ac8876a010c56a5441a2e89355c7027d"}
Sep 07 10:35:47.773351 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.773265    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx" event={"ID":"7216486c-9719-428c-89b5-a30e80feccd6","Type":"ContainerDied","Data":"525a2af9acf8c2c3f4bf6ca4e642b65a94366cddb0eefd4f0551f5ed10aa0013"}
Sep 07 10:35:47.773351 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.773289    2562 scope.go:117] "RemoveContainer" containerID="d9228ff3e572989c78e7e6ae2b4e3078ac8876a010c56a5441a2e89355c7027d"
Sep 07 10:35:47.783435 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.783416    2562 scope.go:117] "RemoveContainer" containerID="d9228ff3e572989c78e7e6ae2b4e3078ac8876a010c56a5441a2e89355c7027d"
Sep 07 10:35:47.783794 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:35:47.783741    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9228ff3e572989c78e7e6ae2b4e3078ac8876a010c56a5441a2e89355c7027d\": container with ID starting with d9228ff3e572989c78e7e6ae2b4e3078ac8876a010c56a5441a2e89355c7027d not found: ID does not exist" containerID="d9228ff3e572989c78e7e6ae2b4e3078ac8876a010c56a5441a2e89355c7027d"
Sep 07 10:35:47.783895 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.783805    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9228ff3e572989c78e7e6ae2b4e3078ac8876a010c56a5441a2e89355c7027d"} err="failed to get container status \"d9228ff3e572989c78e7e6ae2b4e3078ac8876a010c56a5441a2e89355c7027d\": rpc error: code = NotFound desc = could not find container \"d9228ff3e572989c78e7e6ae2b4e3078ac8876a010c56a5441a2e89355c7027d\": container with ID starting with d9228ff3e572989c78e7e6ae2b4e3078ac8876a010c56a5441a2e89355c7027d not found: ID does not exist"
Sep 07 10:35:47.796952 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.796920    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"]
Sep 07 10:35:47.799399 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.799376    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fbea94-f884c989d-xskbx"]
Sep 07 10:35:47.966607 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:47.966570    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7216486c-9719-428c-89b5-a30e80feccd6" path="/var/lib/kubelet/pods/7216486c-9719-428c-89b5-a30e80feccd6/volumes"
Sep 07 10:35:50.780707 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:35:50.780672    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"]
Sep 07 10:36:05.802643 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:05.802567    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk" podUID="8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb" containerName="main" containerID="cri-o://d43fb42103604565c75b172bbc68360858cf59bc4ddf325c86dd86938ee5c7af" gracePeriod=45
Sep 07 10:36:06.076955 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.076928    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:36:06.143588 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.143553    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-model-cache\") pod \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") "
Sep 07 10:36:06.143775 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.143704    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-dshm\") pod \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") "
Sep 07 10:36:06.143834 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.143751    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-tls-certs\") pod \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") "
Sep 07 10:36:06.143834 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.143827    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-tmp-dir\") pod \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") "
Sep 07 10:36:06.143950 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.143860    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb" (UID: "8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:36:06.143950 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.143909    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhgjk\" (UniqueName: \"kubernetes.io/projected/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-kube-api-access-rhgjk\") pod \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") "
Sep 07 10:36:06.143950 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.143943    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-home\") pod \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\" (UID: \"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb\") "
Sep 07 10:36:06.144142 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.144121    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb" (UID: "8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:36:06.144342 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.144321    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-tmp-dir\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:06.144430 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.144342    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-model-cache\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:06.144430 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.144352    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-home" (OuterVolumeSpecName: "home") pod "8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb" (UID: "8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:36:06.146261 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.146225    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb" (UID: "8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:36:06.146526 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.146488    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-kube-api-access-rhgjk" (OuterVolumeSpecName: "kube-api-access-rhgjk") pod "8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb" (UID: "8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb"). InnerVolumeSpecName "kube-api-access-rhgjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:36:06.146624 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.146517    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-dshm" (OuterVolumeSpecName: "dshm") pod "8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb" (UID: "8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:36:06.245029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.244989    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-dshm\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:06.245029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.245027    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-tls-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:06.245227 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.245042    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhgjk\" (UniqueName: \"kubernetes.io/projected/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-kube-api-access-rhgjk\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:06.245227 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.245059    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb-home\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:06.893439 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.893401    2562 generic.go:356] "Generic (PLEG): container finished" podID="8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb" containerID="d43fb42103604565c75b172bbc68360858cf59bc4ddf325c86dd86938ee5c7af" exitCode=0
Sep 07 10:36:06.893930 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.893444    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"
Sep 07 10:36:06.893930 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.893479    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk" event={"ID":"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb","Type":"ContainerDied","Data":"d43fb42103604565c75b172bbc68360858cf59bc4ddf325c86dd86938ee5c7af"}
Sep 07 10:36:06.893930 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.893520    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk" event={"ID":"8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb","Type":"ContainerDied","Data":"d6d5d71f882d259b1026d9c156b372cc044e934377ba16034eb92199469f064a"}
Sep 07 10:36:06.893930 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.893544    2562 scope.go:117] "RemoveContainer" containerID="d43fb42103604565c75b172bbc68360858cf59bc4ddf325c86dd86938ee5c7af"
Sep 07 10:36:06.903116 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.903096    2562 scope.go:117] "RemoveContainer" containerID="d43fb42103604565c75b172bbc68360858cf59bc4ddf325c86dd86938ee5c7af"
Sep 07 10:36:06.903382 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:36:06.903358    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d43fb42103604565c75b172bbc68360858cf59bc4ddf325c86dd86938ee5c7af\": container with ID starting with d43fb42103604565c75b172bbc68360858cf59bc4ddf325c86dd86938ee5c7af not found: ID does not exist" containerID="d43fb42103604565c75b172bbc68360858cf59bc4ddf325c86dd86938ee5c7af"
Sep 07 10:36:06.903422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.903393    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43fb42103604565c75b172bbc68360858cf59bc4ddf325c86dd86938ee5c7af"} err="failed to get container status \"d43fb42103604565c75b172bbc68360858cf59bc4ddf325c86dd86938ee5c7af\": rpc error: code = NotFound desc = could not find container \"d43fb42103604565c75b172bbc68360858cf59bc4ddf325c86dd86938ee5c7af\": container with ID starting with d43fb42103604565c75b172bbc68360858cf59bc4ddf325c86dd86938ee5c7af not found: ID does not exist"
Sep 07 10:36:06.921756 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.921724    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"]
Sep 07 10:36:06.929025 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:06.928998    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/sub-test-model-0a577c-kserve-7896bfd5c7-kxkrk"]
Sep 07 10:36:07.966579 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:07.966546    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb" path="/var/lib/kubelet/pods/8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb/volumes"
Sep 07 10:36:09.904070 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:09.904030    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"]
Sep 07 10:36:09.904542 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:09.904301    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7" podUID="2ba4ee9f-077a-40f1-938b-2036a404008a" containerName="maas-api" containerID="cri-o://0ab08604d6ce12c7a7e0f51d3d41dbd6ac4da2a9d762c25a98ea169116286e66" gracePeriod=30
Sep 07 10:36:09.983198 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:09.983155    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"]
Sep 07 10:36:09.983551 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:09.983444    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq" podUID="122582ef-848a-4551-8ac4-121103416746" containerName="payload-processing" containerID="cri-o://e4902cbb332c70b151cc838171312fed29d8e18aec36bddd79758206d081327a" gracePeriod=30
Sep 07 10:36:09.995351 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:09.995223    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"]
Sep 07 10:36:09.995505 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:09.995458    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4" podUID="e6a1b364-2910-4b02-801a-940de8b5a286" containerName="payload-pre-processing" containerID="cri-o://b5ceeab8ba339d54152a7de8c500eba0b31eda11dd46944f8cd62685f2acbe4b" gracePeriod=30
Sep 07 10:36:10.303356 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.303330    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:36:10.306635 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.306610    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"
Sep 07 10:36:10.310190 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.310169    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"
Sep 07 10:36:10.384723 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.384692    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ba4ee9f-077a-40f1-938b-2036a404008a-maas-api-tls\") pod \"2ba4ee9f-077a-40f1-938b-2036a404008a\" (UID: \"2ba4ee9f-077a-40f1-938b-2036a404008a\") "
Sep 07 10:36:10.384936 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.384773    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ba4ee9f-077a-40f1-938b-2036a404008a-metrics-certs\") pod \"2ba4ee9f-077a-40f1-938b-2036a404008a\" (UID: \"2ba4ee9f-077a-40f1-938b-2036a404008a\") "
Sep 07 10:36:10.384936 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.384841    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r65xw\" (UniqueName: \"kubernetes.io/projected/2ba4ee9f-077a-40f1-938b-2036a404008a-kube-api-access-r65xw\") pod \"2ba4ee9f-077a-40f1-938b-2036a404008a\" (UID: \"2ba4ee9f-077a-40f1-938b-2036a404008a\") "
Sep 07 10:36:10.384936 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.384914    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba4ee9f-077a-40f1-938b-2036a404008a-openshift-service-ca\") pod \"2ba4ee9f-077a-40f1-938b-2036a404008a\" (UID: \"2ba4ee9f-077a-40f1-938b-2036a404008a\") "
Sep 07 10:36:10.385102 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.384949    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122582ef-848a-4551-8ac4-121103416746-plugins-config-volume\") pod \"122582ef-848a-4551-8ac4-121103416746\" (UID: \"122582ef-848a-4551-8ac4-121103416746\") "
Sep 07 10:36:10.385102 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.385024    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a1b364-2910-4b02-801a-940de8b5a286-plugins-config-volume\") pod \"e6a1b364-2910-4b02-801a-940de8b5a286\" (UID: \"e6a1b364-2910-4b02-801a-940de8b5a286\") "
Sep 07 10:36:10.385102 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.385051    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cx8dr\" (UniqueName: \"kubernetes.io/projected/122582ef-848a-4551-8ac4-121103416746-kube-api-access-cx8dr\") pod \"122582ef-848a-4551-8ac4-121103416746\" (UID: \"122582ef-848a-4551-8ac4-121103416746\") "
Sep 07 10:36:10.385102 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.385074    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzqvs\" (UniqueName: \"kubernetes.io/projected/e6a1b364-2910-4b02-801a-940de8b5a286-kube-api-access-nzqvs\") pod \"e6a1b364-2910-4b02-801a-940de8b5a286\" (UID: \"e6a1b364-2910-4b02-801a-940de8b5a286\") "
Sep 07 10:36:10.385570 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.385502    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a1b364-2910-4b02-801a-940de8b5a286-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6a1b364-2910-4b02-801a-940de8b5a286" (UID: "e6a1b364-2910-4b02-801a-940de8b5a286"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:36:10.385570 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.385518    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122582ef-848a-4551-8ac4-121103416746-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "122582ef-848a-4551-8ac4-121103416746" (UID: "122582ef-848a-4551-8ac4-121103416746"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:36:10.385944 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.385913    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba4ee9f-077a-40f1-938b-2036a404008a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2ba4ee9f-077a-40f1-938b-2036a404008a" (UID: "2ba4ee9f-077a-40f1-938b-2036a404008a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:36:10.387630 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.387600    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba4ee9f-077a-40f1-938b-2036a404008a-kube-api-access-r65xw" (OuterVolumeSpecName: "kube-api-access-r65xw") pod "2ba4ee9f-077a-40f1-938b-2036a404008a" (UID: "2ba4ee9f-077a-40f1-938b-2036a404008a"). InnerVolumeSpecName "kube-api-access-r65xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:36:10.388149 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.388115    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba4ee9f-077a-40f1-938b-2036a404008a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2ba4ee9f-077a-40f1-938b-2036a404008a" (UID: "2ba4ee9f-077a-40f1-938b-2036a404008a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:36:10.388373 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.388319    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122582ef-848a-4551-8ac4-121103416746-kube-api-access-cx8dr" (OuterVolumeSpecName: "kube-api-access-cx8dr") pod "122582ef-848a-4551-8ac4-121103416746" (UID: "122582ef-848a-4551-8ac4-121103416746"). InnerVolumeSpecName "kube-api-access-cx8dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:36:10.388477 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.388450    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a1b364-2910-4b02-801a-940de8b5a286-kube-api-access-nzqvs" (OuterVolumeSpecName: "kube-api-access-nzqvs") pod "e6a1b364-2910-4b02-801a-940de8b5a286" (UID: "e6a1b364-2910-4b02-801a-940de8b5a286"). InnerVolumeSpecName "kube-api-access-nzqvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:36:10.388750 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.388729    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba4ee9f-077a-40f1-938b-2036a404008a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2ba4ee9f-077a-40f1-938b-2036a404008a" (UID: "2ba4ee9f-077a-40f1-938b-2036a404008a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:36:10.486494 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.486451    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba4ee9f-077a-40f1-938b-2036a404008a-openshift-service-ca\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:10.486494 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.486488    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122582ef-848a-4551-8ac4-121103416746-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:10.486494 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.486498    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a1b364-2910-4b02-801a-940de8b5a286-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:10.486748 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.486509    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx8dr\" (UniqueName: \"kubernetes.io/projected/122582ef-848a-4551-8ac4-121103416746-kube-api-access-cx8dr\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:10.486748 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.486518    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzqvs\" (UniqueName: \"kubernetes.io/projected/e6a1b364-2910-4b02-801a-940de8b5a286-kube-api-access-nzqvs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:10.486748 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.486528    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ba4ee9f-077a-40f1-938b-2036a404008a-maas-api-tls\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:10.486748 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.486538    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ba4ee9f-077a-40f1-938b-2036a404008a-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:10.486748 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.486546    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r65xw\" (UniqueName: \"kubernetes.io/projected/2ba4ee9f-077a-40f1-938b-2036a404008a-kube-api-access-r65xw\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:10.920278 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.920239    2562 generic.go:356] "Generic (PLEG): container finished" podID="2ba4ee9f-077a-40f1-938b-2036a404008a" containerID="0ab08604d6ce12c7a7e0f51d3d41dbd6ac4da2a9d762c25a98ea169116286e66" exitCode=0
Sep 07 10:36:10.920719 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.920306    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"
Sep 07 10:36:10.920719 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.920381    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7" event={"ID":"2ba4ee9f-077a-40f1-938b-2036a404008a","Type":"ContainerDied","Data":"0ab08604d6ce12c7a7e0f51d3d41dbd6ac4da2a9d762c25a98ea169116286e66"}
Sep 07 10:36:10.920719 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.920455    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7" event={"ID":"2ba4ee9f-077a-40f1-938b-2036a404008a","Type":"ContainerDied","Data":"46097bc6b96ea19345df516940e584d662285c748b280d3a04e3a4b70664afe6"}
Sep 07 10:36:10.920719 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.920488    2562 scope.go:117] "RemoveContainer" containerID="0ab08604d6ce12c7a7e0f51d3d41dbd6ac4da2a9d762c25a98ea169116286e66"
Sep 07 10:36:10.922096 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.922070    2562 generic.go:356] "Generic (PLEG): container finished" podID="e6a1b364-2910-4b02-801a-940de8b5a286" containerID="b5ceeab8ba339d54152a7de8c500eba0b31eda11dd46944f8cd62685f2acbe4b" exitCode=0
Sep 07 10:36:10.922189 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.922125    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"
Sep 07 10:36:10.922189 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.922161    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4" event={"ID":"e6a1b364-2910-4b02-801a-940de8b5a286","Type":"ContainerDied","Data":"b5ceeab8ba339d54152a7de8c500eba0b31eda11dd46944f8cd62685f2acbe4b"}
Sep 07 10:36:10.922294 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.922212    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4" event={"ID":"e6a1b364-2910-4b02-801a-940de8b5a286","Type":"ContainerDied","Data":"9ef8f310d3c27f5ccb003c8d5a75fddbb0a9f4b0c638ab8f06cd3394489acb88"}
Sep 07 10:36:10.923921 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.923893    2562 generic.go:356] "Generic (PLEG): container finished" podID="122582ef-848a-4551-8ac4-121103416746" containerID="e4902cbb332c70b151cc838171312fed29d8e18aec36bddd79758206d081327a" exitCode=0
Sep 07 10:36:10.924031 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.923941    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq" event={"ID":"122582ef-848a-4551-8ac4-121103416746","Type":"ContainerDied","Data":"e4902cbb332c70b151cc838171312fed29d8e18aec36bddd79758206d081327a"}
Sep 07 10:36:10.924031 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.923979    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"
Sep 07 10:36:10.924150 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.923980    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq" event={"ID":"122582ef-848a-4551-8ac4-121103416746","Type":"ContainerDied","Data":"3b23f616e6fc96396cfd6a82f40204380317bccb123b1c63f77e021429a33c10"}
Sep 07 10:36:10.933891 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.933870    2562 scope.go:117] "RemoveContainer" containerID="0ab08604d6ce12c7a7e0f51d3d41dbd6ac4da2a9d762c25a98ea169116286e66"
Sep 07 10:36:10.934351 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:36:10.934320    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab08604d6ce12c7a7e0f51d3d41dbd6ac4da2a9d762c25a98ea169116286e66\": container with ID starting with 0ab08604d6ce12c7a7e0f51d3d41dbd6ac4da2a9d762c25a98ea169116286e66 not found: ID does not exist" containerID="0ab08604d6ce12c7a7e0f51d3d41dbd6ac4da2a9d762c25a98ea169116286e66"
Sep 07 10:36:10.934471 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.934362    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab08604d6ce12c7a7e0f51d3d41dbd6ac4da2a9d762c25a98ea169116286e66"} err="failed to get container status \"0ab08604d6ce12c7a7e0f51d3d41dbd6ac4da2a9d762c25a98ea169116286e66\": rpc error: code = NotFound desc = could not find container \"0ab08604d6ce12c7a7e0f51d3d41dbd6ac4da2a9d762c25a98ea169116286e66\": container with ID starting with 0ab08604d6ce12c7a7e0f51d3d41dbd6ac4da2a9d762c25a98ea169116286e66 not found: ID does not exist"
Sep 07 10:36:10.934471 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.934384    2562 scope.go:117] "RemoveContainer" containerID="b5ceeab8ba339d54152a7de8c500eba0b31eda11dd46944f8cd62685f2acbe4b"
Sep 07 10:36:10.944745 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.944728    2562 scope.go:117] "RemoveContainer" containerID="b5ceeab8ba339d54152a7de8c500eba0b31eda11dd46944f8cd62685f2acbe4b"
Sep 07 10:36:10.945095 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:36:10.945074    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ceeab8ba339d54152a7de8c500eba0b31eda11dd46944f8cd62685f2acbe4b\": container with ID starting with b5ceeab8ba339d54152a7de8c500eba0b31eda11dd46944f8cd62685f2acbe4b not found: ID does not exist" containerID="b5ceeab8ba339d54152a7de8c500eba0b31eda11dd46944f8cd62685f2acbe4b"
Sep 07 10:36:10.945185 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.945100    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ceeab8ba339d54152a7de8c500eba0b31eda11dd46944f8cd62685f2acbe4b"} err="failed to get container status \"b5ceeab8ba339d54152a7de8c500eba0b31eda11dd46944f8cd62685f2acbe4b\": rpc error: code = NotFound desc = could not find container \"b5ceeab8ba339d54152a7de8c500eba0b31eda11dd46944f8cd62685f2acbe4b\": container with ID starting with b5ceeab8ba339d54152a7de8c500eba0b31eda11dd46944f8cd62685f2acbe4b not found: ID does not exist"
Sep 07 10:36:10.945185 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.945118    2562 scope.go:117] "RemoveContainer" containerID="e4902cbb332c70b151cc838171312fed29d8e18aec36bddd79758206d081327a"
Sep 07 10:36:10.953375 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.953358    2562 scope.go:117] "RemoveContainer" containerID="e4902cbb332c70b151cc838171312fed29d8e18aec36bddd79758206d081327a"
Sep 07 10:36:10.953643 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:36:10.953619    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4902cbb332c70b151cc838171312fed29d8e18aec36bddd79758206d081327a\": container with ID starting with e4902cbb332c70b151cc838171312fed29d8e18aec36bddd79758206d081327a not found: ID does not exist" containerID="e4902cbb332c70b151cc838171312fed29d8e18aec36bddd79758206d081327a"
Sep 07 10:36:10.953686 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.953650    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4902cbb332c70b151cc838171312fed29d8e18aec36bddd79758206d081327a"} err="failed to get container status \"e4902cbb332c70b151cc838171312fed29d8e18aec36bddd79758206d081327a\": rpc error: code = NotFound desc = could not find container \"e4902cbb332c70b151cc838171312fed29d8e18aec36bddd79758206d081327a\": container with ID starting with e4902cbb332c70b151cc838171312fed29d8e18aec36bddd79758206d081327a not found: ID does not exist"
Sep 07 10:36:10.966218 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.966191    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"]
Sep 07 10:36:10.968593 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:10.968571    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-695a13-7d7b57c7bf-mj4k7"]
Sep 07 10:36:11.001049 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:11.001016    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"]
Sep 07 10:36:11.007606 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:11.007577    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-695a13-7cd6f7458c-7lcqq"]
Sep 07 10:36:11.037231 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:11.037198    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"]
Sep 07 10:36:11.052303 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:11.052269    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-695a13-74d7b8f758-z8zm4"]
Sep 07 10:36:11.965445 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:11.965405    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="122582ef-848a-4551-8ac4-121103416746" path="/var/lib/kubelet/pods/122582ef-848a-4551-8ac4-121103416746/volumes"
Sep 07 10:36:11.966017 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:11.965997    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ba4ee9f-077a-40f1-938b-2036a404008a" path="/var/lib/kubelet/pods/2ba4ee9f-077a-40f1-938b-2036a404008a/volumes"
Sep 07 10:36:11.966386 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:11.966368    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6a1b364-2910-4b02-801a-940de8b5a286" path="/var/lib/kubelet/pods/e6a1b364-2910-4b02-801a-940de8b5a286/volumes"
Sep 07 10:36:26.395790 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.395727    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"]
Sep 07 10:36:26.396226 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.396209    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7216486c-9719-428c-89b5-a30e80feccd6" containerName="payload-pre-processing"
Sep 07 10:36:26.396270 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.396228    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7216486c-9719-428c-89b5-a30e80feccd6" containerName="payload-pre-processing"
Sep 07 10:36:26.396270 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.396239    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6a1b364-2910-4b02-801a-940de8b5a286" containerName="payload-pre-processing"
Sep 07 10:36:26.396270 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.396244    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6a1b364-2910-4b02-801a-940de8b5a286" containerName="payload-pre-processing"
Sep 07 10:36:26.396270 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.396253    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="122582ef-848a-4551-8ac4-121103416746" containerName="payload-processing"
Sep 07 10:36:26.396270 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.396259    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="122582ef-848a-4551-8ac4-121103416746" containerName="payload-processing"
Sep 07 10:36:26.396270 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.396269    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba4ee9f-077a-40f1-938b-2036a404008a" containerName="maas-api"
Sep 07 10:36:26.396453 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.396275    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba4ee9f-077a-40f1-938b-2036a404008a" containerName="maas-api"
Sep 07 10:36:26.396453 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.396289    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb" containerName="main"
Sep 07 10:36:26.396453 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.396295    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb" containerName="main"
Sep 07 10:36:26.396453 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.396363    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba4ee9f-077a-40f1-938b-2036a404008a" containerName="maas-api"
Sep 07 10:36:26.396453 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.396372    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab1cfa7-1daa-41fa-8b7d-fc4dd6e029bb" containerName="main"
Sep 07 10:36:26.396453 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.396378    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="122582ef-848a-4551-8ac4-121103416746" containerName="payload-processing"
Sep 07 10:36:26.396453 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.396385    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6a1b364-2910-4b02-801a-940de8b5a286" containerName="payload-pre-processing"
Sep 07 10:36:26.396453 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.396393    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7216486c-9719-428c-89b5-a30e80feccd6" containerName="payload-pre-processing"
Sep 07 10:36:26.401236 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.401216    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.413299 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.413272    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-93e41e\"/\"kube-root-ca.crt\""
Sep 07 10:36:26.414297 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.414265    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-93e41e\"/\"rate-test-model-93e41e-kserve-self-signed-certs\""
Sep 07 10:36:26.414297 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.414281    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-93e41e\"/\"openshift-service-ca.crt\""
Sep 07 10:36:26.414297 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.414265    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-93e41e\"/\"default-dockercfg-pwz5t\""
Sep 07 10:36:26.439707 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.439622    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"]
Sep 07 10:36:26.440031 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.440000    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-tmp-dir\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.440227 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.440201    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhwh6\" (UniqueName: \"kubernetes.io/projected/36347a79-c626-44d2-8d15-8e5ec2ac8727-kube-api-access-lhwh6\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.440332 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.440289    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-home\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.440332 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.440312    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-dshm\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.440461 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.440342    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36347a79-c626-44d2-8d15-8e5ec2ac8727-tls-certs\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.440461 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.440367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-model-cache\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.541422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.541372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-home\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.541422 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.541414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-dshm\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.541678 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.541441    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36347a79-c626-44d2-8d15-8e5ec2ac8727-tls-certs\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.541678 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.541464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-model-cache\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.541678 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.541592    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-tmp-dir\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.541678 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.541628    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhwh6\" (UniqueName: \"kubernetes.io/projected/36347a79-c626-44d2-8d15-8e5ec2ac8727-kube-api-access-lhwh6\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.541954 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.541906    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-home\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.542034 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.542010    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-tmp-dir\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.542332 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.542309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-model-cache\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.543977 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.543947    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-dshm\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.544514 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.544493    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36347a79-c626-44d2-8d15-8e5ec2ac8727-tls-certs\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.561448 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.561413    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhwh6\" (UniqueName: \"kubernetes.io/projected/36347a79-c626-44d2-8d15-8e5ec2ac8727-kube-api-access-lhwh6\") pod \"rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:26.711002 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:26.710967    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:27.031791 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:27.031737    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"]
Sep 07 10:36:27.032291 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:36:27.032265    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36347a79_c626_44d2_8d15_8e5ec2ac8727.slice/crio-8a8e69427fd60c375c8507d5796f51d30920856b24257d9cb7daba0e57d8216d WatchSource:0}: Error finding container 8a8e69427fd60c375c8507d5796f51d30920856b24257d9cb7daba0e57d8216d: Status 404 returned error can't find the container with id 8a8e69427fd60c375c8507d5796f51d30920856b24257d9cb7daba0e57d8216d
Sep 07 10:36:28.027487 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:28.027449    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk" event={"ID":"36347a79-c626-44d2-8d15-8e5ec2ac8727","Type":"ContainerStarted","Data":"7fac49c20be80aca3bdc8982d8d4e13c0808681a1811d09cacaab561ed1358c3"}
Sep 07 10:36:28.027935 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:28.027500    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk" event={"ID":"36347a79-c626-44d2-8d15-8e5ec2ac8727","Type":"ContainerStarted","Data":"8a8e69427fd60c375c8507d5796f51d30920856b24257d9cb7daba0e57d8216d"}
Sep 07 10:36:28.064103 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:28.064045    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk" podStartSLOduration=2.064028398 podStartE2EDuration="2.064028398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:36:28.063289837 +0000 UTC m=+1490.560979746" watchObservedRunningTime="2026-09-07 10:36:28.064028398 +0000 UTC m=+1490.561718306"
Sep 07 10:36:29.854412 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:29.854372    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"]
Sep 07 10:36:29.854906 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:29.854609    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk" podUID="4800952d-0d92-43bc-9996-5f380634da57" containerName="maas-api" containerID="cri-o://b50a0adfb87cc2a8f64d429b9c1338607267ff11bad67f3ac8ed020347cbfa98" gracePeriod=30
Sep 07 10:36:29.975887 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:29.975859    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"]
Sep 07 10:36:29.976096 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:29.976069    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9" podUID="22dfb45e-95e7-4293-acef-03fde4269a51" containerName="payload-processing" containerID="cri-o://7ca90df97953184c878f8c782c4cf06984eaf43a1ffbaf95912a8aefb152cbb1" gracePeriod=30
Sep 07 10:36:30.042868 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.042810    2562 generic.go:356] "Generic (PLEG): container finished" podID="4800952d-0d92-43bc-9996-5f380634da57" containerID="b50a0adfb87cc2a8f64d429b9c1338607267ff11bad67f3ac8ed020347cbfa98" exitCode=0
Sep 07 10:36:30.043029 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.042885    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk" event={"ID":"4800952d-0d92-43bc-9996-5f380634da57","Type":"ContainerDied","Data":"b50a0adfb87cc2a8f64d429b9c1338607267ff11bad67f3ac8ed020347cbfa98"}
Sep 07 10:36:30.147283 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.147252    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:36:30.177822 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.177789    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4800952d-0d92-43bc-9996-5f380634da57-openshift-service-ca\") pod \"4800952d-0d92-43bc-9996-5f380634da57\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") "
Sep 07 10:36:30.178019 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.177865    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4800952d-0d92-43bc-9996-5f380634da57-maas-api-tls\") pod \"4800952d-0d92-43bc-9996-5f380634da57\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") "
Sep 07 10:36:30.178019 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.177948    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4800952d-0d92-43bc-9996-5f380634da57-metrics-certs\") pod \"4800952d-0d92-43bc-9996-5f380634da57\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") "
Sep 07 10:36:30.178019 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.178015    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8l2l\" (UniqueName: \"kubernetes.io/projected/4800952d-0d92-43bc-9996-5f380634da57-kube-api-access-x8l2l\") pod \"4800952d-0d92-43bc-9996-5f380634da57\" (UID: \"4800952d-0d92-43bc-9996-5f380634da57\") "
Sep 07 10:36:30.178264 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.178229    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4800952d-0d92-43bc-9996-5f380634da57-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4800952d-0d92-43bc-9996-5f380634da57" (UID: "4800952d-0d92-43bc-9996-5f380634da57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:36:30.178368 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.178349    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4800952d-0d92-43bc-9996-5f380634da57-openshift-service-ca\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:30.180571 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.180525    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4800952d-0d92-43bc-9996-5f380634da57-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4800952d-0d92-43bc-9996-5f380634da57" (UID: "4800952d-0d92-43bc-9996-5f380634da57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:36:30.180806 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.180749    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4800952d-0d92-43bc-9996-5f380634da57-kube-api-access-x8l2l" (OuterVolumeSpecName: "kube-api-access-x8l2l") pod "4800952d-0d92-43bc-9996-5f380634da57" (UID: "4800952d-0d92-43bc-9996-5f380634da57"). InnerVolumeSpecName "kube-api-access-x8l2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:36:30.181171 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.181147    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4800952d-0d92-43bc-9996-5f380634da57-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4800952d-0d92-43bc-9996-5f380634da57" (UID: "4800952d-0d92-43bc-9996-5f380634da57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:36:30.245615 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.245588    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"
Sep 07 10:36:30.278717 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.278683    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26nhh\" (UniqueName: \"kubernetes.io/projected/22dfb45e-95e7-4293-acef-03fde4269a51-kube-api-access-26nhh\") pod \"22dfb45e-95e7-4293-acef-03fde4269a51\" (UID: \"22dfb45e-95e7-4293-acef-03fde4269a51\") "
Sep 07 10:36:30.278886 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.278801    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22dfb45e-95e7-4293-acef-03fde4269a51-plugins-config-volume\") pod \"22dfb45e-95e7-4293-acef-03fde4269a51\" (UID: \"22dfb45e-95e7-4293-acef-03fde4269a51\") "
Sep 07 10:36:30.279142 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.279120    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4800952d-0d92-43bc-9996-5f380634da57-maas-api-tls\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:30.279246 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.279144    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4800952d-0d92-43bc-9996-5f380634da57-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:30.279246 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.279163    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8l2l\" (UniqueName: \"kubernetes.io/projected/4800952d-0d92-43bc-9996-5f380634da57-kube-api-access-x8l2l\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:30.279246 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.279160    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22dfb45e-95e7-4293-acef-03fde4269a51-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22dfb45e-95e7-4293-acef-03fde4269a51" (UID: "22dfb45e-95e7-4293-acef-03fde4269a51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:36:30.281403 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.281365    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dfb45e-95e7-4293-acef-03fde4269a51-kube-api-access-26nhh" (OuterVolumeSpecName: "kube-api-access-26nhh") pod "22dfb45e-95e7-4293-acef-03fde4269a51" (UID: "22dfb45e-95e7-4293-acef-03fde4269a51"). InnerVolumeSpecName "kube-api-access-26nhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:36:30.380411 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.380324    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22dfb45e-95e7-4293-acef-03fde4269a51-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:30.380411 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:30.380358    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26nhh\" (UniqueName: \"kubernetes.io/projected/22dfb45e-95e7-4293-acef-03fde4269a51-kube-api-access-26nhh\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:31.050221 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:31.050186    2562 generic.go:356] "Generic (PLEG): container finished" podID="22dfb45e-95e7-4293-acef-03fde4269a51" containerID="7ca90df97953184c878f8c782c4cf06984eaf43a1ffbaf95912a8aefb152cbb1" exitCode=0
Sep 07 10:36:31.050708 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:31.050232    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"
Sep 07 10:36:31.050708 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:31.050277    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9" event={"ID":"22dfb45e-95e7-4293-acef-03fde4269a51","Type":"ContainerDied","Data":"7ca90df97953184c878f8c782c4cf06984eaf43a1ffbaf95912a8aefb152cbb1"}
Sep 07 10:36:31.050708 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:31.050342    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9" event={"ID":"22dfb45e-95e7-4293-acef-03fde4269a51","Type":"ContainerDied","Data":"02a239fd592bd603632137b702611b433ce73aea3faa8c206dadea6961d8d81a"}
Sep 07 10:36:31.050708 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:31.050379    2562 scope.go:117] "RemoveContainer" containerID="7ca90df97953184c878f8c782c4cf06984eaf43a1ffbaf95912a8aefb152cbb1"
Sep 07 10:36:31.051943 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:31.051922    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk" event={"ID":"4800952d-0d92-43bc-9996-5f380634da57","Type":"ContainerDied","Data":"ed37f96c31df8ab6d1770a3793980d3db5e202e342855d362df8a7bd591272f6"}
Sep 07 10:36:31.052007 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:31.051940    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"
Sep 07 10:36:31.062690 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:31.062670    2562 scope.go:117] "RemoveContainer" containerID="7ca90df97953184c878f8c782c4cf06984eaf43a1ffbaf95912a8aefb152cbb1"
Sep 07 10:36:31.063029 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:36:31.063009    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca90df97953184c878f8c782c4cf06984eaf43a1ffbaf95912a8aefb152cbb1\": container with ID starting with 7ca90df97953184c878f8c782c4cf06984eaf43a1ffbaf95912a8aefb152cbb1 not found: ID does not exist" containerID="7ca90df97953184c878f8c782c4cf06984eaf43a1ffbaf95912a8aefb152cbb1"
Sep 07 10:36:31.063110 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:31.063041    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca90df97953184c878f8c782c4cf06984eaf43a1ffbaf95912a8aefb152cbb1"} err="failed to get container status \"7ca90df97953184c878f8c782c4cf06984eaf43a1ffbaf95912a8aefb152cbb1\": rpc error: code = NotFound desc = could not find container \"7ca90df97953184c878f8c782c4cf06984eaf43a1ffbaf95912a8aefb152cbb1\": container with ID starting with 7ca90df97953184c878f8c782c4cf06984eaf43a1ffbaf95912a8aefb152cbb1 not found: ID does not exist"
Sep 07 10:36:31.063110 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:31.063068    2562 scope.go:117] "RemoveContainer" containerID="b50a0adfb87cc2a8f64d429b9c1338607267ff11bad67f3ac8ed020347cbfa98"
Sep 07 10:36:31.086142 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:31.086112    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"]
Sep 07 10:36:31.088383 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:31.088359    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7bdcc7-567b85d498-mwshk"]
Sep 07 10:36:31.112541 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:31.112513    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"]
Sep 07 10:36:31.120876 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:31.120810    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7bdcc7-7cd6f49b69-m2vg9"]
Sep 07 10:36:31.965544 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:31.965507    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22dfb45e-95e7-4293-acef-03fde4269a51" path="/var/lib/kubelet/pods/22dfb45e-95e7-4293-acef-03fde4269a51/volumes"
Sep 07 10:36:31.965946 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:31.965931    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4800952d-0d92-43bc-9996-5f380634da57" path="/var/lib/kubelet/pods/4800952d-0d92-43bc-9996-5f380634da57/volumes"
Sep 07 10:36:36.711726 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:36.711681    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:36.712158 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:36.711739    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:36.724518 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:36.724478    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:37.104797 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:37.104685    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:36:50.019652 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:50.019614    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"]
Sep 07 10:36:50.020263 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:50.019899    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh" podUID="b37e438a-00cd-4a5e-822a-034d9674a18b" containerName="payload-pre-processing" containerID="cri-o://9fa4ea6035724a3720cdd6240354d047a040d2954a6458f47a7ee70809457dc7" gracePeriod=30
Sep 07 10:36:50.172725 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:50.172636    2562 generic.go:356] "Generic (PLEG): container finished" podID="b37e438a-00cd-4a5e-822a-034d9674a18b" containerID="9fa4ea6035724a3720cdd6240354d047a040d2954a6458f47a7ee70809457dc7" exitCode=0
Sep 07 10:36:50.172725 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:50.172713    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh" event={"ID":"b37e438a-00cd-4a5e-822a-034d9674a18b","Type":"ContainerDied","Data":"9fa4ea6035724a3720cdd6240354d047a040d2954a6458f47a7ee70809457dc7"}
Sep 07 10:36:50.288994 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:50.288966    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"
Sep 07 10:36:50.364657 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:50.364627    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x24pz\" (UniqueName: \"kubernetes.io/projected/b37e438a-00cd-4a5e-822a-034d9674a18b-kube-api-access-x24pz\") pod \"b37e438a-00cd-4a5e-822a-034d9674a18b\" (UID: \"b37e438a-00cd-4a5e-822a-034d9674a18b\") "
Sep 07 10:36:50.364822 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:50.364715    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37e438a-00cd-4a5e-822a-034d9674a18b-plugins-config-volume\") pod \"b37e438a-00cd-4a5e-822a-034d9674a18b\" (UID: \"b37e438a-00cd-4a5e-822a-034d9674a18b\") "
Sep 07 10:36:50.365078 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:50.365052    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37e438a-00cd-4a5e-822a-034d9674a18b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b37e438a-00cd-4a5e-822a-034d9674a18b" (UID: "b37e438a-00cd-4a5e-822a-034d9674a18b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:36:50.366779 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:50.366743    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37e438a-00cd-4a5e-822a-034d9674a18b-kube-api-access-x24pz" (OuterVolumeSpecName: "kube-api-access-x24pz") pod "b37e438a-00cd-4a5e-822a-034d9674a18b" (UID: "b37e438a-00cd-4a5e-822a-034d9674a18b"). InnerVolumeSpecName "kube-api-access-x24pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:36:50.466148 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:50.466108    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x24pz\" (UniqueName: \"kubernetes.io/projected/b37e438a-00cd-4a5e-822a-034d9674a18b-kube-api-access-x24pz\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:50.466148 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:50.466140    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37e438a-00cd-4a5e-822a-034d9674a18b-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:36:51.179785 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:51.179733    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh" event={"ID":"b37e438a-00cd-4a5e-822a-034d9674a18b","Type":"ContainerDied","Data":"0861208983b4fbfd853559e8118e8db71f16839267c854c92a9b17c794b86f82"}
Sep 07 10:36:51.180221 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:51.179822    2562 scope.go:117] "RemoveContainer" containerID="9fa4ea6035724a3720cdd6240354d047a040d2954a6458f47a7ee70809457dc7"
Sep 07 10:36:51.180221 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:51.179745    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"
Sep 07 10:36:51.212045 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:51.212011    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"]
Sep 07 10:36:51.214082 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:51.214060    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-319d4c-b6577d6c4-7d7jh"]
Sep 07 10:36:51.965513 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:36:51.965475    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b37e438a-00cd-4a5e-822a-034d9674a18b" path="/var/lib/kubelet/pods/b37e438a-00cd-4a5e-822a-034d9674a18b/volumes"
Sep 07 10:37:25.934121 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:25.934083    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"]
Sep 07 10:37:32.341403 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.341360    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"]
Sep 07 10:37:32.341995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.341962    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b37e438a-00cd-4a5e-822a-034d9674a18b" containerName="payload-pre-processing"
Sep 07 10:37:32.341995 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.341977    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b37e438a-00cd-4a5e-822a-034d9674a18b" containerName="payload-pre-processing"
Sep 07 10:37:32.342121 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.342013    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22dfb45e-95e7-4293-acef-03fde4269a51" containerName="payload-processing"
Sep 07 10:37:32.342121 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.342022    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="22dfb45e-95e7-4293-acef-03fde4269a51" containerName="payload-processing"
Sep 07 10:37:32.342121 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.342042    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4800952d-0d92-43bc-9996-5f380634da57" containerName="maas-api"
Sep 07 10:37:32.342121 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.342052    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4800952d-0d92-43bc-9996-5f380634da57" containerName="maas-api"
Sep 07 10:37:32.342328 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.342135    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4800952d-0d92-43bc-9996-5f380634da57" containerName="maas-api"
Sep 07 10:37:32.342328 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.342150    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b37e438a-00cd-4a5e-822a-034d9674a18b" containerName="payload-pre-processing"
Sep 07 10:37:32.342328 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.342159    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="22dfb45e-95e7-4293-acef-03fde4269a51" containerName="payload-processing"
Sep 07 10:37:32.345396 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.345371    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:37:32.348180 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.348157    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-92aa2e-serving-cert\""
Sep 07 10:37:32.358003 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.357973    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"]
Sep 07 10:37:32.459854 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.459809    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh4d8\" (UniqueName: \"kubernetes.io/projected/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-kube-api-access-bh4d8\") pod \"maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5\" (UID: \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:37:32.460062 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.459867    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5\" (UID: \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:37:32.460062 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.459961    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-metrics-certs\") pod \"maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5\" (UID: \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:37:32.460062 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.460033    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-maas-api-tls\") pod \"maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5\" (UID: \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:37:32.561062 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.561029    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-metrics-certs\") pod \"maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5\" (UID: \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:37:32.561277 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.561090    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-maas-api-tls\") pod \"maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5\" (UID: \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:37:32.561277 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.561130    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh4d8\" (UniqueName: \"kubernetes.io/projected/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-kube-api-access-bh4d8\") pod \"maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5\" (UID: \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:37:32.561277 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.561156    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5\" (UID: \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:37:32.561963 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.561938    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5\" (UID: \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:37:32.563601 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.563577    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-maas-api-tls\") pod \"maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5\" (UID: \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:37:32.563702 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.563640    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-metrics-certs\") pod \"maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5\" (UID: \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:37:32.579036 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.579004    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh4d8\" (UniqueName: \"kubernetes.io/projected/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-kube-api-access-bh4d8\") pod \"maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5\" (UID: \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:37:32.657312 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.657228    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:37:32.954678 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:37:32.954647    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ebf8a1_11e9_4182_b405_2f7e10bf22ca.slice/crio-1e709523625d6d301348c4dad35a63e85f44ce07c934f50035f9eac2920060b9 WatchSource:0}: Error finding container 1e709523625d6d301348c4dad35a63e85f44ce07c934f50035f9eac2920060b9: Status 404 returned error can't find the container with id 1e709523625d6d301348c4dad35a63e85f44ce07c934f50035f9eac2920060b9
Sep 07 10:37:32.955800 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:32.955775    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"]
Sep 07 10:37:33.416047 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:33.415964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5" event={"ID":"68ebf8a1-11e9-4182-b405-2f7e10bf22ca","Type":"ContainerStarted","Data":"1e709523625d6d301348c4dad35a63e85f44ce07c934f50035f9eac2920060b9"}
Sep 07 10:37:34.423844 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:34.423796    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5" event={"ID":"68ebf8a1-11e9-4182-b405-2f7e10bf22ca","Type":"ContainerStarted","Data":"8a57ef865f454d6338889e81f73a1eb67bd8fa2ea0f11c62848ce60348501e9f"}
Sep 07 10:37:34.424333 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:34.423904    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:37:38.200226 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:38.200192    2562 scope.go:117] "RemoveContainer" containerID="ea078a79164178918864194c552e90533085447e6b501d6f35ddeab9eb2e4a35"
Sep 07 10:37:38.211649 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:38.211629    2562 scope.go:117] "RemoveContainer" containerID="5364b45d1712154d8b450f5efd5e3ee11357013b7ce23edbaa891f4bc446c3e4"
Sep 07 10:37:40.433884 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:40.433853    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:37:40.470547 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:40.470487    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5" podStartSLOduration=8.0093881 podStartE2EDuration="8.470471094s" totalImagesPullingTime="461.082994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:37:34.466512585 +0000 UTC m=+1556.964202494" watchObservedRunningTime="2026-09-07 10:37:40.470471094 +0000 UTC m=+1562.968161003"
Sep 07 10:37:40.959306 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:40.959268    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk" podUID="36347a79-c626-44d2-8d15-8e5ec2ac8727" containerName="main" containerID="cri-o://7fac49c20be80aca3bdc8982d8d4e13c0808681a1811d09cacaab561ed1358c3" gracePeriod=45
Sep 07 10:37:41.205697 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.205672    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:37:41.343811 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.343694    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-model-cache\") pod \"36347a79-c626-44d2-8d15-8e5ec2ac8727\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") "
Sep 07 10:37:41.343811 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.343748    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-dshm\") pod \"36347a79-c626-44d2-8d15-8e5ec2ac8727\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") "
Sep 07 10:37:41.343811 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.343798    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36347a79-c626-44d2-8d15-8e5ec2ac8727-tls-certs\") pod \"36347a79-c626-44d2-8d15-8e5ec2ac8727\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") "
Sep 07 10:37:41.344109 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.343832    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-home\") pod \"36347a79-c626-44d2-8d15-8e5ec2ac8727\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") "
Sep 07 10:37:41.344109 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.343904    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhwh6\" (UniqueName: \"kubernetes.io/projected/36347a79-c626-44d2-8d15-8e5ec2ac8727-kube-api-access-lhwh6\") pod \"36347a79-c626-44d2-8d15-8e5ec2ac8727\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") "
Sep 07 10:37:41.344109 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.343935    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-tmp-dir\") pod \"36347a79-c626-44d2-8d15-8e5ec2ac8727\" (UID: \"36347a79-c626-44d2-8d15-8e5ec2ac8727\") "
Sep 07 10:37:41.344109 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.343988    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-model-cache" (OuterVolumeSpecName: "model-cache") pod "36347a79-c626-44d2-8d15-8e5ec2ac8727" (UID: "36347a79-c626-44d2-8d15-8e5ec2ac8727"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:37:41.344333 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.344187    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-home" (OuterVolumeSpecName: "home") pod "36347a79-c626-44d2-8d15-8e5ec2ac8727" (UID: "36347a79-c626-44d2-8d15-8e5ec2ac8727"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:37:41.344333 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.344235    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "36347a79-c626-44d2-8d15-8e5ec2ac8727" (UID: "36347a79-c626-44d2-8d15-8e5ec2ac8727"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:37:41.344333 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.344327    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-model-cache\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:37:41.344496 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.344345    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-home\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:37:41.344496 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.344362    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-tmp-dir\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:37:41.346087 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.346063    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36347a79-c626-44d2-8d15-8e5ec2ac8727-kube-api-access-lhwh6" (OuterVolumeSpecName: "kube-api-access-lhwh6") pod "36347a79-c626-44d2-8d15-8e5ec2ac8727" (UID: "36347a79-c626-44d2-8d15-8e5ec2ac8727"). InnerVolumeSpecName "kube-api-access-lhwh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:37:41.347086 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.347068    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-dshm" (OuterVolumeSpecName: "dshm") pod "36347a79-c626-44d2-8d15-8e5ec2ac8727" (UID: "36347a79-c626-44d2-8d15-8e5ec2ac8727"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:37:41.347153 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.347097    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36347a79-c626-44d2-8d15-8e5ec2ac8727-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36347a79-c626-44d2-8d15-8e5ec2ac8727" (UID: "36347a79-c626-44d2-8d15-8e5ec2ac8727"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:37:41.444821 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.444779    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36347a79-c626-44d2-8d15-8e5ec2ac8727-dshm\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:37:41.444821 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.444815    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36347a79-c626-44d2-8d15-8e5ec2ac8727-tls-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:37:41.444821 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.444826    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhwh6\" (UniqueName: \"kubernetes.io/projected/36347a79-c626-44d2-8d15-8e5ec2ac8727-kube-api-access-lhwh6\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:37:41.467449 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.467412    2562 generic.go:356] "Generic (PLEG): container finished" podID="36347a79-c626-44d2-8d15-8e5ec2ac8727" containerID="7fac49c20be80aca3bdc8982d8d4e13c0808681a1811d09cacaab561ed1358c3" exitCode=0
Sep 07 10:37:41.467612 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.467457    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"
Sep 07 10:37:41.467612 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.467491    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk" event={"ID":"36347a79-c626-44d2-8d15-8e5ec2ac8727","Type":"ContainerDied","Data":"7fac49c20be80aca3bdc8982d8d4e13c0808681a1811d09cacaab561ed1358c3"}
Sep 07 10:37:41.467612 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.467543    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk" event={"ID":"36347a79-c626-44d2-8d15-8e5ec2ac8727","Type":"ContainerDied","Data":"8a8e69427fd60c375c8507d5796f51d30920856b24257d9cb7daba0e57d8216d"}
Sep 07 10:37:41.467612 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.467562    2562 scope.go:117] "RemoveContainer" containerID="7fac49c20be80aca3bdc8982d8d4e13c0808681a1811d09cacaab561ed1358c3"
Sep 07 10:37:41.477670 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.477650    2562 scope.go:117] "RemoveContainer" containerID="7fac49c20be80aca3bdc8982d8d4e13c0808681a1811d09cacaab561ed1358c3"
Sep 07 10:37:41.477980 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:37:41.477959    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fac49c20be80aca3bdc8982d8d4e13c0808681a1811d09cacaab561ed1358c3\": container with ID starting with 7fac49c20be80aca3bdc8982d8d4e13c0808681a1811d09cacaab561ed1358c3 not found: ID does not exist" containerID="7fac49c20be80aca3bdc8982d8d4e13c0808681a1811d09cacaab561ed1358c3"
Sep 07 10:37:41.478032 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.477988    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fac49c20be80aca3bdc8982d8d4e13c0808681a1811d09cacaab561ed1358c3"} err="failed to get container status \"7fac49c20be80aca3bdc8982d8d4e13c0808681a1811d09cacaab561ed1358c3\": rpc error: code = NotFound desc = could not find container \"7fac49c20be80aca3bdc8982d8d4e13c0808681a1811d09cacaab561ed1358c3\": container with ID starting with 7fac49c20be80aca3bdc8982d8d4e13c0808681a1811d09cacaab561ed1358c3 not found: ID does not exist"
Sep 07 10:37:41.495502 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.495460    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"]
Sep 07 10:37:41.501450 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.501420    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/rate-test-model-93e41e-kserve-785c7fdd6c-k9hkk"]
Sep 07 10:37:41.969879 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:37:41.969841    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36347a79-c626-44d2-8d15-8e5ec2ac8727" path="/var/lib/kubelet/pods/36347a79-c626-44d2-8d15-8e5ec2ac8727/volumes"
Sep 07 10:38:00.877405 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:00.877361    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"]
Sep 07 10:38:00.879834 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:00.877969    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36347a79-c626-44d2-8d15-8e5ec2ac8727" containerName="main"
Sep 07 10:38:00.879834 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:00.877985    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="36347a79-c626-44d2-8d15-8e5ec2ac8727" containerName="main"
Sep 07 10:38:00.879834 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:00.878100    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="36347a79-c626-44d2-8d15-8e5ec2ac8727" containerName="main"
Sep 07 10:38:00.881259 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:00.881242    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:00.885058 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:00.885024    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-125f28-serving-cert\""
Sep 07 10:38:00.893450 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:00.893427    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"]
Sep 07 10:38:00.925492 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:00.925460    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a34ccbc-86fc-4682-ab05-86c197754fbb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:00.925682 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:00.925522    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a34ccbc-86fc-4682-ab05-86c197754fbb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:00.925682 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:00.925662    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a34ccbc-86fc-4682-ab05-86c197754fbb-metrics-certs\") pod \"maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:00.925825 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:00.925700    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz2ff\" (UniqueName: \"kubernetes.io/projected/8a34ccbc-86fc-4682-ab05-86c197754fbb-kube-api-access-kz2ff\") pod \"maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:01.026662 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.026622    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a34ccbc-86fc-4682-ab05-86c197754fbb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:01.026848 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.026694    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a34ccbc-86fc-4682-ab05-86c197754fbb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:01.026848 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:38:01.026795    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-serving-cert: secret "maas-api-e2e-ipp-b-125f28-serving-cert" not found
Sep 07 10:38:01.026936 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:38:01.026854    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a34ccbc-86fc-4682-ab05-86c197754fbb-maas-api-tls podName:8a34ccbc-86fc-4682-ab05-86c197754fbb nodeName:}" failed. No retries permitted until 2026-09-07 10:38:01.526837452 +0000 UTC m=+1584.024527342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a34ccbc-86fc-4682-ab05-86c197754fbb-maas-api-tls") pod "maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl" (UID: "8a34ccbc-86fc-4682-ab05-86c197754fbb") : secret "maas-api-e2e-ipp-b-125f28-serving-cert" not found
Sep 07 10:38:01.026936 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.026874    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a34ccbc-86fc-4682-ab05-86c197754fbb-metrics-certs\") pod \"maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:01.026936 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.026905    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz2ff\" (UniqueName: \"kubernetes.io/projected/8a34ccbc-86fc-4682-ab05-86c197754fbb-kube-api-access-kz2ff\") pod \"maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:01.027524 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.027489    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a34ccbc-86fc-4682-ab05-86c197754fbb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:01.029338 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.029316    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a34ccbc-86fc-4682-ab05-86c197754fbb-metrics-certs\") pod \"maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:01.041631 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.041604    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz2ff\" (UniqueName: \"kubernetes.io/projected/8a34ccbc-86fc-4682-ab05-86c197754fbb-kube-api-access-kz2ff\") pod \"maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:01.531424 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.531386    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a34ccbc-86fc-4682-ab05-86c197754fbb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:01.534695 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.534467    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a34ccbc-86fc-4682-ab05-86c197754fbb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:01.636852 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.636812    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"]
Sep 07 10:38:01.641332 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.641304    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"
Sep 07 10:38:01.646402 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.646370    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-125f28\""
Sep 07 10:38:01.651516 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.651488    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"]
Sep 07 10:38:01.733407 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.733371    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqkb8\" (UniqueName: \"kubernetes.io/projected/72639a22-38c5-43ef-a2e0-6b6373cd7cf0-kube-api-access-hqkb8\") pod \"payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm\" (UID: \"72639a22-38c5-43ef-a2e0-6b6373cd7cf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"
Sep 07 10:38:01.733584 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.733478    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72639a22-38c5-43ef-a2e0-6b6373cd7cf0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm\" (UID: \"72639a22-38c5-43ef-a2e0-6b6373cd7cf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"
Sep 07 10:38:01.793753 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.793661    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:01.835917 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.834517    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72639a22-38c5-43ef-a2e0-6b6373cd7cf0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm\" (UID: \"72639a22-38c5-43ef-a2e0-6b6373cd7cf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"
Sep 07 10:38:01.835917 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.834625    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqkb8\" (UniqueName: \"kubernetes.io/projected/72639a22-38c5-43ef-a2e0-6b6373cd7cf0-kube-api-access-hqkb8\") pod \"payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm\" (UID: \"72639a22-38c5-43ef-a2e0-6b6373cd7cf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"
Sep 07 10:38:01.835917 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.835859    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72639a22-38c5-43ef-a2e0-6b6373cd7cf0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm\" (UID: \"72639a22-38c5-43ef-a2e0-6b6373cd7cf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"
Sep 07 10:38:01.844158 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.844128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqkb8\" (UniqueName: \"kubernetes.io/projected/72639a22-38c5-43ef-a2e0-6b6373cd7cf0-kube-api-access-hqkb8\") pod \"payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm\" (UID: \"72639a22-38c5-43ef-a2e0-6b6373cd7cf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"
Sep 07 10:38:01.954731 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:01.954689    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"
Sep 07 10:38:02.078431 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:02.078401    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"]
Sep 07 10:38:02.079753 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:38:02.079728    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a34ccbc_86fc_4682_ab05_86c197754fbb.slice/crio-3c1d519e3411a2a3ca13e9d9cc5383239462108d74a02b637663cf238cdcf4a7 WatchSource:0}: Error finding container 3c1d519e3411a2a3ca13e9d9cc5383239462108d74a02b637663cf238cdcf4a7: Status 404 returned error can't find the container with id 3c1d519e3411a2a3ca13e9d9cc5383239462108d74a02b637663cf238cdcf4a7
Sep 07 10:38:02.232558 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:02.232517    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"]
Sep 07 10:38:02.233863 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:38:02.233835    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72639a22_38c5_43ef_a2e0_6b6373cd7cf0.slice/crio-f6f270edff89c4ea0455712a4eea6634ad04317d0189067e22e80aeb845a9fc5 WatchSource:0}: Error finding container f6f270edff89c4ea0455712a4eea6634ad04317d0189067e22e80aeb845a9fc5: Status 404 returned error can't find the container with id f6f270edff89c4ea0455712a4eea6634ad04317d0189067e22e80aeb845a9fc5
Sep 07 10:38:02.598705 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:02.598663    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl" event={"ID":"8a34ccbc-86fc-4682-ab05-86c197754fbb","Type":"ContainerStarted","Data":"1fe024ce6df1f214fb4f639e3eb979fdf7eeb261a30245458d96e73aeeb2824e"}
Sep 07 10:38:02.598958 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:02.598734    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl" event={"ID":"8a34ccbc-86fc-4682-ab05-86c197754fbb","Type":"ContainerStarted","Data":"3c1d519e3411a2a3ca13e9d9cc5383239462108d74a02b637663cf238cdcf4a7"}
Sep 07 10:38:02.598958 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:02.598936    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:02.600997 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:02.600965    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm" event={"ID":"72639a22-38c5-43ef-a2e0-6b6373cd7cf0","Type":"ContainerStarted","Data":"0a20d30020dfa6b2c902d5d4cb9cb7d3f9e9be08e707dda7e8ea21fd18502490"}
Sep 07 10:38:02.601145 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:02.601020    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm" event={"ID":"72639a22-38c5-43ef-a2e0-6b6373cd7cf0","Type":"ContainerStarted","Data":"f6f270edff89c4ea0455712a4eea6634ad04317d0189067e22e80aeb845a9fc5"}
Sep 07 10:38:02.601145 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:02.601052    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"
Sep 07 10:38:02.619750 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:02.619698    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl" podStartSLOduration=2.233196163 podStartE2EDuration="2.619680771s" totalImagesPullingTime="386.484608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:38:02.617961731 +0000 UTC m=+1585.115651651" watchObservedRunningTime="2026-09-07 10:38:02.619680771 +0000 UTC m=+1585.117370679"
Sep 07 10:38:02.639912 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:02.639842    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm" podStartSLOduration=1.639819878 podStartE2EDuration="1.639819878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:38:02.635744656 +0000 UTC m=+1585.133434590" watchObservedRunningTime="2026-09-07 10:38:02.639819878 +0000 UTC m=+1585.137509790"
Sep 07 10:38:08.613067 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:08.613035    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:38:13.610836 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:13.610800    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"
Sep 07 10:38:29.769532 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:38:29.769493    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm_72639a22-38c5-43ef-a2e0-6b6373cd7cf0/payload-processing/0.log"
Sep 07 10:39:05.585914 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:05.585885    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm_72639a22-38c5-43ef-a2e0-6b6373cd7cf0/payload-processing/0.log"
Sep 07 10:39:10.402271 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:10.402239    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"]
Sep 07 10:39:10.405711 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:10.405689    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:10.410118 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:10.410089    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-c43305-serving-cert\""
Sep 07 10:39:10.417254 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:10.417218    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"]
Sep 07 10:39:10.562575 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:10.562534    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:10.562784 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:10.562605    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:10.562784 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:10.562633    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:10.562784 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:10.562655    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnx67\" (UniqueName: \"kubernetes.io/projected/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-kube-api-access-pnx67\") pod \"maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:10.663906 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:10.663802    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:10.663906 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:10.663847    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:10.663906 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:10.663870    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnx67\" (UniqueName: \"kubernetes.io/projected/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-kube-api-access-pnx67\") pod \"maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:10.664221 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:10.663941    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:10.664221 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:39:10.663964    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-serving-cert: secret "maas-api-e2e-ipp-cleanup-c43305-serving-cert" not found
Sep 07 10:39:10.664221 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:39:10.664043    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-maas-api-tls podName:ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8 nodeName:}" failed. No retries permitted until 2026-09-07 10:39:11.164024859 +0000 UTC m=+1653.661714766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs" (UID: "ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8") : secret "maas-api-e2e-ipp-cleanup-c43305-serving-cert" not found
Sep 07 10:39:10.664621 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:10.664602    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:10.666295 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:10.666275    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:10.674160 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:10.674142    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnx67\" (UniqueName: \"kubernetes.io/projected/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-kube-api-access-pnx67\") pod \"maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:11.168171 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.168138    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:11.170556 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.170531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:11.318503 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.318467    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:11.354000 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.353966    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"]
Sep 07 10:39:11.357726 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.357700    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"
Sep 07 10:39:11.361354 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.361331    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-c43305\""
Sep 07 10:39:11.363934 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.363911    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-c43305-dockercfg-9qk7n\""
Sep 07 10:39:11.403990 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.403949    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"]
Sep 07 10:39:11.471709 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.471672    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s92dn\" (UniqueName: \"kubernetes.io/projected/3e60a8a9-a60b-42e8-a6cc-528618314c5c-kube-api-access-s92dn\") pod \"payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp\" (UID: \"3e60a8a9-a60b-42e8-a6cc-528618314c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"
Sep 07 10:39:11.471920 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.471754    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e60a8a9-a60b-42e8-a6cc-528618314c5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp\" (UID: \"3e60a8a9-a60b-42e8-a6cc-528618314c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"
Sep 07 10:39:11.573122 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.573080    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s92dn\" (UniqueName: \"kubernetes.io/projected/3e60a8a9-a60b-42e8-a6cc-528618314c5c-kube-api-access-s92dn\") pod \"payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp\" (UID: \"3e60a8a9-a60b-42e8-a6cc-528618314c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"
Sep 07 10:39:11.573332 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.573196    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e60a8a9-a60b-42e8-a6cc-528618314c5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp\" (UID: \"3e60a8a9-a60b-42e8-a6cc-528618314c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"
Sep 07 10:39:11.573954 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.573929    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e60a8a9-a60b-42e8-a6cc-528618314c5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp\" (UID: \"3e60a8a9-a60b-42e8-a6cc-528618314c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"
Sep 07 10:39:11.583145 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.583114    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s92dn\" (UniqueName: \"kubernetes.io/projected/3e60a8a9-a60b-42e8-a6cc-528618314c5c-kube-api-access-s92dn\") pod \"payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp\" (UID: \"3e60a8a9-a60b-42e8-a6cc-528618314c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"
Sep 07 10:39:11.602565 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.602531    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"]
Sep 07 10:39:11.603942 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:39:11.603919    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9a0c07_7e69_4a54_ab9b_1bc2765f19b8.slice/crio-87942a69c98f0bf0f104945f6d458b1d957fb5874c25979e2d96a117aaa0718c WatchSource:0}: Error finding container 87942a69c98f0bf0f104945f6d458b1d957fb5874c25979e2d96a117aaa0718c: Status 404 returned error can't find the container with id 87942a69c98f0bf0f104945f6d458b1d957fb5874c25979e2d96a117aaa0718c
Sep 07 10:39:11.605297 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.605279    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 10:39:11.672677 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.672637    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"
Sep 07 10:39:11.951536 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:11.951504    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"]
Sep 07 10:39:11.952856 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:39:11.952829    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e60a8a9_a60b_42e8_a6cc_528618314c5c.slice/crio-b9b4385dd73b96123a3e77901d2fa74c133839f5478921fa97a7792cefb42888 WatchSource:0}: Error finding container b9b4385dd73b96123a3e77901d2fa74c133839f5478921fa97a7792cefb42888: Status 404 returned error can't find the container with id b9b4385dd73b96123a3e77901d2fa74c133839f5478921fa97a7792cefb42888
Sep 07 10:39:12.012008 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:12.011977    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp" event={"ID":"3e60a8a9-a60b-42e8-a6cc-528618314c5c","Type":"ContainerStarted","Data":"b9b4385dd73b96123a3e77901d2fa74c133839f5478921fa97a7792cefb42888"}
Sep 07 10:39:12.013272 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:12.013246    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs" event={"ID":"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8","Type":"ContainerStarted","Data":"87942a69c98f0bf0f104945f6d458b1d957fb5874c25979e2d96a117aaa0718c"}
Sep 07 10:39:13.020425 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:13.020386    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp" event={"ID":"3e60a8a9-a60b-42e8-a6cc-528618314c5c","Type":"ContainerStarted","Data":"7bce8a4f590c0718a38cf95f551a7e414b30a1f00852f93c044fd36175f96d10"}
Sep 07 10:39:13.020901 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:13.020459    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"
Sep 07 10:39:13.022008 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:13.021984    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs" event={"ID":"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8","Type":"ContainerStarted","Data":"1298174229c6eeb54e543a4282dac3bbb05a011f4e4a3dae55ea2e68c2233df4"}
Sep 07 10:39:13.022131 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:13.022113    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:13.042417 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:13.042358    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp" podStartSLOduration=2.042335754 podStartE2EDuration="2.042335754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:39:13.038799022 +0000 UTC m=+1655.536488955" watchObservedRunningTime="2026-09-07 10:39:13.042335754 +0000 UTC m=+1655.540025665"
Sep 07 10:39:13.059284 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:13.059208    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs" podStartSLOduration=2.680623211 podStartE2EDuration="3.059183327s" totalImagesPullingTime="378.560116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:39:13.058555399 +0000 UTC m=+1655.556245334" watchObservedRunningTime="2026-09-07 10:39:13.059183327 +0000 UTC m=+1655.556873244"
Sep 07 10:39:19.034867 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:19.034834    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:24.031698 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:24.031666    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"
Sep 07 10:39:40.891941 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:40.891901    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"]
Sep 07 10:39:40.892466 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:40.892128    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs" podUID="ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8" containerName="maas-api" containerID="cri-o://1298174229c6eeb54e543a4282dac3bbb05a011f4e4a3dae55ea2e68c2233df4" gracePeriod=30
Sep 07 10:39:40.960308 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:40.960269    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"]
Sep 07 10:39:40.960553 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:40.960506    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp" podUID="3e60a8a9-a60b-42e8-a6cc-528618314c5c" containerName="payload-pre-processing" containerID="cri-o://7bce8a4f590c0718a38cf95f551a7e414b30a1f00852f93c044fd36175f96d10" gracePeriod=30
Sep 07 10:39:41.186397 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.186359    2562 generic.go:356] "Generic (PLEG): container finished" podID="3e60a8a9-a60b-42e8-a6cc-528618314c5c" containerID="7bce8a4f590c0718a38cf95f551a7e414b30a1f00852f93c044fd36175f96d10" exitCode=0
Sep 07 10:39:41.186576 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.186514    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp" event={"ID":"3e60a8a9-a60b-42e8-a6cc-528618314c5c","Type":"ContainerDied","Data":"7bce8a4f590c0718a38cf95f551a7e414b30a1f00852f93c044fd36175f96d10"}
Sep 07 10:39:41.188507 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.188480    2562 generic.go:356] "Generic (PLEG): container finished" podID="ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8" containerID="1298174229c6eeb54e543a4282dac3bbb05a011f4e4a3dae55ea2e68c2233df4" exitCode=0
Sep 07 10:39:41.188637 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.188513    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs" event={"ID":"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8","Type":"ContainerDied","Data":"1298174229c6eeb54e543a4282dac3bbb05a011f4e4a3dae55ea2e68c2233df4"}
Sep 07 10:39:41.210010 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.209986    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:41.241349 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.241319    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-openshift-service-ca\") pod \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") "
Sep 07 10:39:41.241514 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.241365    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnx67\" (UniqueName: \"kubernetes.io/projected/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-kube-api-access-pnx67\") pod \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") "
Sep 07 10:39:41.241514 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.241436    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-metrics-certs\") pod \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") "
Sep 07 10:39:41.241514 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.241492    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-maas-api-tls\") pod \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\" (UID: \"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8\") "
Sep 07 10:39:41.241752 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.241711    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8" (UID: "ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:39:41.244086 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.244054    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8" (UID: "ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:39:41.244216 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.244193    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8" (UID: "ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:39:41.244265 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.244213    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-kube-api-access-pnx67" (OuterVolumeSpecName: "kube-api-access-pnx67") pod "ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8" (UID: "ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8"). InnerVolumeSpecName "kube-api-access-pnx67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:39:41.246065 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.246046    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"
Sep 07 10:39:41.342677 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.342636    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e60a8a9-a60b-42e8-a6cc-528618314c5c-plugins-config-volume\") pod \"3e60a8a9-a60b-42e8-a6cc-528618314c5c\" (UID: \"3e60a8a9-a60b-42e8-a6cc-528618314c5c\") "
Sep 07 10:39:41.342677 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.342684    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s92dn\" (UniqueName: \"kubernetes.io/projected/3e60a8a9-a60b-42e8-a6cc-528618314c5c-kube-api-access-s92dn\") pod \"3e60a8a9-a60b-42e8-a6cc-528618314c5c\" (UID: \"3e60a8a9-a60b-42e8-a6cc-528618314c5c\") "
Sep 07 10:39:41.342986 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.342971    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:39:41.342986 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.342986    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-maas-api-tls\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:39:41.343120 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.342996    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-openshift-service-ca\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:39:41.343120 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.343007    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnx67\" (UniqueName: \"kubernetes.io/projected/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8-kube-api-access-pnx67\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:39:41.343120 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.343049    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e60a8a9-a60b-42e8-a6cc-528618314c5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e60a8a9-a60b-42e8-a6cc-528618314c5c" (UID: "3e60a8a9-a60b-42e8-a6cc-528618314c5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:39:41.344802 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.344780    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e60a8a9-a60b-42e8-a6cc-528618314c5c-kube-api-access-s92dn" (OuterVolumeSpecName: "kube-api-access-s92dn") pod "3e60a8a9-a60b-42e8-a6cc-528618314c5c" (UID: "3e60a8a9-a60b-42e8-a6cc-528618314c5c"). InnerVolumeSpecName "kube-api-access-s92dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:39:41.444284 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.444195    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e60a8a9-a60b-42e8-a6cc-528618314c5c-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:39:41.444284 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:41.444228    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s92dn\" (UniqueName: \"kubernetes.io/projected/3e60a8a9-a60b-42e8-a6cc-528618314c5c-kube-api-access-s92dn\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:39:42.195727 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:42.195629    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"
Sep 07 10:39:42.195727 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:42.195668    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp" event={"ID":"3e60a8a9-a60b-42e8-a6cc-528618314c5c","Type":"ContainerDied","Data":"b9b4385dd73b96123a3e77901d2fa74c133839f5478921fa97a7792cefb42888"}
Sep 07 10:39:42.196241 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:42.195742    2562 scope.go:117] "RemoveContainer" containerID="7bce8a4f590c0718a38cf95f551a7e414b30a1f00852f93c044fd36175f96d10"
Sep 07 10:39:42.197246 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:42.197180    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs" event={"ID":"ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8","Type":"ContainerDied","Data":"87942a69c98f0bf0f104945f6d458b1d957fb5874c25979e2d96a117aaa0718c"}
Sep 07 10:39:42.197246 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:42.197186    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"
Sep 07 10:39:42.205369 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:42.205347    2562 scope.go:117] "RemoveContainer" containerID="1298174229c6eeb54e543a4282dac3bbb05a011f4e4a3dae55ea2e68c2233df4"
Sep 07 10:39:42.218979 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:42.218951    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"]
Sep 07 10:39:42.223449 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:42.223428    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c43305-7bff478cff-h8hhs"]
Sep 07 10:39:42.237343 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:42.237312    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"]
Sep 07 10:39:42.241415 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:42.241391    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c43305-dc67ffdc4-nw4tp"]
Sep 07 10:39:43.965228 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:43.965194    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e60a8a9-a60b-42e8-a6cc-528618314c5c" path="/var/lib/kubelet/pods/3e60a8a9-a60b-42e8-a6cc-528618314c5c/volumes"
Sep 07 10:39:43.965597 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:39:43.965580    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8" path="/var/lib/kubelet/pods/ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8/volumes"
Sep 07 10:40:06.899871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:06.899834    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"]
Sep 07 10:40:06.900252 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:06.900042    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5" podUID="68ebf8a1-11e9-4182-b405-2f7e10bf22ca" containerName="maas-api" containerID="cri-o://8a57ef865f454d6338889e81f73a1eb67bd8fa2ea0f11c62848ce60348501e9f" gracePeriod=30
Sep 07 10:40:07.165249 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.165187    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:40:07.291699 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.291667    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-maas-api-tls\") pod \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\" (UID: \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\") "
Sep 07 10:40:07.291913 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.291749    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-metrics-certs\") pod \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\" (UID: \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\") "
Sep 07 10:40:07.291913 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.291822    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh4d8\" (UniqueName: \"kubernetes.io/projected/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-kube-api-access-bh4d8\") pod \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\" (UID: \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\") "
Sep 07 10:40:07.291913 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.291846    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-openshift-service-ca\") pod \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\" (UID: \"68ebf8a1-11e9-4182-b405-2f7e10bf22ca\") "
Sep 07 10:40:07.292267 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.292233    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68ebf8a1-11e9-4182-b405-2f7e10bf22ca" (UID: "68ebf8a1-11e9-4182-b405-2f7e10bf22ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:40:07.294012 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.293985    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-kube-api-access-bh4d8" (OuterVolumeSpecName: "kube-api-access-bh4d8") pod "68ebf8a1-11e9-4182-b405-2f7e10bf22ca" (UID: "68ebf8a1-11e9-4182-b405-2f7e10bf22ca"). InnerVolumeSpecName "kube-api-access-bh4d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:40:07.294012 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.293985    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "68ebf8a1-11e9-4182-b405-2f7e10bf22ca" (UID: "68ebf8a1-11e9-4182-b405-2f7e10bf22ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:40:07.294212 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.294041    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68ebf8a1-11e9-4182-b405-2f7e10bf22ca" (UID: "68ebf8a1-11e9-4182-b405-2f7e10bf22ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:40:07.338742 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.338705    2562 generic.go:356] "Generic (PLEG): container finished" podID="68ebf8a1-11e9-4182-b405-2f7e10bf22ca" containerID="8a57ef865f454d6338889e81f73a1eb67bd8fa2ea0f11c62848ce60348501e9f" exitCode=0
Sep 07 10:40:07.338933 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.338773    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"
Sep 07 10:40:07.338933 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.338802    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5" event={"ID":"68ebf8a1-11e9-4182-b405-2f7e10bf22ca","Type":"ContainerDied","Data":"8a57ef865f454d6338889e81f73a1eb67bd8fa2ea0f11c62848ce60348501e9f"}
Sep 07 10:40:07.338933 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.338864    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5" event={"ID":"68ebf8a1-11e9-4182-b405-2f7e10bf22ca","Type":"ContainerDied","Data":"1e709523625d6d301348c4dad35a63e85f44ce07c934f50035f9eac2920060b9"}
Sep 07 10:40:07.338933 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.338887    2562 scope.go:117] "RemoveContainer" containerID="8a57ef865f454d6338889e81f73a1eb67bd8fa2ea0f11c62848ce60348501e9f"
Sep 07 10:40:07.349854 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.349831    2562 scope.go:117] "RemoveContainer" containerID="8a57ef865f454d6338889e81f73a1eb67bd8fa2ea0f11c62848ce60348501e9f"
Sep 07 10:40:07.350153 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:40:07.350131    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a57ef865f454d6338889e81f73a1eb67bd8fa2ea0f11c62848ce60348501e9f\": container with ID starting with 8a57ef865f454d6338889e81f73a1eb67bd8fa2ea0f11c62848ce60348501e9f not found: ID does not exist" containerID="8a57ef865f454d6338889e81f73a1eb67bd8fa2ea0f11c62848ce60348501e9f"
Sep 07 10:40:07.350223 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.350179    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a57ef865f454d6338889e81f73a1eb67bd8fa2ea0f11c62848ce60348501e9f"} err="failed to get container status \"8a57ef865f454d6338889e81f73a1eb67bd8fa2ea0f11c62848ce60348501e9f\": rpc error: code = NotFound desc = could not find container \"8a57ef865f454d6338889e81f73a1eb67bd8fa2ea0f11c62848ce60348501e9f\": container with ID starting with 8a57ef865f454d6338889e81f73a1eb67bd8fa2ea0f11c62848ce60348501e9f not found: ID does not exist"
Sep 07 10:40:07.360919 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.360882    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"]
Sep 07 10:40:07.365932 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.365904    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-92aa2e-5f95bb7f78-pqdz5"]
Sep 07 10:40:07.392819 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.392786    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-maas-api-tls\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:40:07.392819 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.392813    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:40:07.392819 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.392822    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh4d8\" (UniqueName: \"kubernetes.io/projected/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-kube-api-access-bh4d8\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:40:07.393016 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.392834    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ebf8a1-11e9-4182-b405-2f7e10bf22ca-openshift-service-ca\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:40:07.965025 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:07.964982    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68ebf8a1-11e9-4182-b405-2f7e10bf22ca" path="/var/lib/kubelet/pods/68ebf8a1-11e9-4182-b405-2f7e10bf22ca/volumes"
Sep 07 10:40:38.264069 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:40:38.264035    2562 scope.go:117] "RemoveContainer" containerID="fb99ca0deb5f9ecee3e881003af3086da5285133bea4abd46ee3722a709803c4"
Sep 07 10:41:06.607472 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:06.607432    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"]
Sep 07 10:41:06.608743 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:06.608685    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl" podUID="8a34ccbc-86fc-4682-ab05-86c197754fbb" containerName="maas-api" containerID="cri-o://1fe024ce6df1f214fb4f639e3eb979fdf7eeb261a30245458d96e73aeeb2824e" gracePeriod=30
Sep 07 10:41:06.670301 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:06.670216    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"]
Sep 07 10:41:06.670809 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:06.670757    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm" podUID="72639a22-38c5-43ef-a2e0-6b6373cd7cf0" containerName="payload-processing" containerID="cri-o://0a20d30020dfa6b2c902d5d4cb9cb7d3f9e9be08e707dda7e8ea21fd18502490" gracePeriod=30
Sep 07 10:41:06.926778 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:06.926734    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:41:06.953225 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:06.953199    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"
Sep 07 10:41:07.043464 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.043423    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz2ff\" (UniqueName: \"kubernetes.io/projected/8a34ccbc-86fc-4682-ab05-86c197754fbb-kube-api-access-kz2ff\") pod \"8a34ccbc-86fc-4682-ab05-86c197754fbb\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") "
Sep 07 10:41:07.043667 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.043568    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a34ccbc-86fc-4682-ab05-86c197754fbb-openshift-service-ca\") pod \"8a34ccbc-86fc-4682-ab05-86c197754fbb\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") "
Sep 07 10:41:07.043667 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.043599    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a34ccbc-86fc-4682-ab05-86c197754fbb-metrics-certs\") pod \"8a34ccbc-86fc-4682-ab05-86c197754fbb\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") "
Sep 07 10:41:07.043667 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.043633    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a34ccbc-86fc-4682-ab05-86c197754fbb-maas-api-tls\") pod \"8a34ccbc-86fc-4682-ab05-86c197754fbb\" (UID: \"8a34ccbc-86fc-4682-ab05-86c197754fbb\") "
Sep 07 10:41:07.044008 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.043973    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a34ccbc-86fc-4682-ab05-86c197754fbb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a34ccbc-86fc-4682-ab05-86c197754fbb" (UID: "8a34ccbc-86fc-4682-ab05-86c197754fbb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:41:07.045816 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.045756    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a34ccbc-86fc-4682-ab05-86c197754fbb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8a34ccbc-86fc-4682-ab05-86c197754fbb" (UID: "8a34ccbc-86fc-4682-ab05-86c197754fbb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:41:07.045943 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.045792    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a34ccbc-86fc-4682-ab05-86c197754fbb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a34ccbc-86fc-4682-ab05-86c197754fbb" (UID: "8a34ccbc-86fc-4682-ab05-86c197754fbb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:41:07.046009 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.045968    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a34ccbc-86fc-4682-ab05-86c197754fbb-kube-api-access-kz2ff" (OuterVolumeSpecName: "kube-api-access-kz2ff") pod "8a34ccbc-86fc-4682-ab05-86c197754fbb" (UID: "8a34ccbc-86fc-4682-ab05-86c197754fbb"). InnerVolumeSpecName "kube-api-access-kz2ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:41:07.145150 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.145064    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72639a22-38c5-43ef-a2e0-6b6373cd7cf0-plugins-config-volume\") pod \"72639a22-38c5-43ef-a2e0-6b6373cd7cf0\" (UID: \"72639a22-38c5-43ef-a2e0-6b6373cd7cf0\") "
Sep 07 10:41:07.145150 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.145128    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqkb8\" (UniqueName: \"kubernetes.io/projected/72639a22-38c5-43ef-a2e0-6b6373cd7cf0-kube-api-access-hqkb8\") pod \"72639a22-38c5-43ef-a2e0-6b6373cd7cf0\" (UID: \"72639a22-38c5-43ef-a2e0-6b6373cd7cf0\") "
Sep 07 10:41:07.145388 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.145366    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a34ccbc-86fc-4682-ab05-86c197754fbb-openshift-service-ca\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:07.145388 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.145376    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a34ccbc-86fc-4682-ab05-86c197754fbb-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:07.145388 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.145385    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a34ccbc-86fc-4682-ab05-86c197754fbb-maas-api-tls\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:07.145566 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.145395    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz2ff\" (UniqueName: \"kubernetes.io/projected/8a34ccbc-86fc-4682-ab05-86c197754fbb-kube-api-access-kz2ff\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:07.145566 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.145514    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72639a22-38c5-43ef-a2e0-6b6373cd7cf0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72639a22-38c5-43ef-a2e0-6b6373cd7cf0" (UID: "72639a22-38c5-43ef-a2e0-6b6373cd7cf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:41:07.147173 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.147150    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72639a22-38c5-43ef-a2e0-6b6373cd7cf0-kube-api-access-hqkb8" (OuterVolumeSpecName: "kube-api-access-hqkb8") pod "72639a22-38c5-43ef-a2e0-6b6373cd7cf0" (UID: "72639a22-38c5-43ef-a2e0-6b6373cd7cf0"). InnerVolumeSpecName "kube-api-access-hqkb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:41:07.246583 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.246544    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72639a22-38c5-43ef-a2e0-6b6373cd7cf0-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:07.246583 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.246575    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqkb8\" (UniqueName: \"kubernetes.io/projected/72639a22-38c5-43ef-a2e0-6b6373cd7cf0-kube-api-access-hqkb8\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:07.682685 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.682644    2562 generic.go:356] "Generic (PLEG): container finished" podID="8a34ccbc-86fc-4682-ab05-86c197754fbb" containerID="1fe024ce6df1f214fb4f639e3eb979fdf7eeb261a30245458d96e73aeeb2824e" exitCode=0
Sep 07 10:41:07.683162 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.682706    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"
Sep 07 10:41:07.683162 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.682722    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl" event={"ID":"8a34ccbc-86fc-4682-ab05-86c197754fbb","Type":"ContainerDied","Data":"1fe024ce6df1f214fb4f639e3eb979fdf7eeb261a30245458d96e73aeeb2824e"}
Sep 07 10:41:07.683162 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.682826    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl" event={"ID":"8a34ccbc-86fc-4682-ab05-86c197754fbb","Type":"ContainerDied","Data":"3c1d519e3411a2a3ca13e9d9cc5383239462108d74a02b637663cf238cdcf4a7"}
Sep 07 10:41:07.683162 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.682860    2562 scope.go:117] "RemoveContainer" containerID="1fe024ce6df1f214fb4f639e3eb979fdf7eeb261a30245458d96e73aeeb2824e"
Sep 07 10:41:07.684305 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.684264    2562 generic.go:356] "Generic (PLEG): container finished" podID="72639a22-38c5-43ef-a2e0-6b6373cd7cf0" containerID="0a20d30020dfa6b2c902d5d4cb9cb7d3f9e9be08e707dda7e8ea21fd18502490" exitCode=0
Sep 07 10:41:07.684376 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.684330    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"
Sep 07 10:41:07.684423 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.684370    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm" event={"ID":"72639a22-38c5-43ef-a2e0-6b6373cd7cf0","Type":"ContainerDied","Data":"0a20d30020dfa6b2c902d5d4cb9cb7d3f9e9be08e707dda7e8ea21fd18502490"}
Sep 07 10:41:07.684471 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.684420    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm" event={"ID":"72639a22-38c5-43ef-a2e0-6b6373cd7cf0","Type":"ContainerDied","Data":"f6f270edff89c4ea0455712a4eea6634ad04317d0189067e22e80aeb845a9fc5"}
Sep 07 10:41:07.692609 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.692587    2562 scope.go:117] "RemoveContainer" containerID="1fe024ce6df1f214fb4f639e3eb979fdf7eeb261a30245458d96e73aeeb2824e"
Sep 07 10:41:07.692997 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:41:07.692967    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe024ce6df1f214fb4f639e3eb979fdf7eeb261a30245458d96e73aeeb2824e\": container with ID starting with 1fe024ce6df1f214fb4f639e3eb979fdf7eeb261a30245458d96e73aeeb2824e not found: ID does not exist" containerID="1fe024ce6df1f214fb4f639e3eb979fdf7eeb261a30245458d96e73aeeb2824e"
Sep 07 10:41:07.693088 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.693005    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe024ce6df1f214fb4f639e3eb979fdf7eeb261a30245458d96e73aeeb2824e"} err="failed to get container status \"1fe024ce6df1f214fb4f639e3eb979fdf7eeb261a30245458d96e73aeeb2824e\": rpc error: code = NotFound desc = could not find container \"1fe024ce6df1f214fb4f639e3eb979fdf7eeb261a30245458d96e73aeeb2824e\": container with ID starting with 1fe024ce6df1f214fb4f639e3eb979fdf7eeb261a30245458d96e73aeeb2824e not found: ID does not exist"
Sep 07 10:41:07.693088 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.693026    2562 scope.go:117] "RemoveContainer" containerID="0a20d30020dfa6b2c902d5d4cb9cb7d3f9e9be08e707dda7e8ea21fd18502490"
Sep 07 10:41:07.704950 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.704923    2562 scope.go:117] "RemoveContainer" containerID="0a20d30020dfa6b2c902d5d4cb9cb7d3f9e9be08e707dda7e8ea21fd18502490"
Sep 07 10:41:07.705244 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:41:07.705225    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a20d30020dfa6b2c902d5d4cb9cb7d3f9e9be08e707dda7e8ea21fd18502490\": container with ID starting with 0a20d30020dfa6b2c902d5d4cb9cb7d3f9e9be08e707dda7e8ea21fd18502490 not found: ID does not exist" containerID="0a20d30020dfa6b2c902d5d4cb9cb7d3f9e9be08e707dda7e8ea21fd18502490"
Sep 07 10:41:07.705305 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.705251    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a20d30020dfa6b2c902d5d4cb9cb7d3f9e9be08e707dda7e8ea21fd18502490"} err="failed to get container status \"0a20d30020dfa6b2c902d5d4cb9cb7d3f9e9be08e707dda7e8ea21fd18502490\": rpc error: code = NotFound desc = could not find container \"0a20d30020dfa6b2c902d5d4cb9cb7d3f9e9be08e707dda7e8ea21fd18502490\": container with ID starting with 0a20d30020dfa6b2c902d5d4cb9cb7d3f9e9be08e707dda7e8ea21fd18502490 not found: ID does not exist"
Sep 07 10:41:07.730087 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.730054    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"]
Sep 07 10:41:07.743079 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.743046    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-125f28-5b4d75bf5f-79xfm"]
Sep 07 10:41:07.771960 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.771924    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"]
Sep 07 10:41:07.780715 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.780679    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-125f28-7c855799bf-xr4xl"]
Sep 07 10:41:07.965169 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.965134    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72639a22-38c5-43ef-a2e0-6b6373cd7cf0" path="/var/lib/kubelet/pods/72639a22-38c5-43ef-a2e0-6b6373cd7cf0/volumes"
Sep 07 10:41:07.965636 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:07.965615    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a34ccbc-86fc-4682-ab05-86c197754fbb" path="/var/lib/kubelet/pods/8a34ccbc-86fc-4682-ab05-86c197754fbb/volumes"
Sep 07 10:41:24.377626 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.377573    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"]
Sep 07 10:41:24.378149 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.377894    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw" podUID="9901089c-ffc7-434b-8ab1-246e6d33c408" containerName="payload-processing" containerID="cri-o://1008fba885f5cf16eeb8d1353df0e5531955b7e8e2027127d6211385b1747ca0" gracePeriod=30
Sep 07 10:41:24.638548 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.638478    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"
Sep 07 10:41:24.692111 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.692076    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkwkb\" (UniqueName: \"kubernetes.io/projected/9901089c-ffc7-434b-8ab1-246e6d33c408-kube-api-access-zkwkb\") pod \"9901089c-ffc7-434b-8ab1-246e6d33c408\" (UID: \"9901089c-ffc7-434b-8ab1-246e6d33c408\") "
Sep 07 10:41:24.692310 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.692128    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9901089c-ffc7-434b-8ab1-246e6d33c408-plugins-config-volume\") pod \"9901089c-ffc7-434b-8ab1-246e6d33c408\" (UID: \"9901089c-ffc7-434b-8ab1-246e6d33c408\") "
Sep 07 10:41:24.692554 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.692528    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9901089c-ffc7-434b-8ab1-246e6d33c408-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9901089c-ffc7-434b-8ab1-246e6d33c408" (UID: "9901089c-ffc7-434b-8ab1-246e6d33c408"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:41:24.694253 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.694221    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9901089c-ffc7-434b-8ab1-246e6d33c408-kube-api-access-zkwkb" (OuterVolumeSpecName: "kube-api-access-zkwkb") pod "9901089c-ffc7-434b-8ab1-246e6d33c408" (UID: "9901089c-ffc7-434b-8ab1-246e6d33c408"). InnerVolumeSpecName "kube-api-access-zkwkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:41:24.787369 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.787336    2562 generic.go:356] "Generic (PLEG): container finished" podID="9901089c-ffc7-434b-8ab1-246e6d33c408" containerID="1008fba885f5cf16eeb8d1353df0e5531955b7e8e2027127d6211385b1747ca0" exitCode=0
Sep 07 10:41:24.787585 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.787382    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"
Sep 07 10:41:24.787585 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.787429    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw" event={"ID":"9901089c-ffc7-434b-8ab1-246e6d33c408","Type":"ContainerDied","Data":"1008fba885f5cf16eeb8d1353df0e5531955b7e8e2027127d6211385b1747ca0"}
Sep 07 10:41:24.787585 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.787488    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw" event={"ID":"9901089c-ffc7-434b-8ab1-246e6d33c408","Type":"ContainerDied","Data":"7aff9b2ab8d60452abaccda0d94481431502b80871052cf1ff84be3de8740a20"}
Sep 07 10:41:24.787585 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.787520    2562 scope.go:117] "RemoveContainer" containerID="1008fba885f5cf16eeb8d1353df0e5531955b7e8e2027127d6211385b1747ca0"
Sep 07 10:41:24.793294 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.793251    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkwkb\" (UniqueName: \"kubernetes.io/projected/9901089c-ffc7-434b-8ab1-246e6d33c408-kube-api-access-zkwkb\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:24.793294 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.793281    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9901089c-ffc7-434b-8ab1-246e6d33c408-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:24.797849 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.797830    2562 scope.go:117] "RemoveContainer" containerID="1008fba885f5cf16eeb8d1353df0e5531955b7e8e2027127d6211385b1747ca0"
Sep 07 10:41:24.798150 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:41:24.798132    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1008fba885f5cf16eeb8d1353df0e5531955b7e8e2027127d6211385b1747ca0\": container with ID starting with 1008fba885f5cf16eeb8d1353df0e5531955b7e8e2027127d6211385b1747ca0 not found: ID does not exist" containerID="1008fba885f5cf16eeb8d1353df0e5531955b7e8e2027127d6211385b1747ca0"
Sep 07 10:41:24.798202 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.798160    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1008fba885f5cf16eeb8d1353df0e5531955b7e8e2027127d6211385b1747ca0"} err="failed to get container status \"1008fba885f5cf16eeb8d1353df0e5531955b7e8e2027127d6211385b1747ca0\": rpc error: code = NotFound desc = could not find container \"1008fba885f5cf16eeb8d1353df0e5531955b7e8e2027127d6211385b1747ca0\": container with ID starting with 1008fba885f5cf16eeb8d1353df0e5531955b7e8e2027127d6211385b1747ca0 not found: ID does not exist"
Sep 07 10:41:24.819394 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.819355    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"]
Sep 07 10:41:24.823510 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:24.823479    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a577c-79644f549-kmtlw"]
Sep 07 10:41:25.965503 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:25.965466    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9901089c-ffc7-434b-8ab1-246e6d33c408" path="/var/lib/kubelet/pods/9901089c-ffc7-434b-8ab1-246e6d33c408/volumes"
Sep 07 10:41:42.631196 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:42.631159    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"]
Sep 07 10:41:42.631708 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:42.631392    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw" podUID="b20aa2d9-d7db-4d07-a263-f393b64403f0" containerName="maas-api" containerID="cri-o://c8c005ad4437591401eaf546a8ff71b2b24caf4fb81d95eda576247127dd3c33" gracePeriod=30
Sep 07 10:41:42.714838 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:42.714795    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"]
Sep 07 10:41:42.715120 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:42.715092    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r" podUID="920d81be-e890-43ad-81b2-dd8460395676" containerName="payload-pre-processing" containerID="cri-o://cabd7d2644cace78f861523074c4fcd6db2ae90e1f21e528cec1d4a7efaa8ed7" gracePeriod=30
Sep 07 10:41:42.892434 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:42.892276    2562 generic.go:356] "Generic (PLEG): container finished" podID="b20aa2d9-d7db-4d07-a263-f393b64403f0" containerID="c8c005ad4437591401eaf546a8ff71b2b24caf4fb81d95eda576247127dd3c33" exitCode=0
Sep 07 10:41:42.892434 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:42.892381    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw" event={"ID":"b20aa2d9-d7db-4d07-a263-f393b64403f0","Type":"ContainerDied","Data":"c8c005ad4437591401eaf546a8ff71b2b24caf4fb81d95eda576247127dd3c33"}
Sep 07 10:41:42.894455 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:42.894424    2562 generic.go:356] "Generic (PLEG): container finished" podID="920d81be-e890-43ad-81b2-dd8460395676" containerID="cabd7d2644cace78f861523074c4fcd6db2ae90e1f21e528cec1d4a7efaa8ed7" exitCode=0
Sep 07 10:41:42.894629 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:42.894558    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r" event={"ID":"920d81be-e890-43ad-81b2-dd8460395676","Type":"ContainerDied","Data":"cabd7d2644cace78f861523074c4fcd6db2ae90e1f21e528cec1d4a7efaa8ed7"}
Sep 07 10:41:43.000102 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.000071    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:41:43.002719 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.002692    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"
Sep 07 10:41:43.054873 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.054840    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920d81be-e890-43ad-81b2-dd8460395676-plugins-config-volume\") pod \"920d81be-e890-43ad-81b2-dd8460395676\" (UID: \"920d81be-e890-43ad-81b2-dd8460395676\") "
Sep 07 10:41:43.055056 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.054918    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20aa2d9-d7db-4d07-a263-f393b64403f0-metrics-certs\") pod \"b20aa2d9-d7db-4d07-a263-f393b64403f0\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") "
Sep 07 10:41:43.055056 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.054937    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngj9n\" (UniqueName: \"kubernetes.io/projected/920d81be-e890-43ad-81b2-dd8460395676-kube-api-access-ngj9n\") pod \"920d81be-e890-43ad-81b2-dd8460395676\" (UID: \"920d81be-e890-43ad-81b2-dd8460395676\") "
Sep 07 10:41:43.055056 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.054979    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b20aa2d9-d7db-4d07-a263-f393b64403f0-maas-api-tls\") pod \"b20aa2d9-d7db-4d07-a263-f393b64403f0\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") "
Sep 07 10:41:43.055056 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.055013    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20aa2d9-d7db-4d07-a263-f393b64403f0-openshift-service-ca\") pod \"b20aa2d9-d7db-4d07-a263-f393b64403f0\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") "
Sep 07 10:41:43.055265 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.055082    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8ck9\" (UniqueName: \"kubernetes.io/projected/b20aa2d9-d7db-4d07-a263-f393b64403f0-kube-api-access-s8ck9\") pod \"b20aa2d9-d7db-4d07-a263-f393b64403f0\" (UID: \"b20aa2d9-d7db-4d07-a263-f393b64403f0\") "
Sep 07 10:41:43.055265 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.055207    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920d81be-e890-43ad-81b2-dd8460395676-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "920d81be-e890-43ad-81b2-dd8460395676" (UID: "920d81be-e890-43ad-81b2-dd8460395676"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:41:43.055482 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.055459    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b20aa2d9-d7db-4d07-a263-f393b64403f0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b20aa2d9-d7db-4d07-a263-f393b64403f0" (UID: "b20aa2d9-d7db-4d07-a263-f393b64403f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:41:43.055574 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.055535    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920d81be-e890-43ad-81b2-dd8460395676-plugins-config-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:43.057451 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.057423    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b20aa2d9-d7db-4d07-a263-f393b64403f0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b20aa2d9-d7db-4d07-a263-f393b64403f0" (UID: "b20aa2d9-d7db-4d07-a263-f393b64403f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:41:43.057577 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.057525    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b20aa2d9-d7db-4d07-a263-f393b64403f0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b20aa2d9-d7db-4d07-a263-f393b64403f0" (UID: "b20aa2d9-d7db-4d07-a263-f393b64403f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:41:43.057751 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.057705    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920d81be-e890-43ad-81b2-dd8460395676-kube-api-access-ngj9n" (OuterVolumeSpecName: "kube-api-access-ngj9n") pod "920d81be-e890-43ad-81b2-dd8460395676" (UID: "920d81be-e890-43ad-81b2-dd8460395676"). InnerVolumeSpecName "kube-api-access-ngj9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:41:43.058049 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.058027    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20aa2d9-d7db-4d07-a263-f393b64403f0-kube-api-access-s8ck9" (OuterVolumeSpecName: "kube-api-access-s8ck9") pod "b20aa2d9-d7db-4d07-a263-f393b64403f0" (UID: "b20aa2d9-d7db-4d07-a263-f393b64403f0"). InnerVolumeSpecName "kube-api-access-s8ck9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:41:43.156674 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.156569    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20aa2d9-d7db-4d07-a263-f393b64403f0-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:43.156674 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.156608    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngj9n\" (UniqueName: \"kubernetes.io/projected/920d81be-e890-43ad-81b2-dd8460395676-kube-api-access-ngj9n\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:43.156674 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.156618    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b20aa2d9-d7db-4d07-a263-f393b64403f0-maas-api-tls\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:43.156674 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.156628    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20aa2d9-d7db-4d07-a263-f393b64403f0-openshift-service-ca\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:43.156674 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.156637    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8ck9\" (UniqueName: \"kubernetes.io/projected/b20aa2d9-d7db-4d07-a263-f393b64403f0-kube-api-access-s8ck9\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:43.901420 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.901386    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"
Sep 07 10:41:43.901420 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.901402    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw" event={"ID":"b20aa2d9-d7db-4d07-a263-f393b64403f0","Type":"ContainerDied","Data":"d54288d73e3ad535986b3d0f2e6d0a2bf4b64ee1bef8f6b671c561e31fb7ed2a"}
Sep 07 10:41:43.902007 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.901498    2562 scope.go:117] "RemoveContainer" containerID="c8c005ad4437591401eaf546a8ff71b2b24caf4fb81d95eda576247127dd3c33"
Sep 07 10:41:43.903545 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.903510    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"
Sep 07 10:41:43.903642 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.903538    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r" event={"ID":"920d81be-e890-43ad-81b2-dd8460395676","Type":"ContainerDied","Data":"a9bfcf05c8c7c5a03cb48e1b67cece64ae7b9a396fbd05bd360e4b9ab876d521"}
Sep 07 10:41:43.912797 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.912778    2562 scope.go:117] "RemoveContainer" containerID="cabd7d2644cace78f861523074c4fcd6db2ae90e1f21e528cec1d4a7efaa8ed7"
Sep 07 10:41:43.925423 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.925401    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"]
Sep 07 10:41:43.929612 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.929588    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-93e41e-5f74687bcc-924fw"]
Sep 07 10:41:43.940864 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.940838    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"]
Sep 07 10:41:43.945348 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.945316    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-93e41e-d65cb5dc8-nqm9r"]
Sep 07 10:41:43.964449 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.964418    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="920d81be-e890-43ad-81b2-dd8460395676" path="/var/lib/kubelet/pods/920d81be-e890-43ad-81b2-dd8460395676/volumes"
Sep 07 10:41:43.964834 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:43.964819    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b20aa2d9-d7db-4d07-a263-f393b64403f0" path="/var/lib/kubelet/pods/b20aa2d9-d7db-4d07-a263-f393b64403f0/volumes"
Sep 07 10:41:52.558802 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.558745    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6899cfd859-v7wgq"]
Sep 07 10:41:52.559332 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.559112    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6899cfd859-v7wgq" podUID="278ec7ee-2beb-4a50-8961-b10b5c150d5b" containerName="manager" containerID="cri-o://eb2fa5d8121d8b01e30220e75542f9e4868ead1abe8076f124b443d30d812455" gracePeriod=30
Sep 07 10:41:52.811243 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.811170    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899cfd859-v7wgq"
Sep 07 10:41:52.944076 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.944042    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278ec7ee-2beb-4a50-8961-b10b5c150d5b-metrics-certs\") pod \"278ec7ee-2beb-4a50-8961-b10b5c150d5b\" (UID: \"278ec7ee-2beb-4a50-8961-b10b5c150d5b\") "
Sep 07 10:41:52.944288 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.944111    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4kjx\" (UniqueName: \"kubernetes.io/projected/278ec7ee-2beb-4a50-8961-b10b5c150d5b-kube-api-access-v4kjx\") pod \"278ec7ee-2beb-4a50-8961-b10b5c150d5b\" (UID: \"278ec7ee-2beb-4a50-8961-b10b5c150d5b\") "
Sep 07 10:41:52.944288 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.944159    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/278ec7ee-2beb-4a50-8961-b10b5c150d5b-webhook-cert\") pod \"278ec7ee-2beb-4a50-8961-b10b5c150d5b\" (UID: \"278ec7ee-2beb-4a50-8961-b10b5c150d5b\") "
Sep 07 10:41:52.946221 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.946187    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/278ec7ee-2beb-4a50-8961-b10b5c150d5b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "278ec7ee-2beb-4a50-8961-b10b5c150d5b" (UID: "278ec7ee-2beb-4a50-8961-b10b5c150d5b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:41:52.946221 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.946211    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/278ec7ee-2beb-4a50-8961-b10b5c150d5b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "278ec7ee-2beb-4a50-8961-b10b5c150d5b" (UID: "278ec7ee-2beb-4a50-8961-b10b5c150d5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:41:52.946371 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.946198    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278ec7ee-2beb-4a50-8961-b10b5c150d5b-kube-api-access-v4kjx" (OuterVolumeSpecName: "kube-api-access-v4kjx") pod "278ec7ee-2beb-4a50-8961-b10b5c150d5b" (UID: "278ec7ee-2beb-4a50-8961-b10b5c150d5b"). InnerVolumeSpecName "kube-api-access-v4kjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:41:52.957515 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.957481    2562 generic.go:356] "Generic (PLEG): container finished" podID="278ec7ee-2beb-4a50-8961-b10b5c150d5b" containerID="eb2fa5d8121d8b01e30220e75542f9e4868ead1abe8076f124b443d30d812455" exitCode=0
Sep 07 10:41:52.957649 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.957527    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899cfd859-v7wgq"
Sep 07 10:41:52.957649 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.957561    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899cfd859-v7wgq" event={"ID":"278ec7ee-2beb-4a50-8961-b10b5c150d5b","Type":"ContainerDied","Data":"eb2fa5d8121d8b01e30220e75542f9e4868ead1abe8076f124b443d30d812455"}
Sep 07 10:41:52.957649 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.957611    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899cfd859-v7wgq" event={"ID":"278ec7ee-2beb-4a50-8961-b10b5c150d5b","Type":"ContainerDied","Data":"0c58363a3b6c9e62fb6c8cf984d5856509a2ebf21aa50214f4d12126deba0c5c"}
Sep 07 10:41:52.957649 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.957633    2562 scope.go:117] "RemoveContainer" containerID="eb2fa5d8121d8b01e30220e75542f9e4868ead1abe8076f124b443d30d812455"
Sep 07 10:41:52.967384 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.967363    2562 scope.go:117] "RemoveContainer" containerID="eb2fa5d8121d8b01e30220e75542f9e4868ead1abe8076f124b443d30d812455"
Sep 07 10:41:52.967687 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:41:52.967663    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb2fa5d8121d8b01e30220e75542f9e4868ead1abe8076f124b443d30d812455\": container with ID starting with eb2fa5d8121d8b01e30220e75542f9e4868ead1abe8076f124b443d30d812455 not found: ID does not exist" containerID="eb2fa5d8121d8b01e30220e75542f9e4868ead1abe8076f124b443d30d812455"
Sep 07 10:41:52.967785 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.967693    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb2fa5d8121d8b01e30220e75542f9e4868ead1abe8076f124b443d30d812455"} err="failed to get container status \"eb2fa5d8121d8b01e30220e75542f9e4868ead1abe8076f124b443d30d812455\": rpc error: code = NotFound desc = could not find container \"eb2fa5d8121d8b01e30220e75542f9e4868ead1abe8076f124b443d30d812455\": container with ID starting with eb2fa5d8121d8b01e30220e75542f9e4868ead1abe8076f124b443d30d812455 not found: ID does not exist"
Sep 07 10:41:52.982088 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.982062    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6899cfd859-v7wgq"]
Sep 07 10:41:52.987544 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:52.987524    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6899cfd859-v7wgq"]
Sep 07 10:41:53.045592 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:53.045548    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/278ec7ee-2beb-4a50-8961-b10b5c150d5b-webhook-cert\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:53.045592 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:53.045589    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278ec7ee-2beb-4a50-8961-b10b5c150d5b-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:53.045955 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:53.045603    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4kjx\" (UniqueName: \"kubernetes.io/projected/278ec7ee-2beb-4a50-8961-b10b5c150d5b-kube-api-access-v4kjx\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:41:53.965107 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:41:53.965074    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="278ec7ee-2beb-4a50-8961-b10b5c150d5b" path="/var/lib/kubelet/pods/278ec7ee-2beb-4a50-8961-b10b5c150d5b/volumes"
Sep 07 10:42:53.773408 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773368    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6899cfd859-2p682"]
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773734    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e60a8a9-a60b-42e8-a6cc-528618314c5c" containerName="payload-pre-processing"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773744    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e60a8a9-a60b-42e8-a6cc-528618314c5c" containerName="payload-pre-processing"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773754    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8" containerName="maas-api"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773778    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8" containerName="maas-api"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773792    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b20aa2d9-d7db-4d07-a263-f393b64403f0" containerName="maas-api"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773797    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b20aa2d9-d7db-4d07-a263-f393b64403f0" containerName="maas-api"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773807    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a34ccbc-86fc-4682-ab05-86c197754fbb" containerName="maas-api"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773812    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a34ccbc-86fc-4682-ab05-86c197754fbb" containerName="maas-api"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773820    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72639a22-38c5-43ef-a2e0-6b6373cd7cf0" containerName="payload-processing"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773825    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="72639a22-38c5-43ef-a2e0-6b6373cd7cf0" containerName="payload-processing"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773832    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9901089c-ffc7-434b-8ab1-246e6d33c408" containerName="payload-processing"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773840    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9901089c-ffc7-434b-8ab1-246e6d33c408" containerName="payload-processing"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773853    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68ebf8a1-11e9-4182-b405-2f7e10bf22ca" containerName="maas-api"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773862    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="68ebf8a1-11e9-4182-b405-2f7e10bf22ca" containerName="maas-api"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773877    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="278ec7ee-2beb-4a50-8961-b10b5c150d5b" containerName="manager"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773882    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="278ec7ee-2beb-4a50-8961-b10b5c150d5b" containerName="manager"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773890    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920d81be-e890-43ad-81b2-dd8460395676" containerName="payload-pre-processing"
Sep 07 10:42:53.773931 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773895    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="920d81be-e890-43ad-81b2-dd8460395676" containerName="payload-pre-processing"
Sep 07 10:42:53.774521 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773952    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a34ccbc-86fc-4682-ab05-86c197754fbb" containerName="maas-api"
Sep 07 10:42:53.774521 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773960    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec9a0c07-7e69-4a54-ab9b-1bc2765f19b8" containerName="maas-api"
Sep 07 10:42:53.774521 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773966    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="72639a22-38c5-43ef-a2e0-6b6373cd7cf0" containerName="payload-processing"
Sep 07 10:42:53.774521 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773972    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e60a8a9-a60b-42e8-a6cc-528618314c5c" containerName="payload-pre-processing"
Sep 07 10:42:53.774521 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773978    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="68ebf8a1-11e9-4182-b405-2f7e10bf22ca" containerName="maas-api"
Sep 07 10:42:53.774521 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773983    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="920d81be-e890-43ad-81b2-dd8460395676" containerName="payload-pre-processing"
Sep 07 10:42:53.774521 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773989    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b20aa2d9-d7db-4d07-a263-f393b64403f0" containerName="maas-api"
Sep 07 10:42:53.774521 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.773996    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="278ec7ee-2beb-4a50-8961-b10b5c150d5b" containerName="manager"
Sep 07 10:42:53.774521 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.774003    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9901089c-ffc7-434b-8ab1-246e6d33c408" containerName="payload-processing"
Sep 07 10:42:53.777024 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.776999    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899cfd859-2p682"
Sep 07 10:42:53.780790 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.780746    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lh264\""
Sep 07 10:42:53.780790 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.780787    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 07 10:42:53.780989 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.780821    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 07 10:42:53.780989 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.780747    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 07 10:42:53.791112 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.791063    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6899cfd859-2p682"]
Sep 07 10:42:53.876145 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.876095    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2aee83f-9e2a-41b5-b341-d9de5532197e-webhook-cert\") pod \"maas-controller-6899cfd859-2p682\" (UID: \"a2aee83f-9e2a-41b5-b341-d9de5532197e\") " pod="opendatahub/maas-controller-6899cfd859-2p682"
Sep 07 10:42:53.876354 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.876160    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvxdt\" (UniqueName: \"kubernetes.io/projected/a2aee83f-9e2a-41b5-b341-d9de5532197e-kube-api-access-zvxdt\") pod \"maas-controller-6899cfd859-2p682\" (UID: \"a2aee83f-9e2a-41b5-b341-d9de5532197e\") " pod="opendatahub/maas-controller-6899cfd859-2p682"
Sep 07 10:42:53.876354 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.876210    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2aee83f-9e2a-41b5-b341-d9de5532197e-metrics-certs\") pod \"maas-controller-6899cfd859-2p682\" (UID: \"a2aee83f-9e2a-41b5-b341-d9de5532197e\") " pod="opendatahub/maas-controller-6899cfd859-2p682"
Sep 07 10:42:53.977614 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.977574    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvxdt\" (UniqueName: \"kubernetes.io/projected/a2aee83f-9e2a-41b5-b341-d9de5532197e-kube-api-access-zvxdt\") pod \"maas-controller-6899cfd859-2p682\" (UID: \"a2aee83f-9e2a-41b5-b341-d9de5532197e\") " pod="opendatahub/maas-controller-6899cfd859-2p682"
Sep 07 10:42:53.977614 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.977614    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2aee83f-9e2a-41b5-b341-d9de5532197e-metrics-certs\") pod \"maas-controller-6899cfd859-2p682\" (UID: \"a2aee83f-9e2a-41b5-b341-d9de5532197e\") " pod="opendatahub/maas-controller-6899cfd859-2p682"
Sep 07 10:42:53.977944 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.977685    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2aee83f-9e2a-41b5-b341-d9de5532197e-webhook-cert\") pod \"maas-controller-6899cfd859-2p682\" (UID: \"a2aee83f-9e2a-41b5-b341-d9de5532197e\") " pod="opendatahub/maas-controller-6899cfd859-2p682"
Sep 07 10:42:53.980243 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.980213    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2aee83f-9e2a-41b5-b341-d9de5532197e-webhook-cert\") pod \"maas-controller-6899cfd859-2p682\" (UID: \"a2aee83f-9e2a-41b5-b341-d9de5532197e\") " pod="opendatahub/maas-controller-6899cfd859-2p682"
Sep 07 10:42:53.980353 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.980332    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2aee83f-9e2a-41b5-b341-d9de5532197e-metrics-certs\") pod \"maas-controller-6899cfd859-2p682\" (UID: \"a2aee83f-9e2a-41b5-b341-d9de5532197e\") " pod="opendatahub/maas-controller-6899cfd859-2p682"
Sep 07 10:42:53.988517 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:53.988489    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvxdt\" (UniqueName: \"kubernetes.io/projected/a2aee83f-9e2a-41b5-b341-d9de5532197e-kube-api-access-zvxdt\") pod \"maas-controller-6899cfd859-2p682\" (UID: \"a2aee83f-9e2a-41b5-b341-d9de5532197e\") " pod="opendatahub/maas-controller-6899cfd859-2p682"
Sep 07 10:42:54.088065 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:54.087953    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899cfd859-2p682"
Sep 07 10:42:54.365100 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:54.365070    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6899cfd859-2p682"]
Sep 07 10:42:54.366464 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:42:54.366428    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2aee83f_9e2a_41b5_b341_d9de5532197e.slice/crio-b6119c81599e1f8a6b13088d6803217b6d1e5dd6e3e238d2bc038b618fc5dba3 WatchSource:0}: Error finding container b6119c81599e1f8a6b13088d6803217b6d1e5dd6e3e238d2bc038b618fc5dba3: Status 404 returned error can't find the container with id b6119c81599e1f8a6b13088d6803217b6d1e5dd6e3e238d2bc038b618fc5dba3
Sep 07 10:42:55.283830 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:55.283734    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899cfd859-2p682" event={"ID":"a2aee83f-9e2a-41b5-b341-d9de5532197e","Type":"ContainerStarted","Data":"f02ffac8345fb65b0c863040815415293e976e0b6c0dac792c8407da86063ccb"}
Sep 07 10:42:55.283830 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:55.283829    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6899cfd859-2p682"
Sep 07 10:42:55.284285 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:55.283847    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899cfd859-2p682" event={"ID":"a2aee83f-9e2a-41b5-b341-d9de5532197e","Type":"ContainerStarted","Data":"b6119c81599e1f8a6b13088d6803217b6d1e5dd6e3e238d2bc038b618fc5dba3"}
Sep 07 10:42:55.305945 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:42:55.305880    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6899cfd859-2p682" podStartSLOduration=1.721866408 podStartE2EDuration="2.305861456s" totalImagesPullingTime="583.995048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:42:55.303429725 +0000 UTC m=+1877.801119655" watchObservedRunningTime="2026-09-07 10:42:55.305861456 +0000 UTC m=+1877.803551366"
Sep 07 10:43:06.291840 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:06.291804    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6899cfd859-2p682"
Sep 07 10:43:15.636187 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:15.636146    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6899cfd859-2p682"]
Sep 07 10:43:15.636685 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:15.636399    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6899cfd859-2p682" podUID="a2aee83f-9e2a-41b5-b341-d9de5532197e" containerName="manager" containerID="cri-o://f02ffac8345fb65b0c863040815415293e976e0b6c0dac792c8407da86063ccb" gracePeriod=30
Sep 07 10:43:16.087559 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.087529    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899cfd859-2p682"
Sep 07 10:43:16.179315 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.179275    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2aee83f-9e2a-41b5-b341-d9de5532197e-metrics-certs\") pod \"a2aee83f-9e2a-41b5-b341-d9de5532197e\" (UID: \"a2aee83f-9e2a-41b5-b341-d9de5532197e\") "
Sep 07 10:43:16.179488 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.179341    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvxdt\" (UniqueName: \"kubernetes.io/projected/a2aee83f-9e2a-41b5-b341-d9de5532197e-kube-api-access-zvxdt\") pod \"a2aee83f-9e2a-41b5-b341-d9de5532197e\" (UID: \"a2aee83f-9e2a-41b5-b341-d9de5532197e\") "
Sep 07 10:43:16.179488 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.179365    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2aee83f-9e2a-41b5-b341-d9de5532197e-webhook-cert\") pod \"a2aee83f-9e2a-41b5-b341-d9de5532197e\" (UID: \"a2aee83f-9e2a-41b5-b341-d9de5532197e\") "
Sep 07 10:43:16.181497 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.181455    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2aee83f-9e2a-41b5-b341-d9de5532197e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a2aee83f-9e2a-41b5-b341-d9de5532197e" (UID: "a2aee83f-9e2a-41b5-b341-d9de5532197e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:43:16.181632 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.181491    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2aee83f-9e2a-41b5-b341-d9de5532197e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a2aee83f-9e2a-41b5-b341-d9de5532197e" (UID: "a2aee83f-9e2a-41b5-b341-d9de5532197e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:43:16.181632 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.181545    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2aee83f-9e2a-41b5-b341-d9de5532197e-kube-api-access-zvxdt" (OuterVolumeSpecName: "kube-api-access-zvxdt") pod "a2aee83f-9e2a-41b5-b341-d9de5532197e" (UID: "a2aee83f-9e2a-41b5-b341-d9de5532197e"). InnerVolumeSpecName "kube-api-access-zvxdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:43:16.280980 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.280891    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvxdt\" (UniqueName: \"kubernetes.io/projected/a2aee83f-9e2a-41b5-b341-d9de5532197e-kube-api-access-zvxdt\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:43:16.280980 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.280928    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2aee83f-9e2a-41b5-b341-d9de5532197e-webhook-cert\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:43:16.280980 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.280940    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2aee83f-9e2a-41b5-b341-d9de5532197e-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:43:16.394355 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.394319    2562 generic.go:356] "Generic (PLEG): container finished" podID="a2aee83f-9e2a-41b5-b341-d9de5532197e" containerID="f02ffac8345fb65b0c863040815415293e976e0b6c0dac792c8407da86063ccb" exitCode=0
Sep 07 10:43:16.394546 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.394368    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899cfd859-2p682"
Sep 07 10:43:16.394546 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.394408    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899cfd859-2p682" event={"ID":"a2aee83f-9e2a-41b5-b341-d9de5532197e","Type":"ContainerDied","Data":"f02ffac8345fb65b0c863040815415293e976e0b6c0dac792c8407da86063ccb"}
Sep 07 10:43:16.394546 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.394455    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899cfd859-2p682" event={"ID":"a2aee83f-9e2a-41b5-b341-d9de5532197e","Type":"ContainerDied","Data":"b6119c81599e1f8a6b13088d6803217b6d1e5dd6e3e238d2bc038b618fc5dba3"}
Sep 07 10:43:16.394546 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.394477    2562 scope.go:117] "RemoveContainer" containerID="f02ffac8345fb65b0c863040815415293e976e0b6c0dac792c8407da86063ccb"
Sep 07 10:43:16.403664 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.403639    2562 scope.go:117] "RemoveContainer" containerID="f02ffac8345fb65b0c863040815415293e976e0b6c0dac792c8407da86063ccb"
Sep 07 10:43:16.404008 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:43:16.403991    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f02ffac8345fb65b0c863040815415293e976e0b6c0dac792c8407da86063ccb\": container with ID starting with f02ffac8345fb65b0c863040815415293e976e0b6c0dac792c8407da86063ccb not found: ID does not exist" containerID="f02ffac8345fb65b0c863040815415293e976e0b6c0dac792c8407da86063ccb"
Sep 07 10:43:16.404086 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.404019    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f02ffac8345fb65b0c863040815415293e976e0b6c0dac792c8407da86063ccb"} err="failed to get container status \"f02ffac8345fb65b0c863040815415293e976e0b6c0dac792c8407da86063ccb\": rpc error: code = NotFound desc = could not find container \"f02ffac8345fb65b0c863040815415293e976e0b6c0dac792c8407da86063ccb\": container with ID starting with f02ffac8345fb65b0c863040815415293e976e0b6c0dac792c8407da86063ccb not found: ID does not exist"
Sep 07 10:43:16.417916 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.417889    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6899cfd859-2p682"]
Sep 07 10:43:16.422231 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:16.422209    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6899cfd859-2p682"]
Sep 07 10:43:17.323639 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.323599    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6899cfd859-44brf"]
Sep 07 10:43:17.324107 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.324050    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2aee83f-9e2a-41b5-b341-d9de5532197e" containerName="manager"
Sep 07 10:43:17.324107 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.324066    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2aee83f-9e2a-41b5-b341-d9de5532197e" containerName="manager"
Sep 07 10:43:17.324179 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.324148    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2aee83f-9e2a-41b5-b341-d9de5532197e" containerName="manager"
Sep 07 10:43:17.328510 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.328490    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899cfd859-44brf"
Sep 07 10:43:17.332360 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.332337    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 07 10:43:17.332476 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.332379    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lh264\""
Sep 07 10:43:17.332476 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.332386    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 07 10:43:17.332602 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.332391    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 07 10:43:17.346166 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.346132    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6899cfd859-44brf"]
Sep 07 10:43:17.388772 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.388728    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87c13c2c-2f9c-4794-93fa-3b547b623a62-metrics-certs\") pod \"maas-controller-6899cfd859-44brf\" (UID: \"87c13c2c-2f9c-4794-93fa-3b547b623a62\") " pod="opendatahub/maas-controller-6899cfd859-44brf"
Sep 07 10:43:17.388956 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.388789    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87c13c2c-2f9c-4794-93fa-3b547b623a62-webhook-cert\") pod \"maas-controller-6899cfd859-44brf\" (UID: \"87c13c2c-2f9c-4794-93fa-3b547b623a62\") " pod="opendatahub/maas-controller-6899cfd859-44brf"
Sep 07 10:43:17.388956 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.388894    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgglk\" (UniqueName: \"kubernetes.io/projected/87c13c2c-2f9c-4794-93fa-3b547b623a62-kube-api-access-bgglk\") pod \"maas-controller-6899cfd859-44brf\" (UID: \"87c13c2c-2f9c-4794-93fa-3b547b623a62\") " pod="opendatahub/maas-controller-6899cfd859-44brf"
Sep 07 10:43:17.490063 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.490023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgglk\" (UniqueName: \"kubernetes.io/projected/87c13c2c-2f9c-4794-93fa-3b547b623a62-kube-api-access-bgglk\") pod \"maas-controller-6899cfd859-44brf\" (UID: \"87c13c2c-2f9c-4794-93fa-3b547b623a62\") " pod="opendatahub/maas-controller-6899cfd859-44brf"
Sep 07 10:43:17.490244 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.490079    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87c13c2c-2f9c-4794-93fa-3b547b623a62-metrics-certs\") pod \"maas-controller-6899cfd859-44brf\" (UID: \"87c13c2c-2f9c-4794-93fa-3b547b623a62\") " pod="opendatahub/maas-controller-6899cfd859-44brf"
Sep 07 10:43:17.490244 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.490122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87c13c2c-2f9c-4794-93fa-3b547b623a62-webhook-cert\") pod \"maas-controller-6899cfd859-44brf\" (UID: \"87c13c2c-2f9c-4794-93fa-3b547b623a62\") " pod="opendatahub/maas-controller-6899cfd859-44brf"
Sep 07 10:43:17.492544 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.492514    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87c13c2c-2f9c-4794-93fa-3b547b623a62-metrics-certs\") pod \"maas-controller-6899cfd859-44brf\" (UID: \"87c13c2c-2f9c-4794-93fa-3b547b623a62\") " pod="opendatahub/maas-controller-6899cfd859-44brf"
Sep 07 10:43:17.492662 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.492559    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87c13c2c-2f9c-4794-93fa-3b547b623a62-webhook-cert\") pod \"maas-controller-6899cfd859-44brf\" (UID: \"87c13c2c-2f9c-4794-93fa-3b547b623a62\") " pod="opendatahub/maas-controller-6899cfd859-44brf"
Sep 07 10:43:17.499119 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.499098    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgglk\" (UniqueName: \"kubernetes.io/projected/87c13c2c-2f9c-4794-93fa-3b547b623a62-kube-api-access-bgglk\") pod \"maas-controller-6899cfd859-44brf\" (UID: \"87c13c2c-2f9c-4794-93fa-3b547b623a62\") " pod="opendatahub/maas-controller-6899cfd859-44brf"
Sep 07 10:43:17.639449 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.639350    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899cfd859-44brf"
Sep 07 10:43:17.917555 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.917523    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6899cfd859-44brf"]
Sep 07 10:43:17.919538 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:43:17.919508    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c13c2c_2f9c_4794_93fa_3b547b623a62.slice/crio-08313e17bcec71eba587a93359c48774788968c0221acaa2807bb30c07a99e35 WatchSource:0}: Error finding container 08313e17bcec71eba587a93359c48774788968c0221acaa2807bb30c07a99e35: Status 404 returned error can't find the container with id 08313e17bcec71eba587a93359c48774788968c0221acaa2807bb30c07a99e35
Sep 07 10:43:17.965307 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:17.965274    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2aee83f-9e2a-41b5-b341-d9de5532197e" path="/var/lib/kubelet/pods/a2aee83f-9e2a-41b5-b341-d9de5532197e/volumes"
Sep 07 10:43:18.408775 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:18.408721    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899cfd859-44brf" event={"ID":"87c13c2c-2f9c-4794-93fa-3b547b623a62","Type":"ContainerStarted","Data":"5c73405f79338c13323312132786b58564112b13aef2155b06f6cecf82928a0a"}
Sep 07 10:43:18.409197 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:18.408800    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899cfd859-44brf" event={"ID":"87c13c2c-2f9c-4794-93fa-3b547b623a62","Type":"ContainerStarted","Data":"08313e17bcec71eba587a93359c48774788968c0221acaa2807bb30c07a99e35"}
Sep 07 10:43:18.409197 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:18.408830    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6899cfd859-44brf"
Sep 07 10:43:18.429148 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:18.429097    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6899cfd859-44brf" podStartSLOduration=1.079900256 podStartE2EDuration="1.429080714s" totalImagesPullingTime="349.180458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:43:18.426301978 +0000 UTC m=+1900.923991909" watchObservedRunningTime="2026-09-07 10:43:18.429080714 +0000 UTC m=+1900.926770623"
Sep 07 10:43:29.417221 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:43:29.417181    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6899cfd859-44brf"
Sep 07 10:44:26.516858 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:26.516801    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"]
Sep 07 10:44:26.517419 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:26.517126    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw" podUID="37bc40d6-5bc7-4b15-8341-6eb834084e4d" containerName="manager" containerID="cri-o://9897999afb695526be589d5da21b9c93e4c8e40e9e550b59c0ebe417d0962726" gracePeriod=10
Sep 07 10:44:26.778499 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:26.778443    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"
Sep 07 10:44:26.780371 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:26.780331    2562 generic.go:356] "Generic (PLEG): container finished" podID="37bc40d6-5bc7-4b15-8341-6eb834084e4d" containerID="9897999afb695526be589d5da21b9c93e4c8e40e9e550b59c0ebe417d0962726" exitCode=0
Sep 07 10:44:26.780371 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:26.780366    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"
Sep 07 10:44:26.780531 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:26.780412    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw" event={"ID":"37bc40d6-5bc7-4b15-8341-6eb834084e4d","Type":"ContainerDied","Data":"9897999afb695526be589d5da21b9c93e4c8e40e9e550b59c0ebe417d0962726"}
Sep 07 10:44:26.780531 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:26.780472    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw" event={"ID":"37bc40d6-5bc7-4b15-8341-6eb834084e4d","Type":"ContainerDied","Data":"c6add9ba6b6c2f4c8781e817ad63a558640963627b1ab9bdcdaedaa60ff0995a"}
Sep 07 10:44:26.780531 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:26.780498    2562 scope.go:117] "RemoveContainer" containerID="9897999afb695526be589d5da21b9c93e4c8e40e9e550b59c0ebe417d0962726"
Sep 07 10:44:26.789945 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:26.789928    2562 scope.go:117] "RemoveContainer" containerID="9897999afb695526be589d5da21b9c93e4c8e40e9e550b59c0ebe417d0962726"
Sep 07 10:44:26.790218 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:44:26.790198    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9897999afb695526be589d5da21b9c93e4c8e40e9e550b59c0ebe417d0962726\": container with ID starting with 9897999afb695526be589d5da21b9c93e4c8e40e9e550b59c0ebe417d0962726 not found: ID does not exist" containerID="9897999afb695526be589d5da21b9c93e4c8e40e9e550b59c0ebe417d0962726"
Sep 07 10:44:26.790265 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:26.790226    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9897999afb695526be589d5da21b9c93e4c8e40e9e550b59c0ebe417d0962726"} err="failed to get container status \"9897999afb695526be589d5da21b9c93e4c8e40e9e550b59c0ebe417d0962726\": rpc error: code = NotFound desc = could not find container \"9897999afb695526be589d5da21b9c93e4c8e40e9e550b59c0ebe417d0962726\": container with ID starting with 9897999afb695526be589d5da21b9c93e4c8e40e9e550b59c0ebe417d0962726 not found: ID does not exist"
Sep 07 10:44:26.859019 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:26.858984    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37bc40d6-5bc7-4b15-8341-6eb834084e4d-extensions-socket-volume\") pod \"37bc40d6-5bc7-4b15-8341-6eb834084e4d\" (UID: \"37bc40d6-5bc7-4b15-8341-6eb834084e4d\") "
Sep 07 10:44:26.859211 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:26.859023    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnmwh\" (UniqueName: \"kubernetes.io/projected/37bc40d6-5bc7-4b15-8341-6eb834084e4d-kube-api-access-xnmwh\") pod \"37bc40d6-5bc7-4b15-8341-6eb834084e4d\" (UID: \"37bc40d6-5bc7-4b15-8341-6eb834084e4d\") "
Sep 07 10:44:26.859428 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:26.859404    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bc40d6-5bc7-4b15-8341-6eb834084e4d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "37bc40d6-5bc7-4b15-8341-6eb834084e4d" (UID: "37bc40d6-5bc7-4b15-8341-6eb834084e4d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:44:26.861127 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:26.861096    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bc40d6-5bc7-4b15-8341-6eb834084e4d-kube-api-access-xnmwh" (OuterVolumeSpecName: "kube-api-access-xnmwh") pod "37bc40d6-5bc7-4b15-8341-6eb834084e4d" (UID: "37bc40d6-5bc7-4b15-8341-6eb834084e4d"). InnerVolumeSpecName "kube-api-access-xnmwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:44:26.960113 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:26.960078    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37bc40d6-5bc7-4b15-8341-6eb834084e4d-extensions-socket-volume\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:44:26.960113 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:26.960110    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnmwh\" (UniqueName: \"kubernetes.io/projected/37bc40d6-5bc7-4b15-8341-6eb834084e4d-kube-api-access-xnmwh\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:44:27.103587 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:27.103555    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"]
Sep 07 10:44:27.106913 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:27.106884    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-gnptw"]
Sep 07 10:44:27.964103 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:44:27.964067    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37bc40d6-5bc7-4b15-8341-6eb834084e4d" path="/var/lib/kubelet/pods/37bc40d6-5bc7-4b15-8341-6eb834084e4d/volumes"
Sep 07 10:45:27.185846 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:27.185806    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-fcxtf"]
Sep 07 10:45:27.186272 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:27.186175    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37bc40d6-5bc7-4b15-8341-6eb834084e4d" containerName="manager"
Sep 07 10:45:27.186272 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:27.186187    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="37bc40d6-5bc7-4b15-8341-6eb834084e4d" containerName="manager"
Sep 07 10:45:27.186272 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:27.186257    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="37bc40d6-5bc7-4b15-8341-6eb834084e4d" containerName="manager"
Sep 07 10:45:27.189187 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:27.189171    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-fcxtf"
Sep 07 10:45:27.191994 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:27.191968    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-52674\""
Sep 07 10:45:27.203896 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:27.203873    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-fcxtf"]
Sep 07 10:45:27.279686 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:27.279651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78e7a1b3-5ab3-46a5-b80d-dd4a468a6f9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-fcxtf\" (UID: \"78e7a1b3-5ab3-46a5-b80d-dd4a468a6f9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-fcxtf"
Sep 07 10:45:27.279871 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:27.279700    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87fmg\" (UniqueName: \"kubernetes.io/projected/78e7a1b3-5ab3-46a5-b80d-dd4a468a6f9a-kube-api-access-87fmg\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-fcxtf\" (UID: \"78e7a1b3-5ab3-46a5-b80d-dd4a468a6f9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-fcxtf"
Sep 07 10:45:27.381008 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:27.380961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78e7a1b3-5ab3-46a5-b80d-dd4a468a6f9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-fcxtf\" (UID: \"78e7a1b3-5ab3-46a5-b80d-dd4a468a6f9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-fcxtf"
Sep 07 10:45:27.381211 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:27.381115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87fmg\" (UniqueName: \"kubernetes.io/projected/78e7a1b3-5ab3-46a5-b80d-dd4a468a6f9a-kube-api-access-87fmg\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-fcxtf\" (UID: \"78e7a1b3-5ab3-46a5-b80d-dd4a468a6f9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-fcxtf"
Sep 07 10:45:27.381404 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:27.381372    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78e7a1b3-5ab3-46a5-b80d-dd4a468a6f9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-fcxtf\" (UID: \"78e7a1b3-5ab3-46a5-b80d-dd4a468a6f9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-fcxtf"
Sep 07 10:45:27.390389 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:27.390360    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87fmg\" (UniqueName: \"kubernetes.io/projected/78e7a1b3-5ab3-46a5-b80d-dd4a468a6f9a-kube-api-access-87fmg\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-fcxtf\" (UID: \"78e7a1b3-5ab3-46a5-b80d-dd4a468a6f9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-fcxtf"
Sep 07 10:45:27.498300 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:27.498253    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-fcxtf"
Sep 07 10:45:27.779500 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:27.779471    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-fcxtf"]
Sep 07 10:45:27.780848 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:45:27.780820    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e7a1b3_5ab3_46a5_b80d_dd4a468a6f9a.slice/crio-1ebd0144c5276f810d2b2be40ffd5f097ba354b3ad132715b101ada269bd97bb WatchSource:0}: Error finding container 1ebd0144c5276f810d2b2be40ffd5f097ba354b3ad132715b101ada269bd97bb: Status 404 returned error can't find the container with id 1ebd0144c5276f810d2b2be40ffd5f097ba354b3ad132715b101ada269bd97bb
Sep 07 10:45:28.106314 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:28.106226    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-fcxtf" event={"ID":"78e7a1b3-5ab3-46a5-b80d-dd4a468a6f9a","Type":"ContainerStarted","Data":"4de3188e5814d34bbdce8dae700530f28e1a559be272eba7ce1a0b0c592ca641"}
Sep 07 10:45:28.106314 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:28.106287    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-fcxtf" event={"ID":"78e7a1b3-5ab3-46a5-b80d-dd4a468a6f9a","Type":"ContainerStarted","Data":"1ebd0144c5276f810d2b2be40ffd5f097ba354b3ad132715b101ada269bd97bb"}
Sep 07 10:45:28.106505 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:28.106428    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-fcxtf"
Sep 07 10:45:28.133016 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:28.132957    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-fcxtf" podStartSLOduration=1.132941291 podStartE2EDuration="1.132941291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:45:28.129206998 +0000 UTC m=+2030.626896921" watchObservedRunningTime="2026-09-07 10:45:28.132941291 +0000 UTC m=+2030.630631201"
Sep 07 10:45:39.113271 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:45:39.113233    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-fcxtf"
Sep 07 10:46:18.574864 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.574823    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"]
Sep 07 10:46:18.584481 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.584440    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.587329 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.587300    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-49bb7129-kserve-self-signed-certs\""
Sep 07 10:46:18.588384 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.588349    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"]
Sep 07 10:46:18.741695 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.741652    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3270295-9c8a-43d8-bf65-6fa65b3169f0-tls-certs\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.741896 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.741721    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-home\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.741896 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.741756    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-dshm\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.741896 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.741854    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbbz5\" (UniqueName: \"kubernetes.io/projected/e3270295-9c8a-43d8-bf65-6fa65b3169f0-kube-api-access-zbbz5\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.742010 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.741900    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-model-cache\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.742010 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.741924    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-tmp-dir\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.842918 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.842826    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-home\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.842918 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.842894    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-dshm\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.843135 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.842951    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbbz5\" (UniqueName: \"kubernetes.io/projected/e3270295-9c8a-43d8-bf65-6fa65b3169f0-kube-api-access-zbbz5\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.843135 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.842994    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-model-cache\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.843135 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.843027    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-tmp-dir\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.843135 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.843091    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3270295-9c8a-43d8-bf65-6fa65b3169f0-tls-certs\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.843361 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.843292    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-home\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.843421 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.843388    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-model-cache\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.843613 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.843590    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-tmp-dir\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.845137 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.845116    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-dshm\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.845697 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.845677    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3270295-9c8a-43d8-bf65-6fa65b3169f0-tls-certs\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.851159 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.851132    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbbz5\" (UniqueName: \"kubernetes.io/projected/e3270295-9c8a-43d8-bf65-6fa65b3169f0-kube-api-access-zbbz5\") pod \"e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") " pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:18.898027 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:18.897988    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:19.170923 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:19.170893    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"]
Sep 07 10:46:19.172420 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:46:19.172396    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3270295_9c8a_43d8_bf65_6fa65b3169f0.slice/crio-b3c0859005791038eb245ee4b5d882984c4c476e23f6e3645ee70aeb3f8add1a WatchSource:0}: Error finding container b3c0859005791038eb245ee4b5d882984c4c476e23f6e3645ee70aeb3f8add1a: Status 404 returned error can't find the container with id b3c0859005791038eb245ee4b5d882984c4c476e23f6e3645ee70aeb3f8add1a
Sep 07 10:46:19.383779 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:19.383728    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt" event={"ID":"e3270295-9c8a-43d8-bf65-6fa65b3169f0","Type":"ContainerStarted","Data":"1bdf4c69c28919df35903c8e87699f39e5b83100a18583b1110c6fc76f5d0edc"}
Sep 07 10:46:19.383962 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:19.383816    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt" event={"ID":"e3270295-9c8a-43d8-bf65-6fa65b3169f0","Type":"ContainerStarted","Data":"b3c0859005791038eb245ee4b5d882984c4c476e23f6e3645ee70aeb3f8add1a"}
Sep 07 10:46:19.400802 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:19.400722    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt" podStartSLOduration=1.400698828 podStartE2EDuration="1.400698828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:46:19.40018546 +0000 UTC m=+2081.897875391" watchObservedRunningTime="2026-09-07 10:46:19.400698828 +0000 UTC m=+2081.898388738"
Sep 07 10:46:28.899174 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:28.899119    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:28.899174 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:28.899192    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:28.911894 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:28.911871    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:46:29.452732 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:46:29.452693    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:47:02.328696 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:02.328660    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"]
Sep 07 10:47:17.355463 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.355398    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt" podUID="e3270295-9c8a-43d8-bf65-6fa65b3169f0" containerName="main" containerID="cri-o://1bdf4c69c28919df35903c8e87699f39e5b83100a18583b1110c6fc76f5d0edc" gracePeriod=45
Sep 07 10:47:17.609718 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.609658    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:47:17.698460 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.698424    2562 generic.go:356] "Generic (PLEG): container finished" podID="e3270295-9c8a-43d8-bf65-6fa65b3169f0" containerID="1bdf4c69c28919df35903c8e87699f39e5b83100a18583b1110c6fc76f5d0edc" exitCode=0
Sep 07 10:47:17.698632 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.698467    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"
Sep 07 10:47:17.698632 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.698511    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt" event={"ID":"e3270295-9c8a-43d8-bf65-6fa65b3169f0","Type":"ContainerDied","Data":"1bdf4c69c28919df35903c8e87699f39e5b83100a18583b1110c6fc76f5d0edc"}
Sep 07 10:47:17.698632 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.698553    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt" event={"ID":"e3270295-9c8a-43d8-bf65-6fa65b3169f0","Type":"ContainerDied","Data":"b3c0859005791038eb245ee4b5d882984c4c476e23f6e3645ee70aeb3f8add1a"}
Sep 07 10:47:17.698632 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.698570    2562 scope.go:117] "RemoveContainer" containerID="1bdf4c69c28919df35903c8e87699f39e5b83100a18583b1110c6fc76f5d0edc"
Sep 07 10:47:17.707096 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.707077    2562 scope.go:117] "RemoveContainer" containerID="1bdf4c69c28919df35903c8e87699f39e5b83100a18583b1110c6fc76f5d0edc"
Sep 07 10:47:17.707362 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:47:17.707344    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bdf4c69c28919df35903c8e87699f39e5b83100a18583b1110c6fc76f5d0edc\": container with ID starting with 1bdf4c69c28919df35903c8e87699f39e5b83100a18583b1110c6fc76f5d0edc not found: ID does not exist" containerID="1bdf4c69c28919df35903c8e87699f39e5b83100a18583b1110c6fc76f5d0edc"
Sep 07 10:47:17.707417 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.707374    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bdf4c69c28919df35903c8e87699f39e5b83100a18583b1110c6fc76f5d0edc"} err="failed to get container status \"1bdf4c69c28919df35903c8e87699f39e5b83100a18583b1110c6fc76f5d0edc\": rpc error: code = NotFound desc = could not find container \"1bdf4c69c28919df35903c8e87699f39e5b83100a18583b1110c6fc76f5d0edc\": container with ID starting with 1bdf4c69c28919df35903c8e87699f39e5b83100a18583b1110c6fc76f5d0edc not found: ID does not exist"
Sep 07 10:47:17.737781 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.737726    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-dshm\") pod \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") "
Sep 07 10:47:17.737975 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.737799    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3270295-9c8a-43d8-bf65-6fa65b3169f0-tls-certs\") pod \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") "
Sep 07 10:47:17.737975 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.737829    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-tmp-dir\") pod \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") "
Sep 07 10:47:17.737975 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.737858    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-home\") pod \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") "
Sep 07 10:47:17.737975 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.737877    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbbz5\" (UniqueName: \"kubernetes.io/projected/e3270295-9c8a-43d8-bf65-6fa65b3169f0-kube-api-access-zbbz5\") pod \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") "
Sep 07 10:47:17.737975 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.737943    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-model-cache\") pod \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\" (UID: \"e3270295-9c8a-43d8-bf65-6fa65b3169f0\") "
Sep 07 10:47:17.738255 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.738100    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-home" (OuterVolumeSpecName: "home") pod "e3270295-9c8a-43d8-bf65-6fa65b3169f0" (UID: "e3270295-9c8a-43d8-bf65-6fa65b3169f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:17.738255 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.738111    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3270295-9c8a-43d8-bf65-6fa65b3169f0" (UID: "e3270295-9c8a-43d8-bf65-6fa65b3169f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:17.738255 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.738201    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-tmp-dir\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:47:17.738255 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.738219    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-home\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:47:17.738458 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.738303    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3270295-9c8a-43d8-bf65-6fa65b3169f0" (UID: "e3270295-9c8a-43d8-bf65-6fa65b3169f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:17.739978 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.739948    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3270295-9c8a-43d8-bf65-6fa65b3169f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3270295-9c8a-43d8-bf65-6fa65b3169f0" (UID: "e3270295-9c8a-43d8-bf65-6fa65b3169f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:47:17.740093 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.740050    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3270295-9c8a-43d8-bf65-6fa65b3169f0-kube-api-access-zbbz5" (OuterVolumeSpecName: "kube-api-access-zbbz5") pod "e3270295-9c8a-43d8-bf65-6fa65b3169f0" (UID: "e3270295-9c8a-43d8-bf65-6fa65b3169f0"). InnerVolumeSpecName "kube-api-access-zbbz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:47:17.740250 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.740231    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-dshm" (OuterVolumeSpecName: "dshm") pod "e3270295-9c8a-43d8-bf65-6fa65b3169f0" (UID: "e3270295-9c8a-43d8-bf65-6fa65b3169f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:17.839289 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.839251    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-dshm\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:47:17.839289 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.839283    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3270295-9c8a-43d8-bf65-6fa65b3169f0-tls-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:47:17.839289 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.839294    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbbz5\" (UniqueName: \"kubernetes.io/projected/e3270295-9c8a-43d8-bf65-6fa65b3169f0-kube-api-access-zbbz5\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:47:17.839522 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:17.839303    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3270295-9c8a-43d8-bf65-6fa65b3169f0-model-cache\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:47:18.015227 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:18.015194    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"]
Sep 07 10:47:18.018996 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:18.018970    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-49bb7129-kserve-694748d4c5-w9qrt"]
Sep 07 10:47:19.966370 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:19.966306    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3270295-9c8a-43d8-bf65-6fa65b3169f0" path="/var/lib/kubelet/pods/e3270295-9c8a-43d8-bf65-6fa65b3169f0/volumes"
Sep 07 10:47:25.835421 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:25.835383    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"]
Sep 07 10:47:25.835840 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:25.835825    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3270295-9c8a-43d8-bf65-6fa65b3169f0" containerName="main"
Sep 07 10:47:25.835901 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:25.835842    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3270295-9c8a-43d8-bf65-6fa65b3169f0" containerName="main"
Sep 07 10:47:25.835941 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:25.835913    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3270295-9c8a-43d8-bf65-6fa65b3169f0" containerName="main"
Sep 07 10:47:25.840516 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:25.840493    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"
Sep 07 10:47:25.853070 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:25.853020    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"]
Sep 07 10:47:26.013961 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:26.013926    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0689f9f4-1878-45ff-8df8-ef95b17aa0fa-maas-api-tls\") pod \"maas-api-6b9564d484-x92tx\" (UID: \"0689f9f4-1878-45ff-8df8-ef95b17aa0fa\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"
Sep 07 10:47:26.014161 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:26.013987    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0689f9f4-1878-45ff-8df8-ef95b17aa0fa-openshift-service-ca\") pod \"maas-api-6b9564d484-x92tx\" (UID: \"0689f9f4-1878-45ff-8df8-ef95b17aa0fa\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"
Sep 07 10:47:26.014161 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:26.014032    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0689f9f4-1878-45ff-8df8-ef95b17aa0fa-metrics-certs\") pod \"maas-api-6b9564d484-x92tx\" (UID: \"0689f9f4-1878-45ff-8df8-ef95b17aa0fa\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"
Sep 07 10:47:26.014161 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:26.014061    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrxpm\" (UniqueName: \"kubernetes.io/projected/0689f9f4-1878-45ff-8df8-ef95b17aa0fa-kube-api-access-lrxpm\") pod \"maas-api-6b9564d484-x92tx\" (UID: \"0689f9f4-1878-45ff-8df8-ef95b17aa0fa\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"
Sep 07 10:47:26.115693 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:26.115583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0689f9f4-1878-45ff-8df8-ef95b17aa0fa-maas-api-tls\") pod \"maas-api-6b9564d484-x92tx\" (UID: \"0689f9f4-1878-45ff-8df8-ef95b17aa0fa\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"
Sep 07 10:47:26.115693 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:26.115684    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0689f9f4-1878-45ff-8df8-ef95b17aa0fa-openshift-service-ca\") pod \"maas-api-6b9564d484-x92tx\" (UID: \"0689f9f4-1878-45ff-8df8-ef95b17aa0fa\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"
Sep 07 10:47:26.115982 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:26.115740    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0689f9f4-1878-45ff-8df8-ef95b17aa0fa-metrics-certs\") pod \"maas-api-6b9564d484-x92tx\" (UID: \"0689f9f4-1878-45ff-8df8-ef95b17aa0fa\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"
Sep 07 10:47:26.115982 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:26.115826    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrxpm\" (UniqueName: \"kubernetes.io/projected/0689f9f4-1878-45ff-8df8-ef95b17aa0fa-kube-api-access-lrxpm\") pod \"maas-api-6b9564d484-x92tx\" (UID: \"0689f9f4-1878-45ff-8df8-ef95b17aa0fa\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"
Sep 07 10:47:26.116443 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:26.116420    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0689f9f4-1878-45ff-8df8-ef95b17aa0fa-openshift-service-ca\") pod \"maas-api-6b9564d484-x92tx\" (UID: \"0689f9f4-1878-45ff-8df8-ef95b17aa0fa\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"
Sep 07 10:47:26.118274 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:26.118246    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0689f9f4-1878-45ff-8df8-ef95b17aa0fa-metrics-certs\") pod \"maas-api-6b9564d484-x92tx\" (UID: \"0689f9f4-1878-45ff-8df8-ef95b17aa0fa\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"
Sep 07 10:47:26.118389 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:26.118298    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0689f9f4-1878-45ff-8df8-ef95b17aa0fa-maas-api-tls\") pod \"maas-api-6b9564d484-x92tx\" (UID: \"0689f9f4-1878-45ff-8df8-ef95b17aa0fa\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"
Sep 07 10:47:26.125128 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:26.125105    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrxpm\" (UniqueName: \"kubernetes.io/projected/0689f9f4-1878-45ff-8df8-ef95b17aa0fa-kube-api-access-lrxpm\") pod \"maas-api-6b9564d484-x92tx\" (UID: \"0689f9f4-1878-45ff-8df8-ef95b17aa0fa\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"
Sep 07 10:47:26.155303 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:26.155269    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"
Sep 07 10:47:26.438027 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:26.437995    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 10:47:26.438309 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:26.438280    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"]
Sep 07 10:47:26.751363 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:26.751315    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx" event={"ID":"0689f9f4-1878-45ff-8df8-ef95b17aa0fa","Type":"ContainerStarted","Data":"4db4a94104c9b8a22d262a55b71eb2e147d4ceab803ae3862efc66354fcef4f7"}
Sep 07 10:47:27.758514 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:27.758479    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx" event={"ID":"0689f9f4-1878-45ff-8df8-ef95b17aa0fa","Type":"ContainerStarted","Data":"3c15f1ae61cad83a59d4cfbb6450be7fc9771ac9bad81da21732e5fd71ea565f"}
Sep 07 10:47:27.758936 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:27.758553    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"
Sep 07 10:47:27.778442 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:27.778387    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx" podStartSLOduration=2.38066677 podStartE2EDuration="2.7783693s" totalImagesPullingTime="397.70253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:47:27.776540419 +0000 UTC m=+2150.274230401" watchObservedRunningTime="2026-09-07 10:47:27.7783693 +0000 UTC m=+2150.276059209"
Sep 07 10:47:33.769821 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:33.769738    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b9564d484-x92tx"
Sep 07 10:47:33.820004 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:33.819965    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"]
Sep 07 10:47:33.820424 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:33.820376    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc" podUID="00c60791-4193-4a28-bf52-bb5cbd452f07" containerName="maas-api" containerID="cri-o://3a69d45789d91d8eddc9547062df7fe19c2121606fec5a7cc2c71bc17ed48bf4" gracePeriod=30
Sep 07 10:47:35.073097 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.073067    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:47:35.197320 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.197188    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c60791-4193-4a28-bf52-bb5cbd452f07-maas-api-tls\") pod \"00c60791-4193-4a28-bf52-bb5cbd452f07\" (UID: \"00c60791-4193-4a28-bf52-bb5cbd452f07\") "
Sep 07 10:47:35.197320 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.197258    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzmbz\" (UniqueName: \"kubernetes.io/projected/00c60791-4193-4a28-bf52-bb5cbd452f07-kube-api-access-hzmbz\") pod \"00c60791-4193-4a28-bf52-bb5cbd452f07\" (UID: \"00c60791-4193-4a28-bf52-bb5cbd452f07\") "
Sep 07 10:47:35.197568 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.197324    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c60791-4193-4a28-bf52-bb5cbd452f07-openshift-service-ca\") pod \"00c60791-4193-4a28-bf52-bb5cbd452f07\" (UID: \"00c60791-4193-4a28-bf52-bb5cbd452f07\") "
Sep 07 10:47:35.197568 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.197504    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c60791-4193-4a28-bf52-bb5cbd452f07-metrics-certs\") pod \"00c60791-4193-4a28-bf52-bb5cbd452f07\" (UID: \"00c60791-4193-4a28-bf52-bb5cbd452f07\") "
Sep 07 10:47:35.197693 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.197670    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c60791-4193-4a28-bf52-bb5cbd452f07-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "00c60791-4193-4a28-bf52-bb5cbd452f07" (UID: "00c60791-4193-4a28-bf52-bb5cbd452f07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:47:35.197886 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.197864    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c60791-4193-4a28-bf52-bb5cbd452f07-openshift-service-ca\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:47:35.199420 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.199393    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c60791-4193-4a28-bf52-bb5cbd452f07-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "00c60791-4193-4a28-bf52-bb5cbd452f07" (UID: "00c60791-4193-4a28-bf52-bb5cbd452f07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:47:35.199514 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.199419    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c60791-4193-4a28-bf52-bb5cbd452f07-kube-api-access-hzmbz" (OuterVolumeSpecName: "kube-api-access-hzmbz") pod "00c60791-4193-4a28-bf52-bb5cbd452f07" (UID: "00c60791-4193-4a28-bf52-bb5cbd452f07"). InnerVolumeSpecName "kube-api-access-hzmbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:47:35.199514 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.199421    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c60791-4193-4a28-bf52-bb5cbd452f07-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "00c60791-4193-4a28-bf52-bb5cbd452f07" (UID: "00c60791-4193-4a28-bf52-bb5cbd452f07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:47:35.299071 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.299030    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c60791-4193-4a28-bf52-bb5cbd452f07-metrics-certs\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:47:35.299071 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.299066    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c60791-4193-4a28-bf52-bb5cbd452f07-maas-api-tls\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:47:35.299071 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.299077    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzmbz\" (UniqueName: \"kubernetes.io/projected/00c60791-4193-4a28-bf52-bb5cbd452f07-kube-api-access-hzmbz\") on node \"ip-10-0-141-62.ec2.internal\" DevicePath \"\""
Sep 07 10:47:35.803175 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.803135    2562 generic.go:356] "Generic (PLEG): container finished" podID="00c60791-4193-4a28-bf52-bb5cbd452f07" containerID="3a69d45789d91d8eddc9547062df7fe19c2121606fec5a7cc2c71bc17ed48bf4" exitCode=0
Sep 07 10:47:35.803175 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.803176    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"
Sep 07 10:47:35.803406 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.803183    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc" event={"ID":"00c60791-4193-4a28-bf52-bb5cbd452f07","Type":"ContainerDied","Data":"3a69d45789d91d8eddc9547062df7fe19c2121606fec5a7cc2c71bc17ed48bf4"}
Sep 07 10:47:35.803406 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.803233    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc" event={"ID":"00c60791-4193-4a28-bf52-bb5cbd452f07","Type":"ContainerDied","Data":"4f73d886d572935829cafedcedfc58c03a24800c01735bfe82f8aab02c669344"}
Sep 07 10:47:35.803406 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.803257    2562 scope.go:117] "RemoveContainer" containerID="3a69d45789d91d8eddc9547062df7fe19c2121606fec5a7cc2c71bc17ed48bf4"
Sep 07 10:47:35.812575 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.812550    2562 scope.go:117] "RemoveContainer" containerID="3a69d45789d91d8eddc9547062df7fe19c2121606fec5a7cc2c71bc17ed48bf4"
Sep 07 10:47:35.812884 ip-10-0-141-62 kubenswrapper[2562]: E0907 10:47:35.812862    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a69d45789d91d8eddc9547062df7fe19c2121606fec5a7cc2c71bc17ed48bf4\": container with ID starting with 3a69d45789d91d8eddc9547062df7fe19c2121606fec5a7cc2c71bc17ed48bf4 not found: ID does not exist" containerID="3a69d45789d91d8eddc9547062df7fe19c2121606fec5a7cc2c71bc17ed48bf4"
Sep 07 10:47:35.812960 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.812893    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a69d45789d91d8eddc9547062df7fe19c2121606fec5a7cc2c71bc17ed48bf4"} err="failed to get container status \"3a69d45789d91d8eddc9547062df7fe19c2121606fec5a7cc2c71bc17ed48bf4\": rpc error: code = NotFound desc = could not find container \"3a69d45789d91d8eddc9547062df7fe19c2121606fec5a7cc2c71bc17ed48bf4\": container with ID starting with 3a69d45789d91d8eddc9547062df7fe19c2121606fec5a7cc2c71bc17ed48bf4 not found: ID does not exist"
Sep 07 10:47:35.836910 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.836873    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"]
Sep 07 10:47:35.844620 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.844586    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7596b5dfb4-2hklc"]
Sep 07 10:47:35.965399 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:47:35.965358    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00c60791-4193-4a28-bf52-bb5cbd452f07" path="/var/lib/kubelet/pods/00c60791-4193-4a28-bf52-bb5cbd452f07/volumes"
Sep 07 10:48:04.171685 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:04.171649    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-fz2ql_2774810d-622e-4fce-9911-1e9684c90ab3/manager/0.log"
Sep 07 10:48:04.290381 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:04.290338    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hww6q_84ac334d-8bfc-40cd-81cd-0e12f7179acc/manager/0.log"
Sep 07 10:48:04.421403 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:04.421361    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6899cfd859-44brf_87c13c2c-2f9c-4794-93fa-3b547b623a62/manager/0.log"
Sep 07 10:48:04.538949 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:04.538917    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-kjmb6_489fcc35-40b7-48d0-93f7-ecfb91e71d19/server/0.log"
Sep 07 10:48:04.911340 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:04.911259    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685c7f779d-jpm2r_b1e2706d-036b-464d-8c8b-e34513f33a39/copy-manifests/0.log"
Sep 07 10:48:04.917174 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:04.917151    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685c7f779d-jpm2r_b1e2706d-036b-464d-8c8b-e34513f33a39/manager/0.log"
Sep 07 10:48:05.500486 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:05.500453    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b9564d484-x92tx_0689f9f4-1878-45ff-8df8-ef95b17aa0fa/maas-api/0.log"
Sep 07 10:48:05.831463 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:05.831378    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-46pxc_657ea806-9abd-4930-bb33-12f76eebf5b2/postgres/0.log"
Sep 07 10:48:06.736461 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:06.736431    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-qck72_05332adc-2bbd-4592-8bb6-5c22a0183df6/manager/0.log"
Sep 07 10:48:06.846576 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:06.846547    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-5pdhl_7412841d-c20e-464a-b4c9-3fb5656954b1/manager/0.log"
Sep 07 10:48:07.086644 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:07.086571    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-fcxtf_78e7a1b3-5ab3-46a5-b80d-dd4a468a6f9a/manager/0.log"
Sep 07 10:48:07.317569 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:07.317541    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-wvfb9_f4b88551-5dee-416b-bbf7-a514dc9ee2f1/manager/0.log"
Sep 07 10:48:08.020540 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:08.020513    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7959f5887b-z7h6k_8fa26e1a-d2b7-49fe-b0ab-ca422cd60929/kube-auth-proxy/0.log"
Sep 07 10:48:08.262385 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:08.262358    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-btccx_c83ac846-f741-4fd9-bf3c-d0733fc9a4c2/payload-pre-processing/0.log"
Sep 07 10:48:09.110590 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:09.110556    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-6vh4d_4d7052df-af1d-475e-9f29-151723309ba5/main/0.log"
Sep 07 10:48:09.349260 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:09.349225    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9p5c_ceeff70b-0f4d-4088-ae6a-89d03d9c385f/main/0.log"
Sep 07 10:48:09.579279 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:09.579247    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-vws5s_d654b656-f1c9-4a80-9c62-63df56dac3dd/main/0.log"
Sep 07 10:48:18.574315 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:18.574287    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tk7lm_e0bf9977-9b65-4b22-958e-46b52f65079e/global-pull-secret-syncer/0.log"
Sep 07 10:48:18.701722 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:18.701693    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xgmhs_6ba3d830-3508-49a6-8d41-36409f735098/konnectivity-agent/0.log"
Sep 07 10:48:18.773118 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:18.773090    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-62.ec2.internal_236f13cb4cc1d42288493688c2e3dab9/haproxy/0.log"
Sep 07 10:48:23.234709 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:23.234681    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-qck72_05332adc-2bbd-4592-8bb6-5c22a0183df6/manager/0.log"
Sep 07 10:48:23.270551 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:23.270523    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-5pdhl_7412841d-c20e-464a-b4c9-3fb5656954b1/manager/0.log"
Sep 07 10:48:23.337664 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:23.337631    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-fcxtf_78e7a1b3-5ab3-46a5-b80d-dd4a468a6f9a/manager/0.log"
Sep 07 10:48:23.428548 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:23.428517    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-wvfb9_f4b88551-5dee-416b-bbf7-a514dc9ee2f1/manager/0.log"
Sep 07 10:48:25.126260 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:25.126227    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7c58fd999c-nwwxd_92c0c47f-d30b-48f8-8619-e558dd51c3e2/metrics-server/0.log"
Sep 07 10:48:25.267833 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:25.267801    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tk2c8_e7ef93be-7d82-4003-ac92-7f2a9eb75eef/node-exporter/0.log"
Sep 07 10:48:25.291381 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:25.291354    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tk2c8_e7ef93be-7d82-4003-ac92-7f2a9eb75eef/kube-rbac-proxy/0.log"
Sep 07 10:48:25.314438 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:25.314412    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tk2c8_e7ef93be-7d82-4003-ac92-7f2a9eb75eef/init-textfile/0.log"
Sep 07 10:48:25.654967 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:25.654937    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-flj7x_25edec58-0c81-443b-8fd7-1a40acd6d7ea/prometheus-operator/0.log"
Sep 07 10:48:25.672568 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:25.672543    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-flj7x_25edec58-0c81-443b-8fd7-1a40acd6d7ea/kube-rbac-proxy/0.log"
Sep 07 10:48:29.060343 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.060308    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"]
Sep 07 10:48:29.060737 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.060671    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00c60791-4193-4a28-bf52-bb5cbd452f07" containerName="maas-api"
Sep 07 10:48:29.060737 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.060682    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00c60791-4193-4a28-bf52-bb5cbd452f07" containerName="maas-api"
Sep 07 10:48:29.060874 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.060781    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="00c60791-4193-4a28-bf52-bb5cbd452f07" containerName="maas-api"
Sep 07 10:48:29.063884 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.063866    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.066188 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.066163    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhr67\"/\"kube-root-ca.crt\""
Sep 07 10:48:29.066314 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.066224    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dhr67\"/\"default-dockercfg-ftpfk\""
Sep 07 10:48:29.067142 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.067124    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhr67\"/\"openshift-service-ca.crt\""
Sep 07 10:48:29.074066 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.074041    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"]
Sep 07 10:48:29.120937 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.120902    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7db6c54f-ef8c-4be8-a7be-9ca787f2fb80-sys\") pod \"perf-node-gather-daemonset-rcmvk\" (UID: \"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.121138 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.120956    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7db6c54f-ef8c-4be8-a7be-9ca787f2fb80-proc\") pod \"perf-node-gather-daemonset-rcmvk\" (UID: \"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.121138 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.120984    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7db6c54f-ef8c-4be8-a7be-9ca787f2fb80-lib-modules\") pod \"perf-node-gather-daemonset-rcmvk\" (UID: \"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.121138 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.121042    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7db6c54f-ef8c-4be8-a7be-9ca787f2fb80-podres\") pod \"perf-node-gather-daemonset-rcmvk\" (UID: \"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.121138 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.121077    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sbzf\" (UniqueName: \"kubernetes.io/projected/7db6c54f-ef8c-4be8-a7be-9ca787f2fb80-kube-api-access-5sbzf\") pod \"perf-node-gather-daemonset-rcmvk\" (UID: \"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.221969 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.221929    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7db6c54f-ef8c-4be8-a7be-9ca787f2fb80-sys\") pod \"perf-node-gather-daemonset-rcmvk\" (UID: \"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.222169 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.221985    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7db6c54f-ef8c-4be8-a7be-9ca787f2fb80-proc\") pod \"perf-node-gather-daemonset-rcmvk\" (UID: \"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.222169 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.222022    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7db6c54f-ef8c-4be8-a7be-9ca787f2fb80-lib-modules\") pod \"perf-node-gather-daemonset-rcmvk\" (UID: \"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.222169 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.222044    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7db6c54f-ef8c-4be8-a7be-9ca787f2fb80-podres\") pod \"perf-node-gather-daemonset-rcmvk\" (UID: \"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.222169 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.222061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sbzf\" (UniqueName: \"kubernetes.io/projected/7db6c54f-ef8c-4be8-a7be-9ca787f2fb80-kube-api-access-5sbzf\") pod \"perf-node-gather-daemonset-rcmvk\" (UID: \"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.222169 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.222068    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7db6c54f-ef8c-4be8-a7be-9ca787f2fb80-sys\") pod \"perf-node-gather-daemonset-rcmvk\" (UID: \"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.222169 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.222159    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7db6c54f-ef8c-4be8-a7be-9ca787f2fb80-proc\") pod \"perf-node-gather-daemonset-rcmvk\" (UID: \"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.222404 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.222188    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7db6c54f-ef8c-4be8-a7be-9ca787f2fb80-podres\") pod \"perf-node-gather-daemonset-rcmvk\" (UID: \"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.222404 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.222181    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7db6c54f-ef8c-4be8-a7be-9ca787f2fb80-lib-modules\") pod \"perf-node-gather-daemonset-rcmvk\" (UID: \"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.232435 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.232405    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sbzf\" (UniqueName: \"kubernetes.io/projected/7db6c54f-ef8c-4be8-a7be-9ca787f2fb80-kube-api-access-5sbzf\") pod \"perf-node-gather-daemonset-rcmvk\" (UID: \"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.374985 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.374895    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:29.647413 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:29.647384    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"]
Sep 07 10:48:29.649714 ip-10-0-141-62 kubenswrapper[2562]: W0907 10:48:29.649684    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7db6c54f_ef8c_4be8_a7be_9ca787f2fb80.slice/crio-2b1f9af9f50d3d854090edac3cb7e18ad7428ac737284d407632fc1dd4415205 WatchSource:0}: Error finding container 2b1f9af9f50d3d854090edac3cb7e18ad7428ac737284d407632fc1dd4415205: Status 404 returned error can't find the container with id 2b1f9af9f50d3d854090edac3cb7e18ad7428ac737284d407632fc1dd4415205
Sep 07 10:48:30.094907 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:30.094870    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk" event={"ID":"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80","Type":"ContainerStarted","Data":"871d32bf696bd84e017747aed259e96e1b7097506745e871253eb143dda23830"}
Sep 07 10:48:30.095394 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:30.094929    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk" event={"ID":"7db6c54f-ef8c-4be8-a7be-9ca787f2fb80","Type":"ContainerStarted","Data":"2b1f9af9f50d3d854090edac3cb7e18ad7428ac737284d407632fc1dd4415205"}
Sep 07 10:48:30.095394 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:30.094999    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:30.110181 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:30.110129    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk" podStartSLOduration=1.110110936 podStartE2EDuration="1.110110936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:48:30.109534498 +0000 UTC m=+2212.607224407" watchObservedRunningTime="2026-09-07 10:48:30.110110936 +0000 UTC m=+2212.607800849"
Sep 07 10:48:30.733830 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:30.733801    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6j256_f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20/dns/0.log"
Sep 07 10:48:30.754472 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:30.754430    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6j256_f4f159ed-ccc9-40b9-8fc4-f1feb8fe7f20/kube-rbac-proxy/0.log"
Sep 07 10:48:30.909502 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:30.909474    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qjnz6_ac0c45e5-27f8-4c26-929a-001e4b644d46/dns-node-resolver/0.log"
Sep 07 10:48:31.516678 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:31.516646    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v86rj_f37b4a59-9c92-41fd-9561-ed2d017c6751/node-ca/0.log"
Sep 07 10:48:33.027636 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:33.027602    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7959f5887b-z7h6k_8fa26e1a-d2b7-49fe-b0ab-ca422cd60929/kube-auth-proxy/0.log"
Sep 07 10:48:33.129229 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:33.129194    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-btccx_c83ac846-f741-4fd9-bf3c-d0733fc9a4c2/payload-pre-processing/0.log"
Sep 07 10:48:34.050583 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:34.050554    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d2r97_da5527f2-1ae4-4b33-a1f9-dff0689388e8/serve-healthcheck-canary/0.log"
Sep 07 10:48:34.576927 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:34.576895    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4f5vn_295fae86-047c-4eb7-a003-622609960fe9/kube-rbac-proxy/0.log"
Sep 07 10:48:34.597087 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:34.597060    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4f5vn_295fae86-047c-4eb7-a003-622609960fe9/exporter/0.log"
Sep 07 10:48:34.621014 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:34.620990    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4f5vn_295fae86-047c-4eb7-a003-622609960fe9/extractor/0.log"
Sep 07 10:48:36.110552 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:36.110520    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-rcmvk"
Sep 07 10:48:36.768815 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:36.768756    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-fz2ql_2774810d-622e-4fce-9911-1e9684c90ab3/manager/0.log"
Sep 07 10:48:36.951513 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:36.951476    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hww6q_84ac334d-8bfc-40cd-81cd-0e12f7179acc/manager/0.log"
Sep 07 10:48:36.997502 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:36.997465    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6899cfd859-44brf_87c13c2c-2f9c-4794-93fa-3b547b623a62/manager/0.log"
Sep 07 10:48:37.016904 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:37.016877    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-kjmb6_489fcc35-40b7-48d0-93f7-ecfb91e71d19/server/0.log"
Sep 07 10:48:37.222561 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:37.222533    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685c7f779d-jpm2r_b1e2706d-036b-464d-8c8b-e34513f33a39/manager/0.log"
Sep 07 10:48:37.243013 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:37.242985    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685c7f779d-jpm2r_b1e2706d-036b-464d-8c8b-e34513f33a39/copy-manifests/0.log"
Sep 07 10:48:38.853585 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:38.853539    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-nsp2p_c9d853ee-540f-4886-904f-b7e32d365e30/openshift-lws-operator/0.log"
Sep 07 10:48:40.184708 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:40.184678    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7959f5887b-z7h6k_8fa26e1a-d2b7-49fe-b0ab-ca422cd60929/kube-auth-proxy/0.log"
Sep 07 10:48:40.251650 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:40.251616    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-btccx_c83ac846-f741-4fd9-bf3c-d0733fc9a4c2/payload-pre-processing/0.log"
Sep 07 10:48:42.435130 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:42.435088    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7959f5887b-z7h6k_8fa26e1a-d2b7-49fe-b0ab-ca422cd60929/kube-auth-proxy/0.log"
Sep 07 10:48:42.507961 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:42.507930    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-btccx_c83ac846-f741-4fd9-bf3c-d0733fc9a4c2/payload-pre-processing/0.log"
Sep 07 10:48:44.378977 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:44.378944    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kzc8f_39ac5c95-1a0d-479e-9427-33688fdf5751/migrator/0.log"
Sep 07 10:48:44.401622 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:44.401587    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kzc8f_39ac5c95-1a0d-479e-9427-33688fdf5751/graceful-termination/0.log"
Sep 07 10:48:45.981720 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:45.981693    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6kdt9_757b7332-c3c9-4849-a0b7-cf8a4d46b3c1/kube-multus-additional-cni-plugins/0.log"
Sep 07 10:48:46.002124 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:46.002094    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6kdt9_757b7332-c3c9-4849-a0b7-cf8a4d46b3c1/egress-router-binary-copy/0.log"
Sep 07 10:48:46.022954 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:46.022925    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6kdt9_757b7332-c3c9-4849-a0b7-cf8a4d46b3c1/cni-plugins/0.log"
Sep 07 10:48:46.043018 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:46.042992    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6kdt9_757b7332-c3c9-4849-a0b7-cf8a4d46b3c1/bond-cni-plugin/0.log"
Sep 07 10:48:46.062859 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:46.062832    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6kdt9_757b7332-c3c9-4849-a0b7-cf8a4d46b3c1/routeoverride-cni/0.log"
Sep 07 10:48:46.083149 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:46.083124    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6kdt9_757b7332-c3c9-4849-a0b7-cf8a4d46b3c1/whereabouts-cni-bincopy/0.log"
Sep 07 10:48:46.103092 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:46.103063    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6kdt9_757b7332-c3c9-4849-a0b7-cf8a4d46b3c1/whereabouts-cni/0.log"
Sep 07 10:48:46.419296 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:46.419214    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-tmj6t_70b33bfb-c309-4092-9e67-f09a30673a7d/kube-multus/0.log"
Sep 07 10:48:46.485771 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:46.485730    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l7n5z_7254bd39-6628-42fd-96e9-6a01b723c275/network-metrics-daemon/0.log"
Sep 07 10:48:46.502053 ip-10-0-141-62 kubenswrapper[2562]: I0907 10:48:46.502028    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l7n5z_7254bd39-6628-42fd-96e9-6a01b723c275/kube-rbac-proxy/0.log"